Re: [HACKERS] pgbench more operators & functions - Mailing list pgsql-hackers

>> As it stands right now you haven't provided enough context for this patch
>> and only the social difficulty of actually marking a patch rejected has
>> prevented its demise in its current form - because while it has interesting
>> ideas its added maintenance burden for -core without any in-core usage.
>> But it you make it the first patch in a 3-patch series that implements the
>> per-spec tpc-b the discussion moves away from these support functions and
>> into the broader framework in which they are made useful.
>
> I think Fabien already did post something of the sort, or at least 
> discussion towards it,

Yep.

> and there was immediately objection as to whether his idea of TPC-B 
> compliance was actually right. I remember complaining that he had a 
> totally artificial idea of what "fetching a data value" requires.

Yep.

I think that the key misunderstanding is that you are honest and assume 
that other people are honest too. This is naïve: There is a long history 
of vendors creatively "cheating" to get better than deserve benchmark 
results. Benchmark specifications try to prevent such behaviors by laying 
careful requirements and procedures.

In this instance, you "know" that when pg has returned the result of the 
query the data is actually on the client side, so you considered it is 
fetched. That is fine for you, but from a benchmarking perspective with 
external auditors your belief is not good enough.

For instance, the vendor could implement a new version of the protocol 
where the data are only transfered on demand, and the result just tells 
that the data is indeed somewhere on the server (eg on "SELECT abalance" 
it could just check that the key exists, no need to actually fetch the 
data from the table, so no need to read the table, the index is 
enough...). That would be pretty stupid for real application performance, 
but the benchmark would could get better tps by doing so.

Without even intentionnaly cheating, this could be part of a useful 
"streaming mode" protocol option which make sense for very large results 
but would be activated for a small result.

Another point is that decoding the message may be a little expensive, so 
that by not actually extracting the data into the client but just keeping 
it in the connection/OS one gets better performance.

Thus, TPC-B 2.0.0 benchmark specification says:

"1.3.2 Each transaction shall return to the driver the Account_Balance 
resulting from successful commit of the transaction.

Comment: It is the intent of this clause that the account balance in the 
database be returned to the driver, i.e., that the application retrieve 
the account balance."

For me the correct interpretation of "the APPLICATION retrieve the account 
balance" is that the client application code, pgbench in this context, did 
indeed get the value from the vendor code, here "libpq" which is handling 
the connection.

Having the value discarded from libpq by calling PQclear instead of 
PQntuples/PQgetvalue/... skips a key part of the client code that no real 
application would skip. This looks strange and is not representative of 
real client code: as a potential auditor, because of this I would not 
check the corresponding item in the audit check list:
 "11.3.1.2 Verify that transaction inputs and outputs satisfy Clause 1.3."

So the benchmark implementation would not be validated.


Another trivial reason to be able to actually retrieve data is that for 
benchmarking purpose it is very easy to want to test a scenario where you 
want to do different things based on data received, which imply that the 
data can be manipulated somehow on the benchmarking client side, which is 
currently not possible.

-- 
Fabien.