During ANALYZE, in analyze.c, functions compute_minimal_stats
and compute_scalar_stats, values whose length exceed
WIDTH_THRESHOLD (= 1024) are not used for calculating statistics
other than that they are counted as "too wide rows" and assumed
to be all different.
This works fine with regular tables; values exceeding that threshold
don't get detoasted and won't consume excessive memory.
With foreign tables the situation is different. Even though
values exceeding WIDTH_THRESHOLD won't get used, the complete
rows will be fetched from the foreign table. This can easily
exhaust maintenance_work_mem.
A foreign data wrapper has no good way to counter the problem.
It can return truncated values in ist AcquireSampleRowsFunc,
but WIDTH_THRESHOLD is private to analyze.c and it's a bad idea
to hard code a cutoff limit of 1025.
I can think of two remedies:
1) Expose WIDTH_THRESHOLD in commands/vacuum.h and add documentation so that the authors of foreign data wrappers are
awareof the problem and can avoid it on their side. This would be quite simple.
2) Instead of one callback that returns all sample rows, have a callback that just returns the next table row (or the
next table row from a subset of the pages of the table in the internal case). This function could be called in a
short-lived memory context. Vitter's algorithm for selecting a sample and the truncation of excessively long values
wouldthen be handled in analyze.c. This would avoid the problem completely and make it easier to write a foreign
datawrapper. I haven't thought this out completely, and it would require bigger changes to analyze.c and the API than
areprobably welcome this close to beta.
What is your opinion?
Yours,
Laurenz Albe
