Thread: 2GB limit for temp_file_limit on 32bit platform
Hello I did a backport of temp_file_limit feature to 9.1, but when we tested this patch, we found very restristrictive limit to 2GB. 2GB is nonsense, because this is session limit of temp files, and these files should be longer than 2GB. Regards Pavel
On 07/19/2012 01:04 PM, Pavel Stehule wrote: > I did a backport of temp_file_limit feature to 9.1, but when we tested > this patch, we found very restristrictive limit to 2GB. > > 2GB is nonsense, because this is session limit of temp files, and > these files should be longer than 2GB. I haven't read the patch but... don't all 32bit platforms have a 2GB limit (by default)? Sincerely, jD -- Command Prompt, Inc. - http://www.commandprompt.com/ PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC @cmdpromptinc - 509-416-6579
On Thu, Jul 19, 2012 at 4:29 PM, Joshua D. Drake <jd@commandprompt.com> wrote: > > On 07/19/2012 01:04 PM, Pavel Stehule wrote: >> >> I did a backport of temp_file_limit feature to 9.1, but when we tested >> this patch, we found very restristrictive limit to 2GB. >> >> 2GB is nonsense, because this is session limit of temp files, and >> these files should be longer than 2GB. > > > I haven't read the patch but... don't all 32bit platforms have a 2GB limit > (by default)? I don't think so. LFS got done in the mid-90s, which is long enough ago for people to start forgetting about it. Are there any supported platforms that didn't adopt LFS? http://en.wikipedia.org/wiki/Large_file_support -- When confronted by a difficult problem, solve it by reducing it to the question, "How would the Lone Ranger handle this?"
On 07/19/2012 01:48 PM, Christopher Browne wrote: > > On Thu, Jul 19, 2012 at 4:29 PM, Joshua D. Drake <jd@commandprompt.com> wrote: >> >> On 07/19/2012 01:04 PM, Pavel Stehule wrote: >>> >>> I did a backport of temp_file_limit feature to 9.1, but when we tested >>> this patch, we found very restristrictive limit to 2GB. >>> >>> 2GB is nonsense, because this is session limit of temp files, and >>> these files should be longer than 2GB. >> >> >> I haven't read the patch but... don't all 32bit platforms have a 2GB limit >> (by default)? > > I don't think so. > > LFS got done in the mid-90s, which is long enough ago for people to > start forgetting about it. Are there any supported platforms that > didn't adopt LFS? > > http://en.wikipedia.org/wiki/Large_file_support Note: "by default" :). I know they could support LFS but as I recall you had to compile specifically for it (at least on linux and old versions of pg). So I was curious if it was that specific limitation or a limitation within the Pg code itself. Sincerely, Joshua D. Drake -- Command Prompt, Inc. - http://www.commandprompt.com/ PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC @cmdpromptinc - 509-416-6579
On 20/07/12 09:08, Joshua D. Drake wrote: > > On 07/19/2012 01:48 PM, Christopher Browne wrote: >> >> On Thu, Jul 19, 2012 at 4:29 PM, Joshua D. Drake >> <jd@commandprompt.com> wrote: >>> >>> On 07/19/2012 01:04 PM, Pavel Stehule wrote: >>>> >>>> I did a backport of temp_file_limit feature to 9.1, but when we tested >>>> this patch, we found very restristrictive limit to 2GB. >>>> >>>> 2GB is nonsense, because this is session limit of temp files, and >>>> these files should be longer than 2GB. >>> >>> >>> I haven't read the patch but... don't all 32bit platforms have a 2GB >>> limit >>> (by default)? >> >> I don't think so. >> >> LFS got done in the mid-90s, which is long enough ago for people to >> start forgetting about it. Are there any supported platforms that >> didn't adopt LFS? >> >> http://en.wikipedia.org/wiki/Large_file_support > > Note: "by default" :). I know they could support LFS but as I recall > you had to compile specifically for it (at least on linux and old > versions of pg). > > So I was curious if it was that specific limitation or a limitation > within the Pg code itself. > > It is to do with the datatype of the GUC used for the setting - I haven't got the patch in from of me to look at, but recall that going larger meant using a float type which meant you couldn't get nice units displayed (MB, GB etc). I'll take a proper look later. Cheers Mark
On 20/07/12 09:58, Mark Kirkwood wrote: > On 20/07/12 09:08, Joshua D. Drake wrote: >> >> On 07/19/2012 01:48 PM, Christopher Browne wrote: >>> >>> On Thu, Jul 19, 2012 at 4:29 PM, Joshua D. Drake >>> <jd@commandprompt.com> wrote: >>>> >>>> On 07/19/2012 01:04 PM, Pavel Stehule wrote: >>>>> >>>>> I did a backport of temp_file_limit feature to 9.1, but when we >>>>> tested >>>>> this patch, we found very restristrictive limit to 2GB. >>>>> >>>>> 2GB is nonsense, because this is session limit of temp files, and >>>>> these files should be longer than 2GB. >>>> >>>> >>>> I haven't read the patch but... don't all 32bit platforms have a >>>> 2GB limit >>>> (by default)? >>> >>> I don't think so. >>> >>> LFS got done in the mid-90s, which is long enough ago for people to >>> start forgetting about it. Are there any supported platforms that >>> didn't adopt LFS? >>> >>> http://en.wikipedia.org/wiki/Large_file_support >> >> Note: "by default" :). I know they could support LFS but as I recall >> you had to compile specifically for it (at least on linux and old >> versions of pg). >> >> So I was curious if it was that specific limitation or a limitation >> within the Pg code itself. >> >> > > It is to do with the datatype of the GUC used for the setting - I > haven't got the patch in from of me to look at, but recall that going > larger meant using a float type which meant you couldn't get nice > units displayed (MB, GB etc). > > I'll take a proper look later. > > From src/backend/utils/misc/guc.c {"temp_file_limit", PGC_SUSET, RESOURCES_DISK, gettext_noop("Limits the total size of all temp files used by each session."), gettext_noop("-1 means no limit."), GUC_UNIT_KB }, &temp_file_limit, -1, -1, INT_MAX, NULL, NULL, NULL },
Pavel Stehule <pavel.stehule@gmail.com> writes:
> I did a backport of temp_file_limit feature to 9.1, but when we tested
> this patch, we found very restristrictive limit to 2GB.
> 2GB is nonsense, because this is session limit of temp files, and
> these files should be longer than 2GB.
This claim is nonsense. The variable's value is measured in KB, so the
effective limit is actually 2TB not 2GB.
regards, tom lane
On 20/07/12 12:02, Tom Lane wrote:
> Pavel Stehule <pavel.stehule@gmail.com> writes:
>> I did a backport of temp_file_limit feature to 9.1, but when we tested
>> this patch, we found very restristrictive limit to 2GB.
>> 2GB is nonsense, because this is session limit of temp files, and
>> these files should be longer than 2GB.
> This claim is nonsense. The variable's value is measured in KB, so the
> effective limit is actually 2TB not 2GB.
>
>
Did you guys perchance pick up one of the earlier patches that had
MAX_KILOBYTES instead of INT_MAX as the limit? i.e:
{"temp_file_limit", PGC_SUSET, RESOURCES_DISK, gettext_noop("Limits the total size of all temp files
used
by each session."), gettext_noop("-1 means no limit."), GUC_UNIT_KB },
&temp_file_limit, -1, -1, **MAX_KILOBYTES**, <==== NULL, NULL, NULL },
2012/7/20 Tom Lane <tgl@sss.pgh.pa.us>: > Pavel Stehule <pavel.stehule@gmail.com> writes: >> I did a backport of temp_file_limit feature to 9.1, but when we tested >> this patch, we found very restristrictive limit to 2GB. > >> 2GB is nonsense, because this is session limit of temp files, and >> these files should be longer than 2GB. > > This claim is nonsense. The variable's value is measured in KB, so the > effective limit is actually 2TB not 2GB. you have true - it works on 9.2. The problem will be somewhere in backport on 9.1 sorry for false alarm Regards Pavel Stěhule > > regards, tom lane
2012/7/20 Pavel Stehule <pavel.stehule@gmail.com>: > 2012/7/20 Tom Lane <tgl@sss.pgh.pa.us>: >> Pavel Stehule <pavel.stehule@gmail.com> writes: >>> I did a backport of temp_file_limit feature to 9.1, but when we tested >>> this patch, we found very restristrictive limit to 2GB. >> >>> 2GB is nonsense, because this is session limit of temp files, and >>> these files should be longer than 2GB. >> >> This claim is nonsense. The variable's value is measured in KB, so the >> effective limit is actually 2TB not 2GB. > > you have true - it works on 9.2. The problem will be somewhere in > backport on 9.1 it works well with 9.1.4, but not with 9.1.3 > > sorry for false alarm > > Regards > > Pavel Stěhule > >> >> regards, tom lane