Thread: Postgres storing time in strange manner

Postgres storing time in strange manner

From
Casey Allen Shobe
Date:
I don't have a clue why it's doing this - has anyone else seen this sort of
behavior, or know why it might be doing it?  It shows times wrong, for
instance, it shows 00:04:60 where it should show 00:05:00.  See below:

gabrielle=# select * from scheduled_query_groups;
 group_id | group_name | group_type_id | interval |   run_at    |
last_run        | exec_upon_completion
----------+------------+---------------+----------+-------------+------------
------------+---------------------- 2 | test       |             1 | 00:04:60

 | 00:00:00-05 | 0001-01-01 04:59:60.00 |

(1 row)

gabrielle=# \d scheduled_query_groups;
                                       Table "scheduled_query_groups"
        Column        |           Type           |
Modifiers
----------------------+--------------------------+---------------------------
------------------------------- group_id             | integer

   | not null default

nextval('sq_groups_group_id_seq'::text)
 group_name           | character varying(32)    | not null
 group_type_id        | integer                  | not null
 interval             | interval                 | not null default '5
minutes'
 run_at               | time with time zone      | not null default '00:00:00
EST'
 last_run             | timestamp with time zone | not null default
'0001-01-01 00:00:00 EST'
 exec_upon_completion | character varying(128)   |
Primary key: scheduled_query_groups_pkey
Unique keys: scheduled_query__group_name_key
Triggers: RI_ConstraintTrigger_16607

--
Casey Allen Shobe / Network Security Analyst & PHP Developer
SecureWorks, Inc. / 404.327.6339 x169 / Fax: 404.728.0144
cshobe@secureworks.net / http://www.secureworks.net
Content is my own and does not necessarily represent my company.

Lost Terminal.

Re: Postgres storing time in strange manner

From
Rod Taylor
Date:
On Sun, 2002-09-15 at 03:57, Casey Allen Shobe wrote:
> I don't have a clue why it's doing this - has anyone else seen this sort of
> behavior, or know why it might be doing it?  It shows times wrong, for
> instance, it shows 00:04:60 where it should show 00:05:00.  See below:

There are actually 61 seconds in some minutes.  In order to accommodate
leap seconds, PostgreSQL allows this to happen -- similarly to how it
will also allow 366 days in some years.

--
  Rod Taylor


Re: Postgres storing time in strange manner

From
Casey Allen Shobe
Date:
On Sunday 15 September 2002 10:11 am, Rod Taylor wrote:
> On Sun, 2002-09-15 at 03:57, Casey Allen Shobe wrote:
> > I don't have a clue why it's doing this - has anyone else seen this sort
> > of behavior, or know why it might be doing it?  It shows times wrong, for
> > instance, it shows 00:04:60 where it should show 00:05:00.  See below:
>
> There are actually 61 seconds in some minutes.  In order to accommodate
> leap seconds, PostgreSQL allows this to happen -- similarly to how it
> will also allow 366 days in some years.

How then, am I supposed to explain to a web interface user that when they just
entered 5:00:00, it's going to sometimes show up to 4:59:60?

I'm entering an exact timestamp, that being 5:00:00.  Regardless of how many
seconds you claim were in the former minute, it should not subtract a second
from my entry, because 5:00:00 by your definition would mean 4:59 and 61
seconds.

--
Casey Allen Shobe / Network Security Analyst & PHP Developer
SecureWorks, Inc. / 404.327.6339 x169 / Fax: 404.728.0144
cshobe@secureworks.net / http://www.secureworks.net
Content is my own and does not necessarily represent my company.

Dance like nobody's watching.

Re: Postgres storing time in strange manner

From
Tom Lane
Date:
> On Sunday 15 September 2002 10:11 am, Rod Taylor wrote:
>> On Sun, 2002-09-15 at 03:57, Casey Allen Shobe wrote:
>>> I don't have a clue why it's doing this - has anyone else seen this sort
>>> of behavior, or know why it might be doing it?  It shows times wrong, for
>>> instance, it shows 00:04:60 where it should show 00:05:00.  See below:
>>
>> There are actually 61 seconds in some minutes.  In order to accommodate
>> leap seconds, PostgreSQL allows this to happen -- similarly to how it
>> will also allow 366 days in some years.

True but irrelevant -- PG does not do accounting for leap seconds.

The roundoff bug Casey is looking at is fixed in recent PG versions;
I'd recommend an update to 7.2.2.

            regards, tom lane

Re: Postgres storing time in strange manner

From
Rod Taylor
Date:
> How then, am I supposed to explain to a web interface user that when they just
> entered 5:00:00, it's going to sometimes show up to 4:59:60?

Oh, I see.  I thought you were doing some interval math on it.  Yes,
your right, it should be taken as being 5:00:00.

> I'm entering an exact timestamp, that being 5:00:00.  Regardless of how many
> seconds you claim were in the former minute, it should not subtract a second
> from my entry, because 5:00:00 by your definition would mean 4:59 and 61
> seconds.

Either way, I've been unable to reproduce it with either 7.2 or 7.3 --
nor do I see any notes about that feature having been removed or carried
over to current releases -- no regression tests for it in 7.3.

Could you provide a complete test case, or confirm that it does what you
expect in 7.3?

7.2.2:
iqdb=# select '0001-01-01 4:59:60'::timestamptz;
ERROR:  Bad timestamp external representation '0001-01-01 4:59:60'


--
  Rod Taylor

Re: Postgres storing time in strange manner

From
Rod Taylor
Date:
On Sun, 2002-09-15 at 10:32, Tom Lane wrote:
> > On Sunday 15 September 2002 10:11 am, Rod Taylor wrote:
> >> On Sun, 2002-09-15 at 03:57, Casey Allen Shobe wrote:
> >>> I don't have a clue why it's doing this - has anyone else seen this sort
> >>> of behavior, or know why it might be doing it?  It shows times wrong, for
> >>> instance, it shows 00:04:60 where it should show 00:05:00.  See below:
> >>
> >> There are actually 61 seconds in some minutes.  In order to accommodate
> >> leap seconds, PostgreSQL allows this to happen -- similarly to how it
> >> will also allow 366 days in some years.
>
> True but irrelevant -- PG does not do accounting for leap seconds.

For some reason I thought it did accommodate it when I was still using
7.1.

Ahh well, if the bug is fixed, then it's all good.

--
  Rod Taylor


Re: Postgres storing time in strange manner

From
Casey Allen Shobe
Date:
On Sunday 15 September 2002 10:32 am, Tom Lane wrote:
> The roundoff bug Casey is looking at is fixed in recent PG versions;
> I'd recommend an update to 7.2.2.

I'm running a freshly compiled version of 7.2.2

--
Casey Allen Shobe / Network Security Analyst & PHP Developer
SecureWorks, Inc. / 404.327.6339 x169 / Fax: 404.728.0144
cshobe@secureworks.net / http://www.secureworks.net
Content is my own and does not necessarily represent my company.

Dance like nobody's watching.

Re: Postgres storing time in strange manner

From
Tom Lane
Date:
Casey Allen Shobe <cshobe@secureworks.net> writes:
> On Sunday 15 September 2002 10:32 am, Tom Lane wrote:
>> The roundoff bug Casey is looking at is fixed in recent PG versions;
>> I'd recommend an update to 7.2.2.

> I'm running a freshly compiled version of 7.2.2

Oh?  On what platform?

            regards, tom lane

Re: Postgres storing time in strange manner

From
Tom Lane
Date:
>> The kernel version is probably not relevant here; what's more
>> interesting is the compiler version, compiler optimization level,
>> and perhaps libc version.

> cshobe@gabrielle:/www/htdocs$ gcc -v
> Reading specs from /usr/lib/gcc-lib/i686-pc-linux-gnu/2.95.3/specs
> gcc version 2.95.3 20010315 (release)
> cshobe@gabrielle:/www/htdocs$ echo $CFLAGS
> -O3 -march=i686 -mcpu=i686 -funroll-loops -ffast-math

Bingo: it's the -ffast-math option that's causing the problem.
See, eg,
http://fts.postgresql.org/db/mw/msg.html?mid=1277557

If you rebuild without that, I think you'll find the problem goes away.

ISTM that at one time we had considered actively discriminating against
-ffast-math in CFLAGS in configure ... but I don't see any sign in
current sources that we make any attempt to remove -ffast-math from
environment-supplied CFLAGS.  Peter, do you think that would be a
reasonable thing to do?

            regards, tom lane

Re: Postgres storing time in strange manner

From
Casey Allen Shobe
Date:
On Sunday 15 September 2002 02:14 pm, Tom Lane wrote:
> Bingo: it's the -ffast-math option that's causing the problem.
> See, eg,
> http://fts.postgresql.org/db/mw/msg.html?mid=3D1277557

Thank you, I hadn't realized that they shouldn't be used together...I've us=
ed=20
them for every package on my box :\...ah well, live and learn.

--=20
Casey Allen Shobe / Network Security Analyst & PHP Developer
SecureWorks, Inc. / 404.327.6339 x169 / Fax: 404.728.0144
cshobe@secureworks.net / http://www.secureworks.net
Content is my own and does not necessarily represent my company.

Dance like nobody's watching.

Re: [NOVICE] Postgres storing time in strange manner

From
David Lloyd
Date:
Casey,

> > There are actually 61 seconds in some minutes.  In order to accommodate
> > leap seconds, PostgreSQL allows this to happen -- similarly to how it
> > will also allow 366 days in some years.
>
> How then, am I supposed to explain to a web interface user that when they just
> entered 5:00:00, it's going to sometimes show up to 4:59:60?

Leap minutes? Oh please. I'm gonna have to account for green martians
next...

DSL
--
Con te partiro, su navi per mari
 Che io lo so, no, no non esistono piu
Con te io li vivro.
 (Sartori F, Quarantotto E)

Re: Postgres storing time in strange manner

From
Peter Eisentraut
Date:
Tom Lane writes:

> ISTM that at one time we had considered actively discriminating against
> -ffast-math in CFLAGS in configure ... but I don't see any sign in
> current sources that we make any attempt to remove -ffast-math from
> environment-supplied CFLAGS.  Peter, do you think that would be a
> reasonable thing to do?

Would it be possible to write a short test case that exhibits this
behavior?

--
Peter Eisentraut   peter_e@gmx.net

Re: Postgres storing time in strange manner

From
Tom Lane
Date:
Peter Eisentraut <peter_e@gmx.net> writes:
> Tom Lane writes:
>> ISTM that at one time we had considered actively discriminating against
>> -ffast-math in CFLAGS in configure ... but I don't see any sign in
>> current sources that we make any attempt to remove -ffast-math from
>> environment-supplied CFLAGS.  Peter, do you think that would be a
>> reasonable thing to do?

> Would it be possible to write a short test case that exhibits this
> behavior?

Not sure; apparently it depends on optimization level, so I'd be
hesitant to assume that any short test case would reliably expose
the problem.  Also, aren't you trying to avoid run-time tests in
configure?

But if you prefer a run-time test, I'll see if I can cons one up.

            regards, tom lane

Re: Postgres storing time in strange manner

From
Peter Eisentraut
Date:
Tom Lane writes:

> Not sure; apparently it depends on optimization level, so I'd be
> hesitant to assume that any short test case would reliably expose
> the problem.  Also, aren't you trying to avoid run-time tests in
> configure?

If the test doesn't use any library function's run-time behavior, you can
usually do something like

main() {
int a[(2.0+2.0==4.0)?1:-1]
}

This will fail to compile if the floating-point arithmetic is broken.

Otherwise a good solution might be to print a warning if configure detects
the flag.  Or we can strip it out unconditionally, but that seems wrong.
The user should be made aware of the problem.

--
Peter Eisentraut   peter_e@gmx.net

Re: Postgres storing time in strange manner

From
Tom Lane
Date:
Peter Eisentraut <peter_e@gmx.net> writes:
> If the test doesn't use any library function's run-time behavior, you can
> usually do something like

> main() {
> int a[(2.0+2.0==4.0)?1:-1]
> }

> This will fail to compile if the floating-point arithmetic is broken.

However, unless gcc itself is compiled with -ffast-math, such an
approach won't show up the bug.

I had success with this test:

#include <stdio.h>

double d18000 = 18000.0;

main() {
  int d = d18000 / 3600;
  printf("18000.0 / 3600 = %d\n", d);
  return 0;
}

Using Red Hat 7.2's compiler:

[tgl@rh1 tgl]$ gcc -v
Reading specs from /usr/lib/gcc-lib/i386-redhat-linux/2.96/specs
gcc version 2.96 20000731 (Red Hat Linux 7.1 2.96-98)

I get:

[tgl@rh1 tgl]$ gcc bug.c
[tgl@rh1 tgl]$ ./a.out
18000.0 / 3600 = 5            -- right
[tgl@rh1 tgl]$ gcc -ffast-math bug.c
[tgl@rh1 tgl]$ ./a.out
18000.0 / 3600 = 4            -- wrong!

You need the dummy global variable to keep the compiler from simplifying
the division at compile time, else you get 5.  With the test as
exhibited, the -O level seems not to matter.

            regards, tom lane

Re: Postgres storing time in strange manner

From
Tom Lane
Date:
I said:
> I had success with this test:

BTW, some digging in the assembly code shows that the actual problem is
this: instead of emitting "x / 3600.0", with -ffast-math the compiler
emits the equivalent of "x * (double) (1.0 / 3600.0)".  It's the
last-bit inaccuracy of the latter constant that's killing us.

            regards, tom lane

Re: [NOVICE] Postgres storing time in strange manner

From
Tom Lane
Date:
Ron Johnson <ron.l.johnson@cox.net> writes:
> Out of curiosity: why does -ffast-math break the datetime rounding code?

We dug into this last night, and it turns out that the culprit is code
like

    int hour = time / 3600;

where time is a double.  This yields an exact result when done
correctly, but with -ffast-math gcc will "improve" it to

    int hour = time * 0.000277777777777778;

the constant being the nearest double value to 1.0 / 3600.0.  The
problem is that the constant is inexact and in fact is slightly too
large; so for example if time is exactly 18000.0, you get a resulting
hour value of 4, not 5, after truncation to integer.  Repeated a couple
more times, what should have been 5:00:00 comes out as 4:59:60 ...

            regards, tom lane

Re: [NOVICE] Postgres storing time in strange manner

From
Bruce Momjian
Date:
Tom Lane wrote:
> Ron Johnson <ron.l.johnson@cox.net> writes:
> > Out of curiosity: why does -ffast-math break the datetime rounding code?
>
> We dug into this last night, and it turns out that the culprit is code
> like
>
>     int hour = time / 3600;
>
> where time is a double.  This yields an exact result when done
> correctly, but with -ffast-math gcc will "improve" it to
>
>     int hour = time * 0.000277777777777778;
>
> the constant being the nearest double value to 1.0 / 3600.0.  The
> problem is that the constant is inexact and in fact is slightly too
> large; so for example if time is exactly 18000.0, you get a resulting
> hour value of 4, not 5, after truncation to integer.  Repeated a couple
> more times, what should have been 5:00:00 comes out as 4:59:60 ...

Hard to imagine why anyone would want such an optimization.  How much
faster could it possibly be?  I guess if you were doing only complex
math approximations, it would be a win, but that isn't really proper for
a database.

--
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Re: [NOVICE] Postgres storing time in strange manner

From
Tom Lane
Date:
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> Tom Lane wrote:
>> with -ffast-math gcc will "improve" it to
>>
>> int hour = time * 0.000277777777777778;

> Hard to imagine why anyone would want such an optimization.  How much
> faster could it possibly be?

Back in ye bad olde days, there was probably an order-of-magnitude
difference between the speed of a float multiply and that of a float
divide; so this used to be a pretty standard sort of optimization.
I can remember doing the equivalent thing by hand in source code.

On modern hardware I doubt it makes much difference...

            regards, tom lane

Re: [NOVICE] Postgres storing time in strange manner

From
Bruce Momjian
Date:
Tom Lane wrote:
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > Tom Lane wrote:
> >> with -ffast-math gcc will "improve" it to
> >>
> >> int hour = time * 0.000277777777777778;
>
> > Hard to imagine why anyone would want such an optimization.  How much
> > faster could it possibly be?
>
> Back in ye bad olde days, there was probably an order-of-magnitude
> difference between the speed of a float multiply and that of a float
> divide; so this used to be a pretty standard sort of optimization.
> I can remember doing the equivalent thing by hand in source code.
>
> On modern hardware I doubt it makes much difference...

And you would have to do a heck of a lot of them to see a difference.

--
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Re: [NOVICE] Postgres storing time in strange manner

From
Sean Chittenden
Date:
> > Out of curiosity: why does -ffast-math break the datetime rounding code?

What code bits is this for?  Is there a place where -fno-fast-math
could be used as a CC option if the CC is gcc?  After looking through
gcc, using -O and -ffast-math will create broken code, but -O2
-ffast-math _should_ be okay.  If it's not, then -O2 -fno-fast-math is
likely the correct work around for GCC.  -sc

--
Sean Chittenden

Re: [NOVICE] Postgres storing time in strange manner

From
Tom Lane
Date:
Sean Chittenden <sean@chittenden.org> writes:
> Is there a place where -fno-fast-math
> could be used as a CC option if the CC is gcc?

configure is what I had in mind ;-).  I can't think of any part of the
code where we'd really want this sort of optimization enabled.

> After looking through gcc, using -O and -ffast-math will create broken
> code, but -O2 -ffast-math _should_ be okay.

At least in the gcc shipped with Red Hat 7.2, it doesn't seem to matter:
you get the wrong answer regardless of -O level.  Here's the test case
I used:

[tgl@rh1 tgl]$ cat bug.c
#include <stdio.h>

double d18000 = 18000.0;

main() {
  int d = d18000 / 3600;
  printf("18000.0 / 3600 = %d\n", d);
  return 0;
}
[tgl@rh1 tgl]$ gcc  bug.c
[tgl@rh1 tgl]$ ./a.out
18000.0 / 3600 = 5                -- right
[tgl@rh1 tgl]$ gcc -O2 -ffast-math bug.c
[tgl@rh1 tgl]$ ./a.out
18000.0 / 3600 = 4                -- wrong
-- I get 4 if -ffast-math, -O doesn't affect it
[tgl@rh1 tgl]$ gcc -v
Reading specs from /usr/lib/gcc-lib/i386-redhat-linux/2.96/specs
gcc version 2.96 20000731 (Red Hat Linux 7.1 2.96-98)

            regards, tom lane

Re: [NOVICE] Postgres storing time in strange manner

From
Sean Chittenden
Date:
> > After looking through gcc, using -O and -ffast-math will create broken
> > code, but -O2 -ffast-math _should_ be okay.
>
> At least in the gcc shipped with Red Hat 7.2, it doesn't seem to matter:
> you get the wrong answer regardless of -O level.  Here's the test case
> I used:
>
> [tgl@rh1 tgl]$ cat bug.c
> #include <stdio.h>
>
> double d18000 = 18000.0;
>
> main() {
>   int d = d18000 / 3600;
>   printf("18000.0 / 3600 = %d\n", d);
>   return 0;
> }
> [tgl@rh1 tgl]$ gcc  bug.c
> [tgl@rh1 tgl]$ ./a.out
> 18000.0 / 3600 = 5                -- right
> [tgl@rh1 tgl]$ gcc -O2 -ffast-math bug.c
> [tgl@rh1 tgl]$ ./a.out
> 18000.0 / 3600 = 4                -- wrong
> -- I get 4 if -ffast-math, -O doesn't affect it
> [tgl@rh1 tgl]$ gcc -v
> Reading specs from /usr/lib/gcc-lib/i386-redhat-linux/2.96/specs
> gcc version 2.96 20000731 (Red Hat Linux 7.1 2.96-98)

Heh, chalk this one up as another Linux-ism then 'cause it's not
present in FreeBSD -stable or -current.  This actually makes me feel
better about setting an option in the -devel port for turning on
compilation with -O3.  -sc

stable$ gcc -v
Using builtin specs.
gcc version 2.95.4 20020320 [FreeBSD]

current$ gcc -v
Using built-in specs.
Configured with: FreeBSD/i386 system compiler
Thread model: posix
gcc version 3.2.1 [FreeBSD] 20020901 (prerelease)

--
Sean Chittenden

Re: [NOVICE] Postgres storing time in strange manner

From
Peter Eisentraut
Date:
Tom Lane writes:

> Sean Chittenden <sean@chittenden.org> writes:
> > Is there a place where -fno-fast-math
> > could be used as a CC option if the CC is gcc?
>
> configure is what I had in mind ;-).  I can't think of any part of the
> code where we'd really want this sort of optimization enabled.

Today I read that __FAST_MATH__ is defined if -ffast-math is used, so it
should be easy to write a test in configure.

--
Peter Eisentraut   peter_e@gmx.net