Thread: timestamp and timestamptz

Niels Jespersen <NJN@dst.dk> writes:

> Hello all
>
>  
>
> We have some data that have entered a timestamp column from a csv. The data in the csv are in utc. We want to access
thedata in
 
> our native timezone (CET). 
>
>  
>
> I am considering a few alternatives: 
>
>  
>
> 1.       Early in the process, convert to timestamptz and keep this datatype. 
>
> 2.       Early in the process, convert to timestamp as understood in CET.  This will imply by convention that the
datain the timestamp
 
> column represents CET. Users will need to be told that data represents CET, even if data is somwhere in the future
keptin another
 
> country in another timezone. 
>
>  
>
> I probably should choose 1 over 2. But I am a bit hesitant, probably because we almost never have used timestamptz. 
>
>  
>
> Can we agree that the below query is selecting both the original utc timestamps and 2 and 1 (as decribed above)?
>
>  
>
> set timezone to 'cet';
>
> select read_time read_time_utc, (read_time at time zone 'utc')::timestamp read_time_cet, (read_time at time zone
'utc')::timestamptz
> read_time_tz from t limit 10;
>
>  
>
> We are on Postgres 12. 
>
>  
>

Keep your life simple - just go with option 1. Keep all timestamps in
UTC and let clients deal with it in whatever way they need to. This will
also help deal with issues associated with daylight savings time (DST
can be a real pain as different locations have it and others don't and
the 'switchover' date is subject to political whims and can change).
Your option 2 will cause all sorts of issues and keep in mind that most
larger countries have multiple timezones, so even if your data is all
associated with a single country, you can potentially have multiple
conversion routines required. On most *nix systems, clock time is UTC as
well, so having everything in UTC really helps when you want to do
diagnosis across your database and system log files etc. 

-- 
Tim Cross

Tim Cross wrote:

> Niels Jespersen <NJN@dst.dk> writes:
> 
> > Hello all
> >
> > We have some data that have entered a timestamp column from a csv. The data in the csv are in utc. We want to
accessthe data in
 
> > our native timezone (CET). 
> >
> > I am considering a few alternatives: 
> >
> > 1.       Early in the process, convert to timestamptz and keep this datatype. 
> >
> > 2.       Early in the process, convert to timestamp as understood in CET.  This will imply by convention that the
datain the timestamp
 
> > column represents CET. Users will need to be told that data represents CET, even if data is somwhere in the future
keptin another
 
> > country in another timezone. 
> >
> > I probably should choose 1 over 2. But I am a bit hesitant, probably because we almost never have used timestamptz.

> >
> > Can we agree that the below query is selecting both the original utc timestamps and 2 and 1 (as decribed above)?
> >
> > set timezone to 'cet';
> >
> > select read_time read_time_utc, (read_time at time zone 'utc')::timestamp read_time_cet, (read_time at time zone
'utc')::timestamptz
> > read_time_tz from t limit 10;
> >
> > We are on Postgres 12. 
> 
> Keep your life simple - just go with option 1. Keep all timestamps in
> UTC and let clients deal with it in whatever way they need to.

That suggestion (keep timestamps in UTC) doesn't sound like option 1 to me
(i.e. convert timestamps-that-are-in-UTC-but-don't-record-the-fact to timestamptz).
Option 1 could be done using any timezone. The point is to record what the
timezone is. It doesn't matter whether it's left as UTC or converted to CET
as long as that decision is recorded in the data by using timestamptz (and
as long as you get postgres to perform the time zone conversion because it
will get it right).

But presumably there is some reason why the data is already in UTC
and there's probably no reason to change that. But its timezone should
be recorded.

> This will
> also help deal with issues associated with daylight savings time (DST
> can be a real pain as different locations have it and others don't and
> the 'switchover' date is subject to political whims and can change).

That's a dim view of the tzdata database which gets updated regularly
to take such political decisions into account. As long as postgres uses
the same tzdata as all good UNIX-based operating systems do, and it's kept
up to date, it should be fine (as long as you never trust what a Windows
host says the timezone is for arbitrary timestamps).

This is required even if you leave data in UTC if it ever needs to be
displayed in any other time zone. Postgres still needs up to date tzdata
to perform the conversions later for users.

> Your option 2 will cause all sorts of issues and keep in mind that most
> larger countries have multiple timezones, so even if your data is all
> associated with a single country, you can potentially have multiple
> conversion routines required. On most *nix systems, clock time is UTC as
> well, so having everything in UTC really helps when you want to do
> diagnosis across your database and system log files etc. 

I don't see much difference in storing a timestamptz in UTC or a timestamptz
in CET. As long as the intended offset from UTC is recorded (which it is
in a timestamptz) it should be fine. If the timestamp is CET, then that fact
is in the data and the user doesn't need to be told it separately. It's obvious
when they see the data because the timezone is part of the data.

And you don't need conversion routines. Postgres can compare timestamptz
values and convert from one time zone to another for display purposes.

However, option 2 seems to be converting the data to CET but not using the
timestamptz datatype. That would be a bad idea. I think timestamptz should
always be preferred to timestamp. Not recording the timezone is where the
problems come from.

Although having a single timezone for log files is a great idea. They hardly
ever include timezone information so keeping everything in the same timezone
is important. Mind you, they hardly ever even include the year. What's with
that? (yes, I'm looking at you /var/log).

> -- 
> Tim Cross

cheers,
raf

David G. Johnston wrote:

> On Wed, Apr 15, 2020 at 4:53 PM raf <raf@raf.org> wrote:
> 
> > I don't see much difference in storing a timestamptz in UTC or a
> > timestamptz
> > in CET. As long as the intended offset from UTC is recorded (which it is
> > in a timestamptz) it should be fine.
> 
> I only really skimmed the entire response but this framing of how
> timestamptz stores results is wrong.  Once you've stored a timestamptz in
> PostgreSQL you no longer have any knowledge of the timezone.  If you truly
> need that you need to record that in a different field.  What you do know
> is that PostgreSQL has a known point-in-time in UTC and can give you back
> the same value expressed in any other timezone according to the rules in
> the timezone database.
> 
> Or, as written verbatim in the documentation:
> """
> For timestamp with time zone, the internally stored value is always in UTC
> (Universal Coordinated Time, traditionally known as Greenwich Mean Time,
> GMT). An input value that has an explicit time zone specified is converted
> to UTC using the appropriate offset for that time zone. If no time zone is
> stated in the input string, then it is assumed to be in the time zone
> indicated by the system's TimeZone parameter, and is converted to UTC using
> the offset for the timezone zone.
> """
> 
> https://www.postgresql.org/docs/12/datatype-datetime.html
> 
> David J.

You are misinterpreting the documentation, or the
documentation is incomplete/misleading at that
location. It doesn't just convert and store the time in
UTC. It stores the time in UTC and it also stores the
offset from UTC as determined by the time zone
specified on input (that section of the documentation
might not mention that fact but it is true
nonetheless).

I store localtimes in the database and always see the
offset when selecting it later. The timezone
information doesn't just disappear as you (or that
piece of documentation) might be suggesting.

If you don't believe me, try this:

  create table example (t timestamptz not null);
  insert into example (t) values (timestamptz '2020-04-16 17:12:33.71768 Australia/Sydney');
  select * from  example;
  drop table example;

When I run that, I see:

  CREATE TABLE
  INSERT 0 1
                t               
  ------------------------------
   2020-04-16 17:12:33.71768+10
  (1 row)
  
  DROP TABLE

So the timezone is stored (or the offset is stored if
you prefer). Had it been daylight savings on that date
in Sydney, the offset would have been "+11". It's all
fine.

cheers,
raf

Steve Baldwin wrote:

> I'm pretty sure you are mistaken. Postgres doesn't store the 'creating'
> time zone in a timestamptz column.
> 
> Try doing this before re-running your test:
> 
> set timezone to 'utc';
> 
> What you are seeing in your test is an artifact of that timezone setting.
> 
> Steve

Thanks. You're right.

  create table example (t timestamptz not null);
  insert into example (t) values (timestamptz '2020-04-16 17:12:33.71768 Australia/Sydney');
  select * from  example;
  set timezone to 'utc';
  select * from  example;
  drop table example;

Does this:

  CREATE TABLE
  INSERT 0 1
              t               
  ------------------------------
   2020-04-16 17:12:33.71768+10
  (1 row)

  SET
                t               
  ------------------------------
   2020-04-16 07:12:33.71768+00
  (1 row)

  DROP TABLE

So it doesn't store the offset, but I've used postgres
for 12 years without knowing that and it hasn't been
a problem. Yay, postgres!

It doesn't store the offset but, by using timestamptz,
it knows that the timezone is UTC. That's what matters.
The fact that it knows the time zone is what makes everything
work. Timestamp without time zone is best avoided I think.

cheers,
raf

On 4/16/20 1:23 AM, raf wrote:
> Steve Baldwin wrote:
> 
>> I'm pretty sure you are mistaken. Postgres doesn't store the 'creating'
>> time zone in a timestamptz column.
>>
>> Try doing this before re-running your test:
>>
>> set timezone to 'utc';
>>
>> What you are seeing in your test is an artifact of that timezone setting.
>>
>> Steve
> 
> Thanks. You're right.
> 
>    create table example (t timestamptz not null);
>    insert into example (t) values (timestamptz '2020-04-16 17:12:33.71768 Australia/Sydney');
>    select * from  example;
>    set timezone to 'utc';
>    select * from  example;
>    drop table example;
> 
> Does this:
> 
>    CREATE TABLE
>    INSERT 0 1
>                t
>    ------------------------------
>     2020-04-16 17:12:33.71768+10
>    (1 row)
> 
>    SET
>                  t
>    ------------------------------
>     2020-04-16 07:12:33.71768+00
>    (1 row)
> 
>    DROP TABLE
> 
> So it doesn't store the offset, but I've used postgres
> for 12 years without knowing that and it hasn't been
> a problem. Yay, postgres!
> 
> It doesn't store the offset but, by using timestamptz,
> it knows that the timezone is UTC. That's what matters.

Well that is somewhat misleading. The value entered is stored as 
timestamp with an implicit tz of 'UTC'. The issue that trips people up 
is the format of the input value. If you enter an input with an offset 
or correct tz value then you have given Postgres an explicit value to 
work off for converting it to 'UTC'. For what is correct see here:

https://www.postgresql.org/docs/12/datatype-datetime.html#DATATYPE-TIMEZONES

If you input a value that does not have the above then Postgres uses the 
SET TimeZone value to implicitly set the input value's tz(as pointed out 
by David Johnson upstream). In other words Postgres does not assume an 
input value is in 'UTC'. For the OP's case that could cause issues if 
the timestamp in the CSV file does not have a proper offset/tz and the 
client is using a tz other then 'UTC'(as pointed by others upstream). 
The bottom line is that when dealing with timestamps explicit is better 
then implicit.

> The fact that it knows the time zone is what makes everything
> work. Timestamp without time zone is best avoided I think.
> 
> cheers,
> raf
> 
> 
> 
> 
> 


-- 
Adrian Klaver
adrian.klaver@aklaver.com

Adrian Klaver wrote:

> On 4/16/20 1:23 AM, raf wrote:
> > Steve Baldwin wrote:
> > 
> > > I'm pretty sure you are mistaken. Postgres doesn't store the 'creating'
> > > time zone in a timestamptz column.
> > > 
> > > Try doing this before re-running your test:
> > > 
> > > set timezone to 'utc';
> > > 
> > > What you are seeing in your test is an artifact of that timezone setting.
> > > 
> > > Steve
> > 
> > Thanks. You're right.
> > 
> >    create table example (t timestamptz not null);
> >    insert into example (t) values (timestamptz '2020-04-16 17:12:33.71768 Australia/Sydney');
> >    select * from  example;
> >    set timezone to 'utc';
> >    select * from  example;
> >    drop table example;
> > 
> > Does this:
> > 
> >    CREATE TABLE
> >    INSERT 0 1
> >                t
> >    ------------------------------
> >     2020-04-16 17:12:33.71768+10
> >    (1 row)
> > 
> >    SET
> >                  t
> >    ------------------------------
> >     2020-04-16 07:12:33.71768+00
> >    (1 row)
> > 
> >    DROP TABLE
> > 
> > So it doesn't store the offset, but I've used postgres
> > for 12 years without knowing that and it hasn't been
> > a problem. Yay, postgres!
> > 
> > It doesn't store the offset but, by using timestamptz,
> > it knows that the timezone is UTC. That's what matters.
> 
> Well that is somewhat misleading. The value entered is stored as timestamp
> with an implicit tz of 'UTC'. The issue that trips people up is the format
> of the input value. If you enter an input with an offset or correct tz value
> then you have given Postgres an explicit value to work off for converting it
> to 'UTC'.

When I said "it knows that the timezone is UTC", I was only referring
to the values once stored as a timestamptz, not the input. Sorry that
wasn't clear enough. I meant to say that once a value is stored in a
timestamptz (as opposed to a timestamp), postgres knows that it is
stored in UTC and will do the right things with it.

> For what is correct see here:
> 
> https://www.postgresql.org/docs/12/datatype-datetime.html#DATATYPE-TIMEZONES
> 
> If you input a value that does not have the above then Postgres uses the SET
> TimeZone value to implicitly set the input value's tz(as pointed out by
> David Johnson upstream). In other words Postgres does not assume an input
> value is in 'UTC'. For the OP's case that could cause issues if the
> timestamp in the CSV file does not have a proper offset/tz and the client is
> using a tz other then 'UTC'(as pointed by others upstream). The bottom line
> is that when dealing with timestamps explicit is better then implicit.

I couldn't agree more.

> > The fact that it knows the time zone is what makes everything
> > work. Timestamp without time zone is best avoided I think.
> > 
> > cheers,
> > raf
> 
> -- 
> Adrian Klaver
> adrian.klaver@aklaver.com
>