Re: Strategies for converting UTC data to local windows for arbitrary resolutions and timezones - Mailing list pgsql-general
Slightly off topic, but has anyone tried TimescaleDB for timeseries databases?
The issues discussed here are still there as they apply to the underlying Postgres ORDBMS.
We solve the problem (around 4 billion records of instrument sensor readings) by using UTC for the "native" timestamp, and working in that. Even though we are ½ way around the world. The local times can easily be determined & applied if desired, but by standardising on the reference time zone at the start, things have "just worked", for around 15 years now.
Brent Wood
Principal Technician, Fisheries
NIWA
DDI: +64 (4) 3860529
Principal Technician, Fisheries
NIWA
DDI: +64 (4) 3860529
From: Lincoln Swaine-Moore <lswainemoore@gmail.com>
Sent: Thursday, October 5, 2023 08:30
To: Alban Hertroys <haramrae@gmail.com>
Cc: Marian Wendt <marian.wendt@yahoo.com>; pgsql-general <pgsql-general@lists.postgresql.org>
Subject: Re: Strategies for converting UTC data to local windows for arbitrary resolutions and timezones
Sent: Thursday, October 5, 2023 08:30
To: Alban Hertroys <haramrae@gmail.com>
Cc: Marian Wendt <marian.wendt@yahoo.com>; pgsql-general <pgsql-general@lists.postgresql.org>
Subject: Re: Strategies for converting UTC data to local windows for arbitrary resolutions and timezones
> What I do in such cases is to add an extra column with the UTC timestamp to serve as a linear scale to the local timestamps. That also helps with ordering buckets in reports and such during DST changes (especially the ones where an hour repeats).
> For hours and quarter hours I found it to be fairly convenient to base a view on a join between a date calendar and an (quarter of an) hour per UTC day table, but materialising that with some indexes may perform better (at the cost of disk space). I do materialise that currently, but our database server doesn’t have a lot of memory so I’m often not hitting the cache and performance suffers a bit (infrastructure is about to change for the better though).
That's an interesting idea, but I'm not sure I fully understand. Assuming you're aggregating data: what do you group by? For instance, at an hourly resolution, if you group by both the UTC timestamp and the local one, you might end up, say, dividing an hour-long bucket in two for time zones with half-hour-based offsets, no?
Thanks for the detailed writeup! Definitely helpful to learn more about what people are using in production to handle this sort of thing.
Lincoln Swaine-Moore
 |
Note: This email is intended solely for the use of the addressee and may contain information that is confidential or subject to legal professional privilege. If you receive this email in error please immediately notify the sender and delete the email.
pgsql-general by date: