Thread: 64-bit sequences
Hello, newbie here. I need 64-bit sequences. I started looking at sequence.h, sequence.c, and some other files, and I have some questions before I start mucking up your code. :-) (I'd rather do it right and contribute something to the pgsql project.) But first, has this issue come up in the past, and was a decision made to keep sequences simple as an "int4" type only? Please clue me in on the history. Is there any reason not to push forward with "int8" sequences, as some sort of compile-time or run-time option? Paul Caskey New Mexico Software paul@nmxs.com
Paul Caskey <paul@nmxs.com> writes:
> But first, has this issue come up in the past, and was a decision made to
> keep sequences simple as an "int4" type only? Please clue me in on the
> history. Is there any reason not to push forward with "int8" sequences,
> as some sort of compile-time or run-time option?
Mainly it's that int8 isn't supported on all our platforms. As a
compile-time option it might be reasonable...
regards, tom lane
Tom Lane wrote: > > Paul Caskey <paul@nmxs.com> writes: > > But first, has this issue come up in the past, and was a decision made to > > keep sequences simple as an "int4" type only? Please clue me in on the > > history. Is there any reason not to push forward with "int8" sequences, > > as some sort of compile-time or run-time option? > > Mainly it's that int8 isn't supported on all our platforms. As a > compile-time option it might be reasonable... > > regards, tom lane Okay, cool. Similar subject: What about making the oid 64-bit? At first glance, this seems easier to change than the sequence generator, since you guys do a good job of using sizeof() and the Oid typedef. Changing the typedef to "unsigned long long" should cover everything...? I will test it. The snag I ran into with sequence.c is a missing Int64GetDatum() macro. My system is Sun Solaris 7, compiled in 32-bit mode. So I HAVE_LONG_LONG_INT_64 but don't HAVE_LONG_INT_64. I do have the "int8" SQL datatype and tested it. In c.h I have these lines: typedef signed char int8; /* == 8 bits */ typedef signed short int16; /* == 16 bits */ typedef signed int int32; /* == 32 bits */ Is there some reason I'm missing the magic fourth line? It should be: typedef signed long long int64; /* == 64 bits */ Isn't it strange I have the "int8" SQL datatype, but not the "int64" C typedef and the Int64GetDatum() macro? I'm not a 64-bit compiling expert, so go easy on me. Paul Caskey New Mexico Software
> > Mainly it's that int8 isn't supported on all our platforms. As a
> > compile-time option it might be reasonable...
Or maybe better, as another type, say SERIAL64? That way both could be
available on some platforms. Also...
> Similar subject: What about making the oid 64-bit? At first
> glance, this seems easier to change than the sequence generator, since
> you guys do a good job of using sizeof() and the Oid typedef.
> Changing the typedef to "unsigned long long" should cover
> everything...? I will test it.
Again, a 64bit vs 32 bit issue. We have "pass by value" and "pass by
reference" data types, and we have conventionally made everything bigger
than 32bits a "by reference" type. Going to a 64bit OID or SERIAL type
may mess with that convention, but it may be good to revisit this and
remind ourselves why we have that convention in the first place.
Your other questions are related (I think) to the by-ref vs by-val
issue.
- Thomas
Paul Caskey <paul@nmxs.com> writes:
> Okay, cool. Similar subject: What about making the oid 64-bit?
Let's just say you'd be opening a *much* larger can of worms there,
because Oid is used all over the place whereas only a few routines
know anything about sequences.
This is (or should be) on the TODO list but I wouldn't recommend it
as your first backend programming project.
> At first glance, this seems easier to change than the sequence
> generator, since you guys do a good job of using sizeof() and the Oid
> typedef.
Except for all the places that assume Oid is interchangeable with int.
Finding them is left as an exercise for the student...
regards, tom lane
Thomas Lockhart <lockhart@alumni.caltech.edu> writes:
>> Similar subject: What about making the oid 64-bit?
> Again, a 64bit vs 32 bit issue. We have "pass by value" and "pass by
> reference" data types, and we have conventionally made everything bigger
> than 32bits a "by reference" type. Going to a 64bit OID or SERIAL type
> may mess with that convention, but it may be good to revisit this and
> remind ourselves why we have that convention in the first place.
I think it would be completely impractical to convert Oid to a pass-by-
reference type --- the palloc() overhead would be intolerable. However,
on machines where it's possible to make Datum a 64-bit integer type,
we could support 64-bit Oids. On those machines where pointers are 64
bits, there wouldn't even be any performance cost because Datum has to
be 64 bits anyway.
I have actually had something like this in the back of my mind while
working on the fmgr conversion. With sufficiently disciplined use of
DatumGetFoo and FooGetDatum macros everywhere, it would become fairly
transparent whether Datum is 32 or 64 bits and whether 64-bit types
are pass-by-value or pass-by-reference. Eventually I'd like to see
compile-time options for the size of Oid and for whether int8,
float4, and float8 are pass-by-val or -by-ref. There's still a lot
of tedious code-cleanup gruntwork to be done before that can happen,
though.
regards, tom lane
> Paul Caskey <paul@nmxs.com> writes: > > Okay, cool. Similar subject: What about making the oid 64-bit? > > Let's just say you'd be opening a *much* larger can of worms there, > because Oid is used all over the place whereas only a few routines > know anything about sequences. > > This is (or should be) on the TODO list but I wouldn't recommend it > as your first backend programming project. > > > At first glance, this seems easier to change than the sequence > > generator, since you guys do a good job of using sizeof() and the Oid > > typedef. > > Except for all the places that assume Oid is interchangeable with int. > Finding them is left as an exercise for the student... It would be nice to get OID to act as an unsigned 'int' in all places. -- Bruce Momjian | http://www.op.net/~candle pgman@candle.pha.pa.us | (610) 853-3000+ If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup. | Drexel Hill, Pennsylvania19026
Bruce Momjian <pgman@candle.pha.pa.us> writes:
>> Except for all the places that assume Oid is interchangeable with int.
>> Finding them is left as an exercise for the student...
> It would be nice to get OID to act as an unsigned 'int' in all places.
Actually, I'd like to get rid of the assumption that it has anything
to do with int. Signed or not is the least of my worries --- I'd like
to be able to equate Oid to long long, for example.
regards, tom lane
Tom Lane wrote: > > Paul Caskey <paul@nmxs.com> writes: > > Okay, cool. Similar subject: What about making the oid 64-bit? > > Let's just say you'd be opening a *much* larger can of worms there, > because Oid is used all over the place whereas only a few routines > know anything about sequences. Aha! Thanks for the warning. > This is (or should be) on the TODO list but I wouldn't recommend it > as your first backend programming project. > > > At first glance, this seems easier to change than the sequence > > generator, since you guys do a good job of using sizeof() and the Oid > > typedef. > > Except for all the places that assume Oid is interchangeable with int. > Finding them is left as an exercise for the student... Again, thanks for the warning. :-) Thanks for the comments on this thread. It sounds too tricky for me to attempt any type of 64-bit sequence at this time, especially on a 32-bit platform. For now, I've made my critical "id" variables type INT8, but still use an INT4 sequence to increment them. If/when an INT8 sequence becomes available in the future, I will drop one in. Otherwise if I start getting close to the 2GB limit, I'll find a workaround to reuse holes in the sequence. There doesn't seem to be any problem using pgsql's int4-->int8 automatic conversion in this way. Hopefully I can also join on int4/int8 values without any snags or big performance problems, although I haven't tested that. -- Paul Caskey paul@nmxs.com Software Engineer New Mexico Software 5041 Indian School NE Albuquerque, NM 87110 --
Tom Lane wrote: > > Bruce Momjian <pgman@candle.pha.pa.us> writes: > >> Except for all the places that assume Oid is interchangeable with int. > >> Finding them is left as an exercise for the student... > > > It would be nice to get OID to act as an unsigned 'int' in all places. > > Actually, I'd like to get rid of the assumption that it has anything > to do with int. Signed or not is the least of my worries --- I'd like > to be able to equate Oid to long long, for example. So would I! > > regards, tom lane BTW, the 32-bit oid implies a hard limit of 2 billion records on a single server, right? What if that number hits ~2 billion over the course of many inserts and deletes? Does it intelligently wrap back to 1 and find holes in the sequence to re-use? Otherwise it's more than a limit on records; it's a limit on inserts. -- Paul Caskey paul@nmxs.com Software Engineer New Mexico Software 5041 Indian School NE Albuquerque, NM 87110 --