Thread: 64-bit sequences

64-bit sequences

From
Paul Caskey
Date:
Hello, newbie here.  I need 64-bit sequences.  I started looking at
sequence.h, sequence.c, and some other files, and I have some questions
before I start mucking up your code.  :-)  (I'd rather do it right and
contribute something to the pgsql project.)

But first, has this issue come up in the past, and was a decision made to
keep sequences simple as an "int4" type only?  Please clue me in on the
history.  Is there any reason not to push forward with "int8" sequences,
as some sort of compile-time or run-time option?


Paul Caskey
New Mexico Software
paul@nmxs.com


Re: 64-bit sequences

From
Tom Lane
Date:
Paul Caskey <paul@nmxs.com> writes:
> But first, has this issue come up in the past, and was a decision made to
> keep sequences simple as an "int4" type only?  Please clue me in on the
> history.  Is there any reason not to push forward with "int8" sequences,
> as some sort of compile-time or run-time option?

Mainly it's that int8 isn't supported on all our platforms.  As a
compile-time option it might be reasonable...
        regards, tom lane


Re: 64-bit sequences

From
Paul Caskey
Date:
Tom Lane wrote:
> 
> Paul Caskey <paul@nmxs.com> writes:
> > But first, has this issue come up in the past, and was a decision made to
> > keep sequences simple as an "int4" type only?  Please clue me in on the
> > history.  Is there any reason not to push forward with "int8" sequences,
> > as some sort of compile-time or run-time option?
> 
> Mainly it's that int8 isn't supported on all our platforms.  As a
> compile-time option it might be reasonable...
> 
>                         regards, tom lane

Okay, cool.  Similar subject: What about making the oid 64-bit?  At first
glance, this seems easier to change than the sequence generator, since you
guys do a good job of using sizeof() and the Oid typedef.  Changing the
typedef to "unsigned long long" should cover everything...?  I will test
it.

The snag I ran into with sequence.c is a missing Int64GetDatum() macro. 
My system is Sun Solaris 7, compiled in 32-bit mode.  So I
HAVE_LONG_LONG_INT_64 but don't HAVE_LONG_INT_64.  I do have the "int8"
SQL datatype and tested it.

In c.h I have these lines:

typedef signed char int8;               /* == 8 bits */
typedef signed short int16;             /* == 16 bits */
typedef signed int int32;               /* == 32 bits */

Is there some reason I'm missing the magic fourth line?  It should be:

typedef signed long long int64;         /* == 64 bits */

Isn't it strange I have the "int8" SQL datatype, but not the "int64" C
typedef and the Int64GetDatum() macro?  I'm not a 64-bit compiling expert,
so go easy on me.   


Paul Caskey
New Mexico Software


Re: 64-bit sequences

From
Thomas Lockhart
Date:
> > Mainly it's that int8 isn't supported on all our platforms.  As a
> > compile-time option it might be reasonable...

Or maybe better, as another type, say SERIAL64? That way both could be
available on some platforms. Also...

> Similar subject: What about making the oid 64-bit?  At first
> glance, this seems easier to change than the sequence generator, since 
> you guys do a good job of using sizeof() and the Oid typedef.  
> Changing the typedef to "unsigned long long" should cover 
> everything...?  I will test it.

Again, a 64bit vs 32 bit issue. We have "pass by value" and "pass by
reference" data types, and we have conventionally made everything bigger
than 32bits a "by reference" type. Going to a 64bit OID or SERIAL type
may mess with that convention, but it may be good to revisit this and
remind ourselves why we have that convention in the first place.

Your other questions are related (I think) to the by-ref vs by-val
issue.
                  - Thomas


Re: 64-bit sequences

From
Tom Lane
Date:
Paul Caskey <paul@nmxs.com> writes:
> Okay, cool.  Similar subject: What about making the oid 64-bit?

Let's just say you'd be opening a *much* larger can of worms there,
because Oid is used all over the place whereas only a few routines
know anything about sequences.

This is (or should be) on the TODO list but I wouldn't recommend it
as your first backend programming project.

> At first glance, this seems easier to change than the sequence
> generator, since you guys do a good job of using sizeof() and the Oid
> typedef.

Except for all the places that assume Oid is interchangeable with int.
Finding them is left as an exercise for the student...
        regards, tom lane


Re: 64-bit sequences

From
Tom Lane
Date:
Thomas Lockhart <lockhart@alumni.caltech.edu> writes:
>> Similar subject: What about making the oid 64-bit?

> Again, a 64bit vs 32 bit issue. We have "pass by value" and "pass by
> reference" data types, and we have conventionally made everything bigger
> than 32bits a "by reference" type. Going to a 64bit OID or SERIAL type
> may mess with that convention, but it may be good to revisit this and
> remind ourselves why we have that convention in the first place.

I think it would be completely impractical to convert Oid to a pass-by-
reference type --- the palloc() overhead would be intolerable.  However,
on machines where it's possible to make Datum a 64-bit integer type,
we could support 64-bit Oids.  On those machines where pointers are 64
bits, there wouldn't even be any performance cost because Datum has to
be 64 bits anyway.

I have actually had something like this in the back of my mind while
working on the fmgr conversion.  With sufficiently disciplined use of
DatumGetFoo and FooGetDatum macros everywhere, it would become fairly
transparent whether Datum is 32 or 64 bits and whether 64-bit types
are pass-by-value or pass-by-reference.  Eventually I'd like to see
compile-time options for the size of Oid and for whether int8,
float4, and float8 are pass-by-val or -by-ref.  There's still a lot
of tedious code-cleanup gruntwork to be done before that can happen,
though.
        regards, tom lane


Re: 64-bit sequences

From
Bruce Momjian
Date:
> Paul Caskey <paul@nmxs.com> writes:
> > Okay, cool.  Similar subject: What about making the oid 64-bit?
> 
> Let's just say you'd be opening a *much* larger can of worms there,
> because Oid is used all over the place whereas only a few routines
> know anything about sequences.
> 
> This is (or should be) on the TODO list but I wouldn't recommend it
> as your first backend programming project.
> 
> > At first glance, this seems easier to change than the sequence
> > generator, since you guys do a good job of using sizeof() and the Oid
> > typedef.
> 
> Except for all the places that assume Oid is interchangeable with int.
> Finding them is left as an exercise for the student...

It would be nice to get OID to act as an unsigned 'int' in all places.

--  Bruce Momjian                        |  http://www.op.net/~candle pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 


Re: 64-bit sequences

From
Tom Lane
Date:
Bruce Momjian <pgman@candle.pha.pa.us> writes:
>> Except for all the places that assume Oid is interchangeable with int.
>> Finding them is left as an exercise for the student...

> It would be nice to get OID to act as an unsigned 'int' in all places.

Actually, I'd like to get rid of the assumption that it has anything
to do with int.  Signed or not is the least of my worries --- I'd like
to be able to equate Oid to long long, for example.
        regards, tom lane


Re: 64-bit sequences

From
Paul Caskey
Date:
Tom Lane wrote:
> 
> Paul Caskey <paul@nmxs.com> writes:
> > Okay, cool.  Similar subject: What about making the oid 64-bit?
> 
> Let's just say you'd be opening a *much* larger can of worms there,
> because Oid is used all over the place whereas only a few routines
> know anything about sequences.

Aha!  Thanks for the warning.

> This is (or should be) on the TODO list but I wouldn't recommend it
> as your first backend programming project.
> 
> > At first glance, this seems easier to change than the sequence
> > generator, since you guys do a good job of using sizeof() and the Oid
> > typedef.
> 
> Except for all the places that assume Oid is interchangeable with int.
> Finding them is left as an exercise for the student...

Again, thanks for the warning.  :-)

Thanks for the comments on this thread.  It sounds too tricky for me to
attempt any type of 64-bit sequence at this time, especially on a 32-bit
platform.  For now, I've made my critical "id" variables type INT8, but
still use an INT4 sequence to increment them.  If/when an INT8 sequence
becomes available in the future, I will drop one in.  Otherwise if I start
getting close to the 2GB limit, I'll find a workaround to reuse holes in
the sequence.

There doesn't seem to be any problem using pgsql's int4-->int8 automatic
conversion in this way.  Hopefully I can also join on int4/int8 values
without any snags or big performance problems, although I haven't tested
that.

-- 
Paul Caskey        paul@nmxs.com        Software Engineer
New Mexico Software    5041 Indian School NE    Albuquerque, NM 87110
--


Re: 64-bit sequences

From
Paul Caskey
Date:
Tom Lane wrote:
> 
> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> >> Except for all the places that assume Oid is interchangeable with int.
> >> Finding them is left as an exercise for the student...
> 
> > It would be nice to get OID to act as an unsigned 'int' in all places.
> 
> Actually, I'd like to get rid of the assumption that it has anything
> to do with int.  Signed or not is the least of my worries --- I'd like
> to be able to equate Oid to long long, for example.

So would I!

> 
>                         regards, tom lane

BTW, the 32-bit oid implies a hard limit of 2 billion records on a single
server, right?  What if that number hits ~2 billion over the course of
many inserts and deletes?  Does it intelligently wrap back to 1 and find
holes in the sequence to re-use?  Otherwise it's more than a limit on
records; it's a limit on inserts.  


-- 
Paul Caskey        paul@nmxs.com        Software Engineer
New Mexico Software    5041 Indian School NE    Albuquerque, NM 87110
--