Thread: [Fwd: Re: [PATCHES] 64-bit CommandIds]

[Fwd: Re: [PATCHES] 64-bit CommandIds]

From

Zoltan Boszormenyi

Date:

10 March 2008, 16:17:36

Hi,

what's your opinion on this?
I saw response only from Alvaro on the -patches list.

Thanks in advance,
Zoltán Böszörményi

-------- Eredeti üzenet --------
Tárgy:     Re: [PATCHES] 64-bit CommandIds
Dátum:     Tue, 04 Mar 2008 21:52:25 +0100
Feladó:     Zoltan Boszormenyi <zb@cybertec.at>
Címzett:     pgsql-patches <pgsql-patches@postgresql.org>
CC:     Alvaro Herrera <alvherre@commandprompt.com>, Hans-Juergen Schoenig
<hs@cybertec.at>
Hivatkozások:     <47CD8665.7070903@cybertec.at>
<20080304174110.GK4755@alvh.no-ip.org>



Alvaro Herrera írta:
> Zoltan Boszormenyi wrote:
>
>
>> attached is our patch against HEAD which enables extending CommandIds
>> to 64-bit. This is for enabling long transactions that really do that much
>> non-read-only work in one transaction.
>>
>
> I think you should add a pg_control field and corresponding check, to
> avoid a 64bit-Cid postmaster to start on a 32bit-Cid data area and vice
> versa.
>

I added the check but I needed to add it BEFORE checking for
toast_max_chunk_size otherwise it complained about this more
cryptic problem. I think it's cleaner to report this failure to know
why toast_max_chunk_size != TOAST_MAX_CHUNK_SIZE.

Best regards,
Zoltán Böszörményi

--
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
http://www.postgresql.at/




--
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
http://www.postgresql.at/

diff -dcrpN pgsql.orig/configure pgsql-cid64/configure
*** pgsql.orig/configure    2008-03-02 13:44:42.000000000 +0100
--- pgsql-cid64/configure    2008-03-04 16:53:46.000000000 +0100
*************** if test -n "$ac_init_help"; then
*** 1349,1354 ****
--- 1349,1355 ----
  Optional Features:
    --disable-FEATURE       do not include FEATURE (same as --enable-FEATURE=no)
    --enable-FEATURE[=ARG]  include FEATURE [ARG=yes]
+   --enable-huge-commandid    enable 64-bit CommandId support
    --enable-integer-datetimes  enable 64-bit integer date/time support
    --enable-nls[=LANGUAGES]  enable Native Language Support
    --disable-shared        do not build shared libraries
*************** fi
*** 2175,2180 ****
--- 2176,2219 ----


  #
+ # 64-bit CommandId
+ #
+ echo "$as_me:$LINENO: checking whether to build with 64-bit CommandId support" >&5
+ echo $ECHO_N "checking whether to build with 64-bit CommandId support... $ECHO_C" >&6
+
+ pgac_args="$pgac_args enable_huge_commandid"
+
+ # Check whether --enable-huge-commandid or --disable-huge-commandid was given.
+ if test "${enable_huge_commandid+set}" = set; then
+   enableval="$enable_huge_commandid"
+
+   case $enableval in
+     yes)
+
+ cat >>confdefs.h <<\_ACEOF
+ #define USE_64BIT_COMMANDID 1
+ _ACEOF
+
+       ;;
+     no)
+       :
+       ;;
+     *)
+       { { echo "$as_me:$LINENO: error: no argument expected for --enable-huge-commandid option" >&5
+ echo "$as_me: error: no argument expected for --enable-huge-commandid option" >&2;}
+    { (exit 1); exit 1; }; }
+       ;;
+   esac
+
+ else
+   enable_huge_commandid=no
+
+ fi;
+
+ echo "$as_me:$LINENO: result: $enable_huge_commandid" >&5
+ echo "${ECHO_T}$enable_huge_commandid" >&6
+
+ #
  # 64-bit integer date/time storage (--enable-integer-datetimes)
  #
  { echo "$as_me:$LINENO: checking whether to build with 64-bit integer date/time support" >&5
diff -dcrpN pgsql.orig/configure.in pgsql-cid64/configure.in
*** pgsql.orig/configure.in    2008-03-02 13:44:43.000000000 +0100
--- pgsql-cid64/configure.in    2008-03-04 16:53:46.000000000 +0100
*************** PGAC_ARG_REQ(with, libs,      [  --with-
*** 128,133 ****
--- 128,142 ----


  #
+ # 64-bit CommandId
+ #
+ AC_MSG_CHECKING([whether to build with 64-bit CommandId support])
+ PGAC_ARG_BOOL(enable, huge-commandid, no, [  --enable-huge-commandid    enable 64-bit CommandId support],
+         [AC_DEFINE([USE_64BIT_COMMANDID], 1,
+             [Define to 1 if you want 64-bit CommandId support. (--enable-huge-commandid)])])
+ AC_MSG_RESULT([$enable_huge_commandid])
+
+ #
  # 64-bit integer date/time storage (--enable-integer-datetimes)
  #
  AC_MSG_CHECKING([whether to build with 64-bit integer date/time support])
diff -dcrpN pgsql.orig/doc/src/sgml/installation.sgml pgsql-cid64/doc/src/sgml/installation.sgml
*** pgsql.orig/doc/src/sgml/installation.sgml    2008-02-18 13:49:58.000000000 +0100
--- pgsql-cid64/doc/src/sgml/installation.sgml    2008-03-04 17:16:14.000000000 +0100
*************** su - postgres
*** 1011,1016 ****
--- 1011,1027 ----
        </varlistentry>

        <varlistentry>
+        <term><option>--enable-huge-commandid</option></term>
+        <listitem>
+         <para>
+          Use 64-bit CommandIds if you are planning to run transactions
+          consisting of more than 4 billion commands.  This is off by default
+          to save disk space.
+         </para>
+        </listitem>
+       </varlistentry>
+
+       <varlistentry>
         <term><option>--enable-integer-datetimes</option></term>
         <listitem>
          <para>
diff -dcrpN pgsql.orig/src/backend/access/transam/xact.c pgsql-cid64/src/backend/access/transam/xact.c
*** pgsql.orig/src/backend/access/transam/xact.c    2008-01-15 19:56:59.000000000 +0100
--- pgsql-cid64/src/backend/access/transam/xact.c    2008-03-04 16:57:54.000000000 +0100
*************** CommandCounterIncrement(void)
*** 592,598 ****
--- 592,602 ----
              currentCommandId -= 1;
              ereport(ERROR,
                      (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
+ #ifdef USE_64BIT_COMMANDID
+           errmsg("cannot have more than 2^64-1 commands in a transaction")));
+ #else
            errmsg("cannot have more than 2^32-1 commands in a transaction")));
+ #endif
          }
          currentCommandIdUsed = false;

diff -dcrpN pgsql.orig/src/backend/access/transam/xlog.c pgsql-cid64/src/backend/access/transam/xlog.c
*** pgsql.orig/src/backend/access/transam/xlog.c    2008-02-18 13:50:00.000000000 +0100
--- pgsql-cid64/src/backend/access/transam/xlog.c    2008-03-04 21:39:30.000000000 +0100
*************** WriteControlFile(void)
*** 3794,3799 ****
--- 3794,3805 ----
      ControlFile->enableIntTimes = FALSE;
  #endif

+ #ifdef USE_64BIT_COMMANDID
+     ControlFile->enable64bitCommandId = TRUE;
+ #else
+     ControlFile->enable64bitCommandId = FALSE;
+ #endif
+
      ControlFile->localeBuflen = LOCALE_NAME_BUFLEN;
      localeptr = setlocale(LC_COLLATE, NULL);
      if (!localeptr)
*************** ReadControlFile(void)
*** 3989,3994 ****
--- 3995,4017 ----
                        " but the server was compiled with INDEX_MAX_KEYS %d.",
                             ControlFile->indexMaxKeys, INDEX_MAX_KEYS),
                   errhint("It looks like you need to recompile or initdb.")));
+
+ #ifdef USE_64BIT_COMMANDID
+     if (ControlFile->enable64bitCommandId != TRUE)
+         ereport(FATAL,
+                 (errmsg("database files are incompatible with server"),
+                  errdetail("The database cluster was initialized without USE_64BIT_COMMANDID"
+                   " but the server was compiled with USE_64BIT_COMMANDID."),
+                  errhint("It looks like you need to recompile or initdb.")));
+ #else
+     if (ControlFile->enable64bitCommandId != FALSE)
+         ereport(FATAL,
+                 (errmsg("database files are incompatible with server"),
+                  errdetail("The database cluster was initialized with USE_64BIT_COMMANDID"
+                " but the server was compiled without USE_64BIT_COMMANDID."),
+                  errhint("It looks like you need to recompile or initdb.")));
+ #endif
+
      if (ControlFile->toast_max_chunk_size != TOAST_MAX_CHUNK_SIZE)
          ereport(FATAL,
                  (errmsg("database files are incompatible with server"),
diff -dcrpN pgsql.orig/src/backend/bootstrap/bootstrap.c pgsql-cid64/src/backend/bootstrap/bootstrap.c
*** pgsql.orig/src/backend/bootstrap/bootstrap.c    2008-02-18 13:50:00.000000000 +0100
--- pgsql-cid64/src/backend/bootstrap/bootstrap.c    2008-03-04 16:53:46.000000000 +0100
*************** static const struct typinfo TypInfo[] =
*** 139,145 ****
--- 139,149 ----
      F_TIDIN, F_TIDOUT},
      {"xid", XIDOID, 0, 4, true, 'i', 'p',
      F_XIDIN, F_XIDOUT},
+ #ifdef USE_64BIT_COMMANDID
+     {"cid", CIDOID, 0, 8, false, 'd', 'p',
+ #else
      {"cid", CIDOID, 0, 4, true, 'i', 'p',
+ #endif
      F_CIDIN, F_CIDOUT},
      {"int2vector", INT2VECTOROID, INT2OID, -1, false, 'i', 'p',
      F_INT2VECTORIN, F_INT2VECTOROUT},
diff -dcrpN pgsql.orig/src/backend/catalog/genbki.sh pgsql-cid64/src/backend/catalog/genbki.sh
*** pgsql.orig/src/backend/catalog/genbki.sh    2008-01-01 20:45:48.000000000 +0100
--- pgsql-cid64/src/backend/catalog/genbki.sh    2008-03-04 16:53:46.000000000 +0100
*************** for dir in $INCLUDE_DIRS; do
*** 114,119 ****
--- 114,136 ----
      fi
  done

+ # Deduce CIDSTORAGELEN from pg_config.h / USE_64BIT_COMMANDID
+ for dir in $INCLUDE_DIRS; do
+     if [ -f "$dir/pg_config.h" ]; then
+     HUGECID=`grep '^#define[    ]*USE_64BIT_COMMANDID' $dir/pg_config.h | $AWK '{ print $3 }'`
+     break
+     fi
+ done
+ if [ "$HUGECID" == "1" ]; then
+     CIDSTORAGELEN="8"
+     CIDSTORAGEALIGN="d"
+     CIDPASSBYVAL="f"
+ else
+     CIDSTORAGELEN="4"
+     CIDSTORAGEALIGN="i"
+     CIDPASSBYVAL="t"
+ fi
+
  # Get BOOTSTRAP_SUPERUSERID from catalog/pg_authid.h
  for dir in $INCLUDE_DIRS; do
      if [ -f "$dir/catalog/pg_authid.h" ]; then
*************** sed -e "s/;[     ]*$//g" \
*** 164,169 ****
--- 181,189 ----
      -e "s/(TransactionId/(xid/g" \
      -e "s/PGUID/$BOOTSTRAP_SUPERUSERID/g" \
      -e "s/NAMEDATALEN/$NAMEDATALEN/g" \
+     -e "s/CIDSTORAGELEN/$CIDSTORAGELEN/g" \
+     -e "s/CIDSTORAGEALIGN/$CIDSTORAGEALIGN/g" \
+     -e "s/CIDPASSBYVAL/$CIDPASSBYVAL/g" \
      -e "s/PGNSP/$PG_CATALOG_NAMESPACE/g" \
  | $AWK '
  # ----------------
diff -dcrpN pgsql.orig/src/backend/catalog/heap.c pgsql-cid64/src/backend/catalog/heap.c
*** pgsql.orig/src/backend/catalog/heap.c    2008-01-01 20:45:48.000000000 +0100
--- pgsql-cid64/src/backend/catalog/heap.c    2008-03-04 16:53:46.000000000 +0100
*************** static FormData_pg_attribute a3 = {
*** 117,123 ****
--- 117,127 ----
  static FormData_pg_attribute a4 = {
      0, {"cmin"}, CIDOID, 0, sizeof(CommandId),
      MinCommandIdAttributeNumber, 0, -1, -1,
+ #ifdef USE_64BIT_COMMANDID
+     false, 'p', 'd', true, false, false, true, 0
+ #else
      true, 'p', 'i', true, false, false, true, 0
+ #endif
  };

  static FormData_pg_attribute a5 = {
*************** static FormData_pg_attribute a5 = {
*** 129,135 ****
--- 133,143 ----
  static FormData_pg_attribute a6 = {
      0, {"cmax"}, CIDOID, 0, sizeof(CommandId),
      MaxCommandIdAttributeNumber, 0, -1, -1,
+ #ifdef USE_64BIT_COMMANDID
+     false, 'p', 'd', true, false, false, true, 0
+ #else
      true, 'p', 'i', true, false, false, true, 0
+ #endif
  };

  /*
diff -dcrpN pgsql.orig/src/backend/catalog/Makefile pgsql-cid64/src/backend/catalog/Makefile
*** pgsql.orig/src/backend/catalog/Makefile    2008-03-02 13:44:44.000000000 +0100
--- pgsql-cid64/src/backend/catalog/Makefile    2008-03-04 16:55:51.000000000 +0100
*************** postgres.description: postgres.bki ;
*** 46,52 ****

  postgres.shdescription: postgres.bki ;

! postgres.bki: genbki.sh $(POSTGRES_BKI_SRCS) $(top_builddir)/src/include/pg_config_manual.h
      AWK='$(AWK)' $(SHELL) $< $(pg_includes) --set-version=$(VERSION) -o postgres $(POSTGRES_BKI_SRCS)

  .PHONY: install-data
--- 46,52 ----

  postgres.shdescription: postgres.bki ;

! postgres.bki: genbki.sh $(POSTGRES_BKI_SRCS) $(top_builddir)/src/include/pg_config_manual.h
$(top_builddir)/src/include/pg_config.h
      AWK='$(AWK)' $(SHELL) $< $(pg_includes) --set-version=$(VERSION) -o postgres $(POSTGRES_BKI_SRCS)

  .PHONY: install-data
diff -dcrpN pgsql.orig/src/include/catalog/pg_attribute.h pgsql-cid64/src/include/catalog/pg_attribute.h
*** pgsql.orig/src/include/catalog/pg_attribute.h    2008-01-01 20:45:56.000000000 +0100
--- pgsql-cid64/src/include/catalog/pg_attribute.h    2008-03-04 16:53:46.000000000 +0100
*************** DATA(insert ( 1247 typdefault        25 -1 -1
*** 278,286 ****
  DATA(insert ( 1247 ctid                27 0  6  -1 0 -1 -1 f p s t f f t 0));
  DATA(insert ( 1247 oid                26 0  4  -2 0 -1 -1 t p i t f f t 0));
  DATA(insert ( 1247 xmin                28 0  4  -3 0 -1 -1 t p i t f f t 0));
! DATA(insert ( 1247 cmin                29 0  4  -4 0 -1 -1 t p i t f f t 0));
  DATA(insert ( 1247 xmax                28 0  4  -5 0 -1 -1 t p i t f f t 0));
! DATA(insert ( 1247 cmax                29 0  4  -6 0 -1 -1 t p i t f f t 0));
  DATA(insert ( 1247 tableoid            26 0  4  -7 0 -1 -1 t p i t f f t 0));

  /* ----------------
--- 278,286 ----
  DATA(insert ( 1247 ctid                27 0  6  -1 0 -1 -1 f p s t f f t 0));
  DATA(insert ( 1247 oid                26 0  4  -2 0 -1 -1 t p i t f f t 0));
  DATA(insert ( 1247 xmin                28 0  4  -3 0 -1 -1 t p i t f f t 0));
! DATA(insert ( 1247 cmin                29 0  CIDSTORAGELEN  -4 0 -1 -1 CIDPASSBYVAL p CIDSTORAGEALIGN t f f t 0));
  DATA(insert ( 1247 xmax                28 0  4  -5 0 -1 -1 t p i t f f t 0));
! DATA(insert ( 1247 cmax                29 0  CIDSTORAGELEN  -6 0 -1 -1 CIDPASSBYVAL p CIDSTORAGEALIGN t f f t 0));
  DATA(insert ( 1247 tableoid            26 0  4  -7 0 -1 -1 t p i t f f t 0));

  /* ----------------
*************** DATA(insert ( 1255 proacl          1034 -1 -1
*** 334,342 ****
  DATA(insert ( 1255 ctid                27 0  6  -1 0 -1 -1 f p s t f f t 0));
  DATA(insert ( 1255 oid                26 0  4  -2 0 -1 -1 t p i t f f t 0));
  DATA(insert ( 1255 xmin                28 0  4  -3 0 -1 -1 t p i t f f t 0));
! DATA(insert ( 1255 cmin                29 0  4  -4 0 -1 -1 t p i t f f t 0));
  DATA(insert ( 1255 xmax                28 0  4  -5 0 -1 -1 t p i t f f t 0));
! DATA(insert ( 1255 cmax                29 0  4  -6 0 -1 -1 t p i t f f t 0));
  DATA(insert ( 1255 tableoid            26 0  4  -7 0 -1 -1 t p i t f f t 0));

  /* ----------------
--- 334,342 ----
  DATA(insert ( 1255 ctid                27 0  6  -1 0 -1 -1 f p s t f f t 0));
  DATA(insert ( 1255 oid                26 0  4  -2 0 -1 -1 t p i t f f t 0));
  DATA(insert ( 1255 xmin                28 0  4  -3 0 -1 -1 t p i t f f t 0));
! DATA(insert ( 1255 cmin                29 0  CIDSTORAGELEN  -4 0 -1 -1 CIDPASSBYVAL p CIDSTORAGEALIGN t f f t 0));
  DATA(insert ( 1255 xmax                28 0  4  -5 0 -1 -1 t p i t f f t 0));
! DATA(insert ( 1255 cmax                29 0  CIDSTORAGELEN  -6 0 -1 -1 CIDPASSBYVAL p CIDSTORAGEALIGN t f f t 0));
  DATA(insert ( 1255 tableoid            26 0  4  -7 0 -1 -1 t p i t f f t 0));

  /* ----------------
*************** DATA(insert ( 1249 attinhcount        23 -1  4
*** 382,390 ****
  DATA(insert ( 1249 ctid                27 0  6  -1 0 -1 -1 f p s t f f t 0));
  /* no OIDs in pg_attribute */
  DATA(insert ( 1249 xmin                28 0  4  -3 0 -1 -1 t p i t f f t 0));
! DATA(insert ( 1249 cmin                29 0  4  -4 0 -1 -1 t p i t f f t 0));
  DATA(insert ( 1249 xmax                28 0  4  -5 0 -1 -1 t p i t f f t 0));
! DATA(insert ( 1249 cmax                29 0  4  -6 0 -1 -1 t p i t f f t 0));
  DATA(insert ( 1249 tableoid            26 0  4  -7 0 -1 -1 t p i t f f t 0));

  /* ----------------
--- 382,390 ----
  DATA(insert ( 1249 ctid                27 0  6  -1 0 -1 -1 f p s t f f t 0));
  /* no OIDs in pg_attribute */
  DATA(insert ( 1249 xmin                28 0  4  -3 0 -1 -1 t p i t f f t 0));
! DATA(insert ( 1249 cmin                29 0  CIDSTORAGELEN  -4 0 -1 -1 CIDPASSBYVAL p CIDSTORAGEALIGN t f f t 0));
  DATA(insert ( 1249 xmax                28 0  4  -5 0 -1 -1 t p i t f f t 0));
! DATA(insert ( 1249 cmax                29 0  CIDSTORAGELEN  -6 0 -1 -1 CIDPASSBYVAL p CIDSTORAGEALIGN t f f t 0));
  DATA(insert ( 1249 tableoid            26 0  4  -7 0 -1 -1 t p i t f f t 0));

  /* ----------------
*************** DATA(insert ( 1259 reloptions      1009 -1
*** 450,458 ****
  DATA(insert ( 1259 ctid                27 0  6  -1 0 -1 -1 f p s t f f t 0));
  DATA(insert ( 1259 oid                26 0  4  -2 0 -1 -1 t p i t f f t 0));
  DATA(insert ( 1259 xmin                28 0  4  -3 0 -1 -1 t p i t f f t 0));
! DATA(insert ( 1259 cmin                29 0  4  -4 0 -1 -1 t p i t f f t 0));
  DATA(insert ( 1259 xmax                28 0  4  -5 0 -1 -1 t p i t f f t 0));
! DATA(insert ( 1259 cmax                29 0  4  -6 0 -1 -1 t p i t f f t 0));
  DATA(insert ( 1259 tableoid            26 0  4  -7 0 -1 -1 t p i t f f t 0));

  /* ----------------
--- 450,458 ----
  DATA(insert ( 1259 ctid                27 0  6  -1 0 -1 -1 f p s t f f t 0));
  DATA(insert ( 1259 oid                26 0  4  -2 0 -1 -1 t p i t f f t 0));
  DATA(insert ( 1259 xmin                28 0  4  -3 0 -1 -1 t p i t f f t 0));
! DATA(insert ( 1259 cmin                29 0  CIDSTORAGELEN  -4 0 -1 -1 CIDPASSBYVAL p CIDSTORAGEALIGN t f f t 0));
  DATA(insert ( 1259 xmax                28 0  4  -5 0 -1 -1 t p i t f f t 0));
! DATA(insert ( 1259 cmax                29 0  CIDSTORAGELEN  -6 0 -1 -1 CIDPASSBYVAL p CIDSTORAGEALIGN t f f t 0));
  DATA(insert ( 1259 tableoid            26 0  4  -7 0 -1 -1 t p i t f f t 0));

  /* ----------------
diff -dcrpN pgsql.orig/src/include/catalog/pg_control.h pgsql-cid64/src/include/catalog/pg_control.h
*** pgsql.orig/src/include/catalog/pg_control.h    2008-02-18 13:50:12.000000000 +0100
--- pgsql-cid64/src/include/catalog/pg_control.h    2008-03-04 21:14:04.000000000 +0100
*************** typedef struct ControlFileData
*** 145,150 ****
--- 145,152 ----
      char        lc_collate[LOCALE_NAME_BUFLEN];
      char        lc_ctype[LOCALE_NAME_BUFLEN];

+     uint32        enable64bitCommandId;
+
      /* CRC of all above ... MUST BE LAST! */
      pg_crc32    crc;
  } ControlFileData;
diff -dcrpN pgsql.orig/src/include/catalog/pg_type.h pgsql-cid64/src/include/catalog/pg_type.h
*** pgsql.orig/src/include/catalog/pg_type.h    2008-01-01 20:45:57.000000000 +0100
--- pgsql-cid64/src/include/catalog/pg_type.h    2008-03-04 16:59:27.000000000 +0100
*************** DATA(insert OID = 28 (    xid           PGNSP PGU
*** 314,320 ****
  DESCR("transaction id");
  #define XIDOID 28

! DATA(insert OID = 29 (    cid           PGNSP PGUID    4 t b t \054 0     0 1012 cidin cidout cidrecv cidsend - - - i
pf 0 -1 0 _null_ _null_ )); 
  DESCR("command identifier type, sequence in transaction id");
  #define CIDOID 29

--- 314,320 ----
  DESCR("transaction id");
  #define XIDOID 28

! DATA(insert OID = 29 (    cid           PGNSP PGUID    CIDSTORAGELEN CIDPASSBYVAL b t \054 0     0 1012 cidin cidout
cidrecvcidsend - - - CIDSTORAGEALIGN p f 0 -1 0 _null_ _null_ )); 
  DESCR("command identifier type, sequence in transaction id");
  #define CIDOID 29

diff -dcrpN pgsql.orig/src/include/c.h pgsql-cid64/src/include/c.h
*** pgsql.orig/src/include/c.h    2008-03-02 13:44:45.000000000 +0100
--- pgsql-cid64/src/include/c.h    2008-03-04 21:05:23.000000000 +0100
*************** typedef TransactionId MultiXactId;
*** 382,388 ****
--- 382,392 ----

  typedef uint32 MultiXactOffset;

+ #ifdef USE_64BIT_COMMANDID
+ typedef uint64 CommandId;
+ #else
  typedef uint32 CommandId;
+ #endif

  #define FirstCommandId    ((CommandId) 0)

diff -dcrpN pgsql.orig/src/include/pg_config.h.in pgsql-cid64/src/include/pg_config.h.in
*** pgsql.orig/src/include/pg_config.h.in    2008-02-18 13:50:12.000000000 +0100
--- pgsql-cid64/src/include/pg_config.h.in    2008-03-04 16:53:46.000000000 +0100
***************
*** 656,661 ****
--- 656,665 ----
     */
  #undef UINT64_FORMAT

+ /* Define to 1 if you want 64-bit CommandId support. (--enable-huge-commandid)
+    */
+ #undef USE_64BIT_COMMANDID
+
  /* Define to 1 to build with assertion checks. (--enable-cassert) */
  #undef USE_ASSERT_CHECKING

diff -dcrpN pgsql.orig/src/include/postgres.h pgsql-cid64/src/include/postgres.h
*** pgsql.orig/src/include/postgres.h    2008-01-01 20:45:56.000000000 +0100
--- pgsql-cid64/src/include/postgres.h    2008-03-04 16:53:46.000000000 +0100
*************** typedef Datum *DatumPtr;
*** 462,475 ****
--- 462,483 ----
   *        Returns command identifier value of a datum.
   */

+ #ifdef USE_64BIT_COMMANDID
+ #define DatumGetCommandId(X) ((CommandId) DatumGetInt64(X))
+ #else
  #define DatumGetCommandId(X) ((CommandId) GET_4_BYTES(X))
+ #endif

  /*
   * CommandIdGetDatum
   *        Returns datum representation for a command identifier.
   */

+ #ifdef USE_64BIT_COMMANDID
+ #define CommandIdGetDatum(X) ((Datum) Int64GetDatum(X))
+ #else
  #define CommandIdGetDatum(X) ((Datum) SET_4_BYTES(X))
+ #endif

  /*
   * DatumGetPointer

Re: [Fwd: Re: [PATCHES] 64-bit CommandIds]

From

Gregory Stark

Date:

10 March 2008, 16:59:45

"Zoltan Boszormenyi" <zb@cybertec.at> writes:

> Hi,
>
> what's your opinion on this?
> I saw response only from Alvaro on the -patches list.

I don't understand. The patch only affects configuration and SQL data type
code. It doesn't actually store the 64-bit commandid anywhere which would be
the actual hard part.

Do "phantom" command ids mean this all just works magically? Ie, the limit of
2^32 <cmin,cmax> pairs is still there but as long as you don't have to store
more than that many you get to have 2^64 raw ephemeral commandids?

--  Gregory Stark EnterpriseDB          http://www.enterprisedb.com Ask me about EnterpriseDB's Slony Replication
support!

Re: [Fwd: Re: [PATCHES] 64-bit CommandIds]

From

"Heikki Linnakangas"

Date:

10 March 2008, 17:06:30

Gregory Stark wrote:
> I don't understand. The patch only affects configuration and SQL data type
> code. It doesn't actually store the 64-bit commandid anywhere which would be
> the actual hard part.

Sure it does, this is the significant part of the patch:

*** pgsql.orig/src/include/c.h    2008-03-02 13:44:45.000000000 +0100
--- pgsql-cid64/src/include/c.h    2008-03-04 21:05:23.000000000 +0100
*************** typedef TransactionId MultiXactId;
*** 382,388 ****
--- 382,392 ----
  typedef uint32 MultiXactOffset;

+ #ifdef USE_64BIT_COMMANDID
+ typedef uint64 CommandId;
+ #else  typedef uint32 CommandId;
+ #endif
  #define FirstCommandId    ((CommandId) 0)

CommandId type is used in htup.h and elsewhere, which changes the 
on-disk format.

--   Heikki Linnakangas  EnterpriseDB   http://www.enterprisedb.com

Re: [Fwd: Re: [PATCHES] 64-bit CommandIds]

From

Decibel!

Date:

20 March 2008, 20:27:39

On Mar 10, 2008, at 12:06 PM, Heikki Linnakangas wrote:
> Gregory Stark wrote:
>> I don't understand. The patch only affects configuration and SQL  
>> data type
>> code. It doesn't actually store the 64-bit commandid anywhere  
>> which would be
>> the actual hard part.
>
> Sure it does, this is the significant part of the patch:
>
> *** pgsql.orig/src/include/c.h    2008-03-02 13:44:45.000000000 +0100
> --- pgsql-cid64/src/include/c.h    2008-03-04 21:05:23.000000000 +0100
> *************** typedef TransactionId MultiXactId;
> *** 382,388 ****
> --- 382,392 ----
>
>   typedef uint32 MultiXactOffset;
>
> + #ifdef USE_64BIT_COMMANDID
> + typedef uint64 CommandId;
> + #else
>   typedef uint32 CommandId;
> + #endif
>
>   #define FirstCommandId    ((CommandId) 0)
>
> CommandId type is used in htup.h and elsewhere, which changes the  
> on-disk format.

If we're going to make this a ./configure option, ISTM we should do  
the same with XID size as well. I know there are high-velocity  
databases that could use that.
-- 
Decibel!, aka Jim C. Nasby, Database Architect  decibel@decibel.org
Give your computer some brain candy! www.distributed.net Team #1828

Re: [Fwd: Re: [PATCHES] 64-bit CommandIds]

From

Gregory Stark

Date:

20 March 2008, 22:15:56

"Decibel!" <decibel@decibel.org> writes:

> If we're going to make this a ./configure option, ISTM we should do  the same
> with XID size as well. I know there are high-velocity  databases that could use
> that.

Keep in mind we just changed things so that read-only transactions don't
consume xids. That means you would have to be actually modifying 2-billion
records before wrap-around becomes an issue.

If you're modifying 2-billion records that quickly presumably you're going to
have other pressing reasons to run vacuum aside from xid freezing...

Also, consider that you're suggesting increasing the per-tuple overhead from
24 bytes to, if my arithmetic is right, 40 bytes. 

So really you would need, say, a system with enough i/o bandwidth to handle
2-billion updates or inserts per day and with enough spare i/o bandwidth that
another 16-bytes on every one of those updates is ok, but without the ability
to run vacuum.

Also, we still have hope that the visibility map info will make running vacuum
even less of an imposition.

All that said I don't really see much reason not to make it an option. I just
don't think anyone really needs it. In 5-10 years though...

--  Gregory Stark EnterpriseDB          http://www.enterprisedb.com Ask me about EnterpriseDB's 24x7 Postgres support!

Re: [Fwd: Re: [PATCHES] 64-bit CommandIds]

From

Tom Lane

Date:

20 March 2008, 22:57:56

Gregory Stark <stark@enterprisedb.com> writes:
> All that said I don't really see much reason not to make it an option. I just
> don't think anyone really needs it. In 5-10 years though...

The manpower we'd have to invest in making it work and keeping it
working would be enough reason ...
        regards, tom lane

Re: [Fwd: Re: [PATCHES] 64-bit CommandIds]

From

Hans-Juergen Schoenig

Date:

21 March 2008, 06:06:52

"Decibel!" <decibel@decibel.org> writes:

If we're going to make this a ./configure option, ISTM we should do the same
with XID size as well. I know there are high-velocity databases that could use
that.

Keep in mind we just changed things so that read-only transactions don't
consume xids. That means you would have to be actually modifying 2-billion
records before wrap-around becomes an issue.

If you're modifying 2-billion records that quickly presumably you're going to
have other pressing reasons to run vacuum aside from xid freezing...

Also, consider that you're suggesting increasing the per-tuple overhead from
24 bytes to, if my arithmetic is right, 40 bytes.

So really you would need, say, a system with enough i/o bandwidth to handle
2-billion updates or inserts per day and with enough spare i/o bandwidth that
another 16-bytes on every one of those updates is ok, but without the ability
to run vacuum.

Also, we still have hope that the visibility map info will make running vacuum
even less of an imposition.

All that said I don't really see much reason not to make it an option. I just
don't think anyone really needs it. In 5-10 years though...

Doing this for XIDs is pretty useless this days.

It is only targeted for command ids which are consumed heavily by stored procedure languages.

It happens once on a while that a complex business logic procedure runs out of command ids inside a transaction.

the idea is to give users a chance to avoid that.

touching XIDs does not make sense to me at all.

many thanks,

hans

Cybertec Schönig & Schönig GmbH

PostgreSQL Solutions and Support

Gröhrmühlgasse 26, 2700 Wiener Neustadt

Tel: +43/1/205 10 35 / 340

www.postgresql.at, www.cybertec.at

Re: [Fwd: Re: [PATCHES] 64-bit CommandIds]

From

Tom Lane

Date:

21 March 2008, 14:41:33

Hans-Juergen Schoenig <hs@cybertec.at> writes:
> Doing this for XIDs is pretty useless this days.
> It is only targeted for command ids which are consumed heavily by  
> stored procedure languages.
> It happens once on a while that a complex business logic procedure  
> runs out of command ids inside a transaction.
> the idea is to give users a chance to avoid that.
> touching XIDs does not make sense to me at all.

In view of the fact that 8.3 greatly reduced the CommandID consumption
of typical plpgsql code
http://archives.postgresql.org/pgsql-committers/2007-11/msg00585.php
I wonder whether the case for wider CIDs hasn't likewise taken a
major hit.
        regards, tom lane