Re: Bit data type header reduction in some cases - Mailing list pgsql-hackers

On 02/25/2014 08:23 AM, Haribabu Kommi wrote:
> It's regarding a Todo item of "Bit data type header reduction" in some
> cases. The header contains two parts. 1)  The varlena header is
> automatically converted to 1 byte header from 4 bytes in case of small
> data. 2) The bit length header called "bit_len" to store the actual bit
> length which is of 4 bytes in size. I want to reduce this bit_len size to 1
> byte in some cases as similar to varlena header. With this change the size
> of the column reduced by 3 bytes, thus shows very good decrease in disk
> usage.
>
> I am planning to the change it based on the total length of bits data. If
> the number of bits is less than 0x7F then one byte header can be chosen
> instead of 4 byte header. During reading of the data, the similar way it
> can be calculated.

Since we're designing a new format, how about using bit padding instead 
of an explicit length field? Add one 1-bit to the end of the bit data, 
followed by zeros. That's even more compact. See 
https://en.wikipedia.org/wiki/Padding_%28cryptography%29#Bit_padding

> The problem I am thinking is, as it can cause problems to old databases
> having bit columns when they
> upgrade to a newer version without using pg_dumpall. Is there anyway to
> handle this problem? Or Is there any better way to handle the problem
> itself? please let me know your suggestions.

On a little-endian system, you could easily use the most-significant bit 
(sign bit) of the bit_len field to indicate that it's a new-format datum:

/* * Little-endian */
typedef struct
{int32        vl_len_;int32        bit_len;bits8        bit_dat[1];
} VarBit_Old;

typedef struct
{int32        vl_len_;uint8        bit_len;bits8        bit_dat[1]; /* var len */
} VarBit_New;

#define IS_NEW_FORMAT(((VarBit_New *) x)->bit_len & 0x80)

On a big-endian system that's more difficult, because the MSB would not 
be the first byte of the field. You could still make it work if the new 
format would have the length byte in fourth byte of the Datum. Something 
like this:

/* * Big-endian */
typedef struct
{int32        vl_len_;int32        bit_len;bits8        bit_dat[1];
} VarBit_Old;

typedef struct
{int32        vl_len_;bits8        bit_dat_first_three[3];uint8        bit_len;bits8        bit_dat_rest[1]; /* var len
*/
} VarBit_New;

#define IS_NEW_FORMAT(((VarBit_New *) x)->bit_len & 0x80)

That's not a complete solution yet, you also need a special case for a 
new-format value with less than 4 bytes of bits. Such a field would 
presumably not have the bit_len field at the fourth byte, because it 
would be shorter than that, but you could distinguish it by looking at 
the length in the varsize header. So in big-endian, you'd have one more 
format for very small varbits:

/* Big-endian, less than 4 bytes of bits data */
typedef struct
{int32        vl_len_;uint8        bit_len;bits8        bit_dat[1]; /* var len */
} VarBit_New_Tiny;

The same basic idea also works with bit padding.

- Heikki