Thread: [PATCH] pgcrypto: pgp_encrypt

Finally, here is pgp_encrypt()/pgp_decrypt() - implementation
of password-based encryption from RFC2440 (OpenPGP).

The goal of this code is to be more featureful encryption solution
than current encrypt(), which only functionality is running cipher
over data.

Compared to encrypt(), pgp_encrypt() does following:

* It uses the equvialent of random Inital Vector to get cipher
  into random state before it processes user data
* Stores SHA-1 of the data into result so any modification
  will be detected.
* Remembers if data was text or binary - thus it can decrypt
  to/from text data.  This was a major nuisance for encrypt().
* Stores info about used algorithms with result, so user needs
  not remember them - more user friendly!
* Uses String2Key algorithms (similar to crypt()) with random salt
  to generate full-length binary key to be used for encrypting.
* Uses standard format for data - you can feed it to GnuPG, if needed.

Optional features (off by default):

* Can use separate session key - user data will be encrypted
  with totally random key, which will be encrypted with S2K
  generated key and attached to result.
* Data compression with zlib.
* Can convert between CRLF<->LF line-endings - to get fully
  RFC2440-compliant behaviour.  This is off by default as
  pgcrypto does not know the line-endings of user data.

Interface is simple:

    pgp_encrypt(data text, key text) returns bytea
    pgp_decrypt(data text, key text) returns text
    pgp_encrypt_bytea(data bytea, key text) returns bytea
    pgp_decrypt_bytea(data bytea, key text) returns bytea

To change parameters (cipher, compression, mdc):

    pgp_encrypt(data text, key text, parms text) returns bytea
    pgp_decrypt(data text, key text, parms text) returns text
    pgp_encrypt_bytea(data bytea, key text, parms text) returns bytea
    pgp_decrypt_bytea(data bytea, key text, parms text) returns bytea

Parameter names I lifted from gpg:

   pgp_encrypt('message', 'key', 'compress-algo=1,cipher-algo=aes256')

For text data, pgp_encrypt simply encrypts the PostgreSQL internal
data.  This maps to RFC2440 data type 't' - 'extenally specified
encoding'.  But this may cause problems if data is dumped and reloaded
into database which as different internal encoding.  My next goal is
to implement data type 'u' - which means data is in UTF-8 encoding
by converting internal encoding to UTF-8 and back.  And there wont
be any compatibility problems with current code, I think its ok to
submit this without UTF-8 support.

--
marko


PS.  openssl/3des regression test fails, I'll send fix once
I understand why its happening.

On Mon, Jun 27, 2005 at 09:08:05PM -0400, Bruce Momjian wrote:
> Your patch has been added to the PostgreSQL unapplied patches list at:
>
>     http://momjian.postgresql.org/cgi-bin/pgpatches
>
> It will be applied as soon as one of the PostgreSQL committers reviews
> and approves it.

Please use following updated patch instead.

It implements utf8 conversion, fixes couple of bugs and has
many code and comment cleanups.

--
marko

Marko Kreen wrote:
> Please use following updated patch instead.
>
> It implements utf8 conversion, fixes couple of bugs and has
> many code and comment cleanups.

The regression tests don't pass on my box. With the default Makefile,
there are a lot of errors WRT "no strong random source". After editing
the Makefile to make use the "random" device, I get the attached
regression.diffs.

While I understand the need to make sure people use a reasonably strong
crypto source, it would be nice if the regression tests passed out of
the box.

-Neil
*** ./expected/pgp-armor.out    Mon Jul  4 16:52:12 2005
--- ./results/pgp-armor.out    Mon Jul  4 16:57:55 2005
***************
*** 3,9 ****
  --
  select armor('');
                               armor
! -----------------------------
   -----BEGIN PGP MESSAGE-----

  =twTO
--- 3,9 ----
  --
  select armor('');
                               armor
! ---------------------------------------------------------------
   -----BEGIN PGP MESSAGE-----

  =twTO
***************
*** 13,19 ****

  select armor('test');
                                   armor
! -----------------------------
   -----BEGIN PGP MESSAGE-----

  dGVzdA==
--- 13,19 ----

  select armor('test');
                                   armor
! ------------------------------------------------------------------------
   -----BEGIN PGP MESSAGE-----

  dGVzdA==
***************
*** 37,43 ****
  select armor('0123456789abcdef0123456789abcdef0123456789abcdef
  0123456789abcdef0123456789abcdef0123456789abcdef');
                                                                                                  armor
                                                                               
! -----------------------------
   -----BEGIN PGP MESSAGE-----

  MDEyMzQ1Njc4OWFiY2RlZjAxMjM0NTY3ODlhYmNkZWYwMTIzNDU2Nzg5YWJjZGVmCjAxMjM0NTY3
--- 37,43 ----
  select armor('0123456789abcdef0123456789abcdef0123456789abcdef
  0123456789abcdef0123456789abcdef0123456789abcdef');
                                                                                                  armor
                                                                               
!
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
   -----BEGIN PGP MESSAGE-----

  MDEyMzQ1Njc4OWFiY2RlZjAxMjM0NTY3ODlhYmNkZWYwMTIzNDU2Nzg5YWJjZGVmCjAxMjM0NTY3

======================================================================

*** ./expected/pgp-encrypt.out    Mon Jul  4 16:52:12 2005
--- ./results/pgp-encrypt.out    Mon Jul  4 16:57:55 2005
***************
*** 2,12 ****
  -- PGP encrypt
  --
  select pgp_decrypt(pgp_encrypt('Secret.', 'key'), 'key');
!  pgp_decrypt
! -------------
!  Secret.
! (1 row)
!
  -- check whether the defaults are ok
  select pgp_decrypt(pgp_encrypt('Secret.', 'key'),
       'key', 'expect-cipher-algo=aes256,
--- 2,8 ----
  -- PGP encrypt
  --
  select pgp_decrypt(pgp_encrypt('Secret.', 'key'), 'key');
! ERROR:  pgp_encrypt: No strong random source
  -- check whether the defaults are ok
  select pgp_decrypt(pgp_encrypt('Secret.', 'key'),
       'key', 'expect-cipher-algo=aes256,
***************
*** 16,26 ****
          expect-s2k-digest-algo=sha1,
          expect-compress-algo=0
          ');
!  pgp_decrypt
! -------------
!  Secret.
! (1 row)
!
  -- maybe the expect- stuff simply does not work
  select pgp_decrypt(pgp_encrypt('Secret.', 'key'),
       'key', 'expect-cipher-algo=bf,
--- 12,18 ----
          expect-s2k-digest-algo=sha1,
          expect-compress-algo=0
          ');
! ERROR:  pgp_encrypt: No strong random source
  -- maybe the expect- stuff simply does not work
  select pgp_decrypt(pgp_encrypt('Secret.', 'key'),
       'key', 'expect-cipher-algo=bf,
***************
*** 30,189 ****
          expect-s2k-digest-algo=md5,
          expect-compress-algo=1
          ');
! NOTICE:  pgp_decrypt: unexpected cipher_algo: expected 4 got 9
! NOTICE:  pgp_decrypt: unexpected s2k_mode: expected 0 got 3
! NOTICE:  pgp_decrypt: unexpected s2k_digest_algo: expected 1 got 2
! NOTICE:  pgp_decrypt: unexpected use_sess_key: expected 1 got 0
! NOTICE:  pgp_decrypt: unexpected disable_mdc: expected 1 got 0
! NOTICE:  pgp_decrypt: unexpected compress_algo: expected 1 got 0
!  pgp_decrypt
! -------------
!  Secret.
! (1 row)
!
  -- bytea as text
  select pgp_decrypt(pgp_encrypt_bytea('Binary', 'baz'), 'baz');
! ERROR:  pgp_decrypt error: Not text data
  -- text as bytea
  select pgp_decrypt_bytea(pgp_encrypt('Text', 'baz'), 'baz');
!  pgp_decrypt_bytea
! -------------------
!  Text
! (1 row)
!
  -- algorithm change
  select pgp_decrypt(
      pgp_encrypt('Secret.', 'key', 'cipher-algo=bf'),
       'key', 'expect-cipher-algo=bf');
!  pgp_decrypt
! -------------
!  Secret.
! (1 row)
!
  select pgp_decrypt(
      pgp_encrypt('Secret.', 'key', 'cipher-algo=aes'),
       'key', 'expect-cipher-algo=aes128');
!  pgp_decrypt
! -------------
!  Secret.
! (1 row)
!
  select pgp_decrypt(
      pgp_encrypt('Secret.', 'key', 'cipher-algo=aes192'),
       'key', 'expect-cipher-algo=aes192');
!  pgp_decrypt
! -------------
!  Secret.
! (1 row)
!
  -- s2k change
  select pgp_decrypt(
      pgp_encrypt('Secret.', 'key', 's2k-mode=0'),
       'key', 'expect-s2k-mode=0');
!  pgp_decrypt
! -------------
!  Secret.
! (1 row)
!
  select pgp_decrypt(
      pgp_encrypt('Secret.', 'key', 's2k-mode=1'),
       'key', 'expect-s2k-mode=1');
!  pgp_decrypt
! -------------
!  Secret.
! (1 row)
!
  select pgp_decrypt(
      pgp_encrypt('Secret.', 'key', 's2k-mode=3'),
       'key', 'expect-s2k-mode=3');
!  pgp_decrypt
! -------------
!  Secret.
! (1 row)
!
  -- s2k digest change
  select pgp_decrypt(
      pgp_encrypt('Secret.', 'key', 's2k-digest-algo=md5'),
       'key', 'expect-s2k-digest-algo=md5');
!  pgp_decrypt
! -------------
!  Secret.
! (1 row)
!
  select pgp_decrypt(
          pgp_encrypt('Secret.', 'key', 's2k-digest-algo=sha1'),
       'key', 'expect-s2k-digest-algo=sha1');
!  pgp_decrypt
! -------------
!  Secret.
! (1 row)
!
  -- sess key
  select pgp_decrypt(
      pgp_encrypt('Secret.', 'key', 'sess-key=0'),
       'key', 'expect-sess-key=0');
!  pgp_decrypt
! -------------
!  Secret.
! (1 row)
!
  select pgp_decrypt(
      pgp_encrypt('Secret.', 'key', 'sess-key=1'),
       'key', 'expect-sess-key=1');
!  pgp_decrypt
! -------------
!  Secret.
! (1 row)
!
  select pgp_decrypt(
      pgp_encrypt('Secret.', 'key', 'sess-key=1, cipher-algo=bf'),
       'key', 'expect-sess-key=1, expect-cipher-algo=bf');
!  pgp_decrypt
! -------------
!  Secret.
! (1 row)
!
  select pgp_decrypt(
      pgp_encrypt('Secret.', 'key', 'sess-key=1, cipher-algo=aes192'),
       'key', 'expect-sess-key=1, expect-cipher-algo=aes192');
!  pgp_decrypt
! -------------
!  Secret.
! (1 row)
!
  select pgp_decrypt(
      pgp_encrypt('Secret.', 'key', 'sess-key=1, cipher-algo=aes256'),
       'key', 'expect-sess-key=1, expect-cipher-algo=aes256');
!  pgp_decrypt
! -------------
!  Secret.
! (1 row)
!
  -- no mdc
  select pgp_decrypt(
          pgp_encrypt('Secret.', 'key', 'disable-mdc=1'),
       'key', 'expect-disable-mdc=1');
!  pgp_decrypt
! -------------
!  Secret.
! (1 row)
!
  -- crlf
  select encode(pgp_decrypt_bytea(
      pgp_encrypt('1\n2\n3\r\n', 'key', 'convert-crlf=1'),
       'key'), 'hex');
!         encode
! ----------------------
!  310d0a320d0a330d0d0a
! (1 row)
!
  -- conversion should be lossless
  select encode(digest(pgp_decrypt(
    pgp_encrypt('\r\n0\n1\r\r\n\n2\r', 'key', 'convert-crlf=1'),
       'key', 'convert-crlf=1'), 'sha1'), 'hex') as result,
    encode(digest('\r\n0\n1\r\r\n\n2\r', 'sha1'), 'hex') as expect;
!                   result                  |                  expect
! ------------------------------------------+------------------------------------------
!  47bde5d88d6ef8770572b9cbb4278b402aa69966 | 47bde5d88d6ef8770572b9cbb4278b402aa69966
! (1 row)
!
--- 22,103 ----
          expect-s2k-digest-algo=md5,
          expect-compress-algo=1
          ');
! ERROR:  pgp_encrypt: No strong random source
  -- bytea as text
  select pgp_decrypt(pgp_encrypt_bytea('Binary', 'baz'), 'baz');
! ERROR:  pgp_encrypt: No strong random source
  -- text as bytea
  select pgp_decrypt_bytea(pgp_encrypt('Text', 'baz'), 'baz');
! ERROR:  pgp_encrypt: No strong random source
  -- algorithm change
  select pgp_decrypt(
      pgp_encrypt('Secret.', 'key', 'cipher-algo=bf'),
       'key', 'expect-cipher-algo=bf');
! ERROR:  pgp_encrypt: No strong random source
  select pgp_decrypt(
      pgp_encrypt('Secret.', 'key', 'cipher-algo=aes'),
       'key', 'expect-cipher-algo=aes128');
! ERROR:  pgp_encrypt: No strong random source
  select pgp_decrypt(
      pgp_encrypt('Secret.', 'key', 'cipher-algo=aes192'),
       'key', 'expect-cipher-algo=aes192');
! ERROR:  pgp_encrypt: No strong random source
  -- s2k change
  select pgp_decrypt(
      pgp_encrypt('Secret.', 'key', 's2k-mode=0'),
       'key', 'expect-s2k-mode=0');
! ERROR:  pgp_encrypt: No strong random source
  select pgp_decrypt(
      pgp_encrypt('Secret.', 'key', 's2k-mode=1'),
       'key', 'expect-s2k-mode=1');
! ERROR:  pgp_encrypt: No strong random source
  select pgp_decrypt(
      pgp_encrypt('Secret.', 'key', 's2k-mode=3'),
       'key', 'expect-s2k-mode=3');
! ERROR:  pgp_encrypt: No strong random source
  -- s2k digest change
  select pgp_decrypt(
      pgp_encrypt('Secret.', 'key', 's2k-digest-algo=md5'),
       'key', 'expect-s2k-digest-algo=md5');
! ERROR:  pgp_encrypt: No strong random source
  select pgp_decrypt(
          pgp_encrypt('Secret.', 'key', 's2k-digest-algo=sha1'),
       'key', 'expect-s2k-digest-algo=sha1');
! ERROR:  pgp_encrypt: No strong random source
  -- sess key
  select pgp_decrypt(
      pgp_encrypt('Secret.', 'key', 'sess-key=0'),
       'key', 'expect-sess-key=0');
! ERROR:  pgp_encrypt: No strong random source
  select pgp_decrypt(
      pgp_encrypt('Secret.', 'key', 'sess-key=1'),
       'key', 'expect-sess-key=1');
! ERROR:  pgp_encrypt: No strong random source
  select pgp_decrypt(
      pgp_encrypt('Secret.', 'key', 'sess-key=1, cipher-algo=bf'),
       'key', 'expect-sess-key=1, expect-cipher-algo=bf');
! ERROR:  pgp_encrypt: No strong random source
  select pgp_decrypt(
      pgp_encrypt('Secret.', 'key', 'sess-key=1, cipher-algo=aes192'),
       'key', 'expect-sess-key=1, expect-cipher-algo=aes192');
! ERROR:  pgp_encrypt: No strong random source
  select pgp_decrypt(
      pgp_encrypt('Secret.', 'key', 'sess-key=1, cipher-algo=aes256'),
       'key', 'expect-sess-key=1, expect-cipher-algo=aes256');
! ERROR:  pgp_encrypt: No strong random source
  -- no mdc
  select pgp_decrypt(
          pgp_encrypt('Secret.', 'key', 'disable-mdc=1'),
       'key', 'expect-disable-mdc=1');
! ERROR:  pgp_encrypt: No strong random source
  -- crlf
  select encode(pgp_decrypt_bytea(
      pgp_encrypt('1\n2\n3\r\n', 'key', 'convert-crlf=1'),
       'key'), 'hex');
! ERROR:  pgp_encrypt: No strong random source
  -- conversion should be lossless
  select encode(digest(pgp_decrypt(
    pgp_encrypt('\r\n0\n1\r\r\n\n2\r', 'key', 'convert-crlf=1'),
       'key', 'convert-crlf=1'), 'sha1'), 'hex') as result,
    encode(digest('\r\n0\n1\r\r\n\n2\r', 'sha1'), 'hex') as expect;
! ERROR:  pgp_encrypt: No strong random source

======================================================================

*** ./expected/pgp-compression.out    Mon Jul  4 16:52:12 2005
--- ./results/pgp-compression.out    Mon Jul  4 16:57:55 2005
***************
*** 9,50 ****
  =tbSn
  -----END PGP MESSAGE-----
  '), 'key', 'expect-compress-algo=1');
!   pgp_decrypt
! ----------------
!  Secret message
! (1 row)
!
  select pgp_decrypt(
      pgp_encrypt('Secret message', 'key', 'compress-algo=0'),
      'key', 'expect-compress-algo=0');
!   pgp_decrypt
! ----------------
!  Secret message
! (1 row)
!
  select pgp_decrypt(
      pgp_encrypt('Secret message', 'key', 'compress-algo=1'),
      'key', 'expect-compress-algo=1');
!   pgp_decrypt
! ----------------
!  Secret message
! (1 row)
!
  select pgp_decrypt(
      pgp_encrypt('Secret message', 'key', 'compress-algo=2'),
      'key', 'expect-compress-algo=2');
!   pgp_decrypt
! ----------------
!  Secret message
! (1 row)
!
  -- level=0 should turn compression off
  select pgp_decrypt(
      pgp_encrypt('Secret message', 'key',
              'compress-algo=2, compress-level=0'),
      'key', 'expect-compress-algo=0');
!   pgp_decrypt
! ----------------
!  Secret message
! (1 row)
!
--- 9,30 ----
  =tbSn
  -----END PGP MESSAGE-----
  '), 'key', 'expect-compress-algo=1');
! ERROR:  pgp_decrypt error: Unsupported compression algorithm
  select pgp_decrypt(
      pgp_encrypt('Secret message', 'key', 'compress-algo=0'),
      'key', 'expect-compress-algo=0');
! ERROR:  pgp_encrypt: No strong random source
  select pgp_decrypt(
      pgp_encrypt('Secret message', 'key', 'compress-algo=1'),
      'key', 'expect-compress-algo=1');
! ERROR:  pgp_encrypt: No strong random source
  select pgp_decrypt(
      pgp_encrypt('Secret message', 'key', 'compress-algo=2'),
      'key', 'expect-compress-algo=2');
! ERROR:  pgp_encrypt: No strong random source
  -- level=0 should turn compression off
  select pgp_decrypt(
      pgp_encrypt('Secret message', 'key',
              'compress-algo=2, compress-level=0'),
      'key', 'expect-compress-algo=0');
! ERROR:  pgp_encrypt: No strong random source

======================================================================

Well, I needed to decide, whether it is realistic to implement
PGP public-key encryption someday, because then I better rename
the function 'pgp_encrypt' to something more specific immidiately.

But after hacking a while, I soon had working implementation,
so it seems silly to wait 8.2 for it.  This is thanks to the
fact that for encryption I can ignore most of the OpenPGP
complex parts - signing and key-management.  IMHO they don't
even make sense inside database, they need an end-user
application.

In short, the goal of this code is not to be full OpenPGP
implementation, but just provide more flexible encryption
than the usual symmetric encryption.

So here is updated pgp_encrypt patch, with support for both
symmetric and public-key encryption.  Also, as Neil requested,
non-failing regression tests - unsupported parts are replaced
with empty test named '*-DISABLED'.

Note: it uses OpenSSL bignum code - so without OpenSSL
no public-key encryption.  Look below for plans for
fixing this dependance.

New names for symmetric-key functions:

    pgp_sym_encrypt(data, key)
    pgp_sym_decrypt(data, key)

Public key functions:

    pgp_pub_encrypt(data, key)
    pgp_pub_decrypt(data, key)
    pgp_pub_decrypt(data, key, psw)

Plus variants with 'arg' and for bytea.

Not mentioned in previous mails:

    armor(bytea) -> text
    dearmor(text) -> bytea

Apply and remove PGP Ascii Armor.



Implemented featureѕ:
* Elgamal-encryption keys - preferred public-key algo for OpenPGP.
* Password-protected secret keys.

Missing features (needs implementing ASAP):
* pgp_key_id() - Function to query the ID of a key, and
  the key ID from encrypted packet - so user can implement
  key rings on his own.

Maybe:
* As it does not parse sign packets, it does not support
  recipient cipher preferences.  It is not important for
  planned use-case, but maybe it would be nice to have.

Does not support (and I see no need):
* RSA-encrypt and sign+encrypt keys - they are deprecated in
  RFC, GnuPG does not even give option to generate them, so IMO
  it's fine to ignore them.
* Picking a key from list of keys - accepting a keyring in
  place of key.
* Several encryption subkeys under one key.
* Any form of signing - that also means no key integrity checks
  (eg - does the subkey belong to master key).


Now, the code is significant in the following respect - the
module 'contrib/pgcrypto' is complete.  As I said, I don't
think rest of the OpenPGP makes sense here, and there isn't
any other generally-useful crypto functionality that would be
good to have.


Still, there are various things to do:

1. Polishing of the new PGP code.
2. Rework documentation and add FAQ.
3. Try to extract OpenSSL and zlib settings from main
   PostgreSQL config.  Basically that means that ./configure
   should put them to separate variables, not into main CFLAGS.
4. Include strong RNG. Yarrow, Fortuna, ?.
   This is for both symmetricy and public-key encryption.
5. Include bignum code (only ops needed for Elgamal).
   This is for public-key encrypt/decrypt.
6. Propose crypt(), gen_salt() into mainline for 8.2.
   They are self-contained, except crypt-md5 depends on a MD5
   implentation, which already exists in mainline.


Points 4 and 5 are suspicious - I'm not yet sure it can
be done in manageable way.  But without those, the PGP
functions are unusable in default build.  It may not be that
bad - Yarrow/Fortuna seem to be small pieces only requiring
external cipher and hash, which we have.  And I could
drop parts of libtommath into separate directory.  But
I don't think such things should be done in hurry.

In the end, I think the code can go into 8.1 in current state -
requiring OpenSSL.  I'll look into dependencies in
8.2 timeframe.

--
marko

On Mon, Jul 04, 2005 at 04:59:58PM +1000, Neil Conway wrote:
> Marko Kreen wrote:
> >Please use following updated patch instead.
> >
> >It implements utf8 conversion, fixes couple of bugs and has
> >many code and comment cleanups.
>
> The regression tests don't pass on my box. With the default Makefile,
> there are a lot of errors WRT "no strong random source". After editing
> the Makefile to make use the "random" device, I get the attached
> regression.diffs.

I don't understand the regression of the dashes in armor test,
kinda seems you have different psql than in CVS.

But for the regression with 'random = dev', I'd say you did not
do 'make clean' after changing Makefile.  It only changes flags
to random.c, so if you have existing random.o, it wont be
recompiled.

> While I understand the need to make sure people use a reasonably strong
> crypto source, it would be nice if the regression tests passed out of
> the box.

Look for the 'pgp_encrypt v3' mail I sent.  There I use special
empty tests for disabled functionality, that succeed, but
hopefully show user through naming that the functionality
is missing.

--
marko