Add key management system

This adds a key management system that stores (currently) two data
encryption keys of length 128, 192, or 256 bits.  The data keys are
AES256 encrypted using a key encryption key, and validated via GCM
cipher mode.  A command to obtain the key encryption key must be
specified at initdb time, and will be run at every database server
start.  New parameters allow a file descriptor open to the terminal to
be passed.  pg_upgrade support has also been added.

Discussion: https://postgr.es/m/CA+fd4k7q5o6Nc_AaX6BcYM9yqTbC6_pnH-6nSD=54Zp6NBQTCQ@mail.gmail.com
Discussion: https://postgr.es/m/20201202213814.GG20285@momjian.us

Author: Masahiko Sawada, me, Stephen Frost

10 files changed, 258 insertions, 3 deletions

diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 4b60382778f..426928f6800 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -7816,6 +7816,52 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
     </variablelist>
    </sect1>
 
+   <sect1 id="runtime-config-encryption">
+    <title>Cluster File Encryption</title>
+
+    <variablelist>
+     <varlistentry id="guc-cluster-key-command" xreflabel="cluster_key_command">
+      <term><varname>cluster_key_command</varname> (<type>string</type>)
+      <indexterm>
+       <primary><varname>cluster_key_command</varname> configuration parameter</primary>
+      </indexterm>
+      </term>
+      <listitem>
+       <para>
+        This option specifies an external command to obtain the cluster-level
+        key for cluster file encryption during server initialization and
+        server start.
+       </para>
+       <para>
+        The command must print the cluster key to the standard output as
+        64 hexadecimal characters, and exit with code 0.  The command
+        can prompt for the passphrase or PIN from the terminal if
+        <option>--authprompt</option> is used.  In the parameter value,
+        <literal>%R</literal> represents the file descriptor number opened
+        to the terminal that started the server.  A file descriptor is only
+        available if enabled at server start.  If <literal>%R</literal>
+        is used and no file descriptor is available, the server will not
+        start.  Value <literal>%p</literal> is replaced by a pre-defined
+        prompt string.  Value <literal>%d</literal> is replaced by the
+        directory containing the keys;  this is useful if the command
+        must create files with the keys, e.g., to store a cluster-level
+        key encryped by a key stored in a hardware security module.
+        (Write <literal>%%</literal> for a literal <literal>%</literal>.)
+        Note that the prompt string will probably contain whitespace,
+        so be sure to quote its use adequately.  Newlines are stripped
+        from the end of the output if present.
+       </para>
+       <para>
+        This parameter can only be set by
+        <application>initdb</application>, in the
+        <filename>postgresql.conf</filename> file, or on the server
+        command line.
+       </para>
+      </listitem>
+     </varlistentry>
+    </variablelist>
+   </sect1>
+
    <sect1 id="runtime-config-client">
     <title>Client Connection Defaults</title>
 
@@ -9637,6 +9683,22 @@ dynamic_library_path = 'C:\tools\postgresql;H:\my_project\lib;$libdir'
       </listitem>
      </varlistentry>
 
+     <varlistentry id="guc-file-encryption-keylen" xreflabel="file_encryption_keylen">
+      <term><varname>file_encryption_keylen</varname> (<type>boolean</type>)
+      <indexterm>
+       <primary>Cluster file encryption key length</primary>
+      </indexterm>
+      </term>
+      <listitem>
+       <para>
+        Reports the bit length of the cluster file
+        encryption key, or zero if disabled.  See <xref
+        linkend="app-initdb-cluster-key-command"/> for more
+        information.
+       </para>
+      </listitem>
+     </varlistentry>
+
      <varlistentry id="guc-data-directory-mode" xreflabel="data_directory_mode">
       <term><varname>data_directory_mode</varname> (<type>integer</type>)
       <indexterm>
diff --git a/doc/src/sgml/database-encryption.sgml b/doc/src/sgml/database-encryption.sgml
new file mode 100644
index 00000000000..f938c9f574a
--- /dev/null
+++ b/doc/src/sgml/database-encryption.sgml
@@ -0,0 +1,97 @@
+<!-- doc/src/sgml/database-encryption.sgml -->
+
+<chapter id="database-file-encryption">
+ <title>Cluster File Encryption</title>
+
+ <indexterm zone="database-file-encryption">
+  <primary>Cluster File Encryption</primary>
+ </indexterm>
+
+ <para>
+  The purpose of cluster file encryption is to prevent users with read
+  access to the directories used to store database files and write-ahead
+  log from being able to access the data stored in those files.
+  For example, when using cluster file encryption, users who have read
+  access to the cluster directories for backup purposes will not be able
+  to decrypt the data stored in the these files.
+ </para>
+
+ <para>
+  Cluster file encryption uses two levels of encryption.  The first level
+  is data encryption keys, specifically keys zero and one.  Key zero is
+  the key used to encrypt database heap and index files which are stored in
+  the file system, plus temporary files created during database operation.
+  Key one is used to encrypt write-ahead log (WAL) files.  Two different
+  keys are used so that primary and standby servers can use different zero
+  (heap/index/temp) keys, but the same one (WAL) key, so that these keys
+  can eventually be rotated by switching the primary to the standby as
+  and then changing the WAL key.
+ </para>
+
+ <para>
+  The second level of encryption is a key used to encrypt first-level
+  keys.  This type of key is often referred to as a Key Encryption Key
+  (<acronym>KEK</acronym>).  This key is <emphasis>not</emphasis> stored
+  in the file system, but provided at <command>initdb</command> time and
+  each time the server is started.  This key prevents anyone with access
+  to the database directories from decrypting the data because they do
+  not know the second-level key which encrypted the first-level keys
+  which encrypted the database cluster files.
+ </para>
+
+ <sect1 id="encryption-file-encryption">
+  <title>Initialization</title>
+
+  <para>
+   Cluster file encryption is enabled when
+   <productname>PostgreSQL</productname> is built
+   with <literal>--with-openssl</literal> and <xref
+   linkend="app-initdb-cluster-key-command"/> is specified
+   during <command>initdb</command>.  The cluster key
+   provided by the <option>--cluster-key-command</option>
+   option during <command>initdb</command> and the one generated
+   by <xref linkend="guc-cluster-key-command"/> in the
+   <filename>postgresql.conf</filename> must match for the database
+   cluster to start. Note that the cluster key command
+   passed to <command>initdb</command> must return a key of
+   64 hexadecimal characters. For example.
+<programlisting>
+initdb -D dbname --cluster-key-command='ckey_passphrase.sh'
+</programlisting>
+  </para>
+ </sect1>
+
+ <sect1 id="key-encryption-key">
+  <title>Internals</title>
+
+  <para>
+   During the <command>initdb</command> process, if
+   <option>--cluster-key-command</option> is specified, two data-level
+   encryption keys are created.   These two keys are then encrypted with
+   the key enryption key (KEK) supplied by the cluster key command before
+   being stored in the database directory.  The key or passphrase that
+   derives the key must be supplied from the terminal or stored in a
+   trusted key store, such as key vault software, hardware security module.
+  </para>
+
+  <para>
+   If the <productname>PostgreSQL</productname> server has
+   been initialized to require a cluster key, each time the
+   server starts the <filename>postgresql.conf</filename>
+   <varname>cluster_key_command</varname> command will be executed
+   and the cluster key retrieved.  The data encryption keys in the
+   <filename>pg_cryptokeys</filename> directory will then be decrypted
+   using the supplied key and integrity-checked to ensure it
+   matches the initdb-supplied key.  If this check fails, the
+   server will refuse to start.
+  </para>
+
+  <para>
+   The data encryption keys are randomly generated and are of 128, 192,
+   or 256-bits in length.  They are encrypted by the key encryption key
+   (KEK) using Advanced Encryption Standard (<acronym>AES256</acronym>)
+   encryption in Galois/Counter Mode (<acronym>GCM</acronym>), which also
+   provides KEK authentication.
+  </para>
+ </sect1>
+</chapter>
diff --git a/doc/src/sgml/filelist.sgml b/doc/src/sgml/filelist.sgml
index 38e8aa0bbf9..b96f4ace6cf 100644
--- a/doc/src/sgml/filelist.sgml
+++ b/doc/src/sgml/filelist.sgml
@@ -49,6 +49,7 @@
 <!ENTITY wal           SYSTEM "wal.sgml">
 <!ENTITY logical-replication    SYSTEM "logical-replication.sgml">
 <!ENTITY jit    SYSTEM "jit.sgml">
+<!ENTITY database-encryption SYSTEM "database-encryption.sgml">
 
 <!-- programmer's guide -->
 <!ENTITY bgworker   SYSTEM "bgworker.sgml">
diff --git a/doc/src/sgml/installation.sgml b/doc/src/sgml/installation.sgml
index 0ac1cb99999..bcc80240498 100644
--- a/doc/src/sgml/installation.sgml
+++ b/doc/src/sgml/installation.sgml
@@ -976,8 +976,9 @@ build-postgresql:
        <listitem>
         <para>
          Build with support for <acronym>SSL</acronym> (encrypted)
-         connections. This requires the <productname>OpenSSL</productname>
-         package to be installed.  <filename>configure</filename> will check
+         connections and cluster file encryption. This requires the
+         <productname>OpenSSL</productname> package to be installed.
+         <filename>configure</filename> will check
          for the required header files and libraries to make sure that
          your <productname>OpenSSL</productname> installation is sufficient
          before proceeding.
diff --git a/doc/src/sgml/postgres.sgml b/doc/src/sgml/postgres.sgml
index 730d5fdc348..0ea7da604b3 100644
--- a/doc/src/sgml/postgres.sgml
+++ b/doc/src/sgml/postgres.sgml
@@ -171,6 +171,7 @@ break is not needed in a wider output rendering.
   &wal;
   &logical-replication;
   &jit;
+  &database-encryption;
   &regress;
 
  </part>
diff --git a/doc/src/sgml/ref/initdb.sgml b/doc/src/sgml/ref/initdb.sgml
index 385ac251506..44a2d69d326 100644
--- a/doc/src/sgml/ref/initdb.sgml
+++ b/doc/src/sgml/ref/initdb.sgml
@@ -163,6 +163,17 @@ PostgreSQL documentation
       </listitem>
      </varlistentry>
 
+     <varlistentry id="app-initdb-cluster-key-command" xreflabel="cluster key command">
+      <term><option>--cluster-key-command=<replaceable class="parameter">command</replaceable></option></term>
+      <listitem>
+       <para>
+        This option specifies an external command to obtain the cluster-level
+        key for cluster file encryption during server initialization and
+        server start;  see <xref linkend="guc-cluster-key-command"/> for details.
+       </para>
+      </listitem>
+     </varlistentry>
+
      <varlistentry>
       <term><option>-D <replaceable class="parameter">directory</replaceable></option></term>
       <term><option>--pgdata=<replaceable class="parameter">directory</replaceable></option></term>
@@ -223,6 +234,18 @@ PostgreSQL documentation
       </listitem>
      </varlistentry>
 
+     <varlistentry id="app-initdb-file-encryption-keylen"
+      xreflabel="file encryption">
+      <term><option>-K</option></term>
+      <term><option>--file-encryption-keylen</option></term>
+      <listitem>
+       <para>
+        Specifies the number of bits for the file encryption keys.  The
+        default is 128 bits.
+       </para>
+      </listitem>
+     </varlistentry>
+
      <varlistentry>
       <term><option>--locale=<replaceable>locale</replaceable></option></term>
       <listitem>
@@ -286,6 +309,17 @@ PostgreSQL documentation
      </varlistentry>
 
      <varlistentry>
+      <term><option>-R</option></term>
+      <term><option>--authprompt</option></term>
+      <listitem>
+       <para>
+        Allows the <option>--cluster-key-command</option> command
+        to prompt for a passphrase or PIN.
+       </para>
+      </listitem>
+     </varlistentry>
+
+     <varlistentry>
       <term><option>-S</option></term>
       <term><option>--sync-only</option></term>
       <listitem>
@@ -308,6 +342,18 @@ PostgreSQL documentation
      </varlistentry>
 
      <varlistentry>
+      <term><option>-u <replaceable>datadir</replaceable></option></term>
+      <term><option>--copy-encryption-keys=<replaceable>datadir</replaceable></option></term>
+      <listitem>
+       <para>
+        Copies cluster file encryption keys from another cluster; required
+        when using <application>pg_upgrade</application> on a cluster
+        with cluster file encryption enabled.
+       </para>
+      </listitem>
+     </varlistentry>
+
+     <varlistentry>
       <term><option>-U <replaceable class="parameter">username</replaceable></option></term>
       <term><option>--username=<replaceable class="parameter">username</replaceable></option></term>
       <listitem>
diff --git a/doc/src/sgml/ref/pg_ctl-ref.sgml b/doc/src/sgml/ref/pg_ctl-ref.sgml
index 3946fa52eab..f04e417745f 100644
--- a/doc/src/sgml/ref/pg_ctl-ref.sgml
+++ b/doc/src/sgml/ref/pg_ctl-ref.sgml
@@ -38,6 +38,7 @@ PostgreSQL documentation
    <arg choice="opt"><option>-s</option></arg>
    <arg choice="opt"><option>-o</option> <replaceable>options</replaceable></arg>
    <arg choice="opt"><option>-p</option> <replaceable>path</replaceable></arg>
+   <arg choice="opt"><option>-R</option></arg>
    <arg choice="opt"><option>-c</option></arg>
   </cmdsynopsis>
 
@@ -72,6 +73,7 @@ PostgreSQL documentation
    <arg choice="opt"><option>-t</option> <replaceable>seconds</replaceable></arg>
    <arg choice="opt"><option>-s</option></arg>
    <arg choice="opt"><option>-o</option> <replaceable>options</replaceable></arg>
+   <arg choice="opt"><option>-R</option></arg>
    <arg choice="opt"><option>-c</option></arg>
   </cmdsynopsis>
 
@@ -374,6 +376,17 @@ PostgreSQL documentation
      </varlistentry>
 
      <varlistentry>
+      <term><option>-R</option></term>
+      <term><option>--authprompt</option></term>
+      <listitem>
+       <para>
+        Allows the <option>--cluster-key-command</option> command
+        to prompt for a passphrase or PIN.
+       </para>
+      </listitem>
+     </varlistentry>
+
+     <varlistentry>
       <term><option>-s</option></term>
       <term><option>--silent</option></term>
       <listitem>
diff --git a/doc/src/sgml/ref/pgupgrade.sgml b/doc/src/sgml/ref/pgupgrade.sgml
index 92e1d09a55c..98be3921cb1 100644
--- a/doc/src/sgml/ref/pgupgrade.sgml
+++ b/doc/src/sgml/ref/pgupgrade.sgml
@@ -168,6 +168,13 @@ PostgreSQL documentation
      </varlistentry>
 
      <varlistentry>
+      <term><option>-R</option></term>
+      <term><option>--authprompt</option></term>
+      <listitem><para>allows prompting for a passphrase or PIN
+      </para></listitem>
+     </varlistentry>
+
+     <varlistentry>
       <term><option>-s</option> <replaceable>dir</replaceable></term>
       <term><option>--socketdir=</option><replaceable>dir</replaceable></term>
       <listitem><para>directory to use for postmaster sockets during upgrade;
@@ -309,7 +316,9 @@ make prefix=/usr/local/pgsql.new install
      Again, use compatible <command>initdb</command>
      flags that match the old cluster. Many
      prebuilt installers do this step automatically. There is no need to
-     start the new cluster.
+     start the new cluster.  If upgrading a cluster that uses
+     cluster file encryption, the <command>initdb</command> option
+     <option>--copy-encryption-keys</option> must be specified.
     </para>
    </step>
 
@@ -838,6 +847,13 @@ psql --username=postgres --file=script.sql postgres
    is down.
   </para>
 
+  <para>
+   If the old cluster uses file encryption, the new cluster must use
+   the same keys, so <command>pg_upgrade</command> copies them to the
+   new cluster.  It is necessary to initialize the new cluster with
+   the same <varname>cluster_key_command</varname> and the same
+   file encryption key length.
+  </para>
  </refsect1>
 
  <refsect1>
diff --git a/doc/src/sgml/ref/postgres-ref.sgml b/doc/src/sgml/ref/postgres-ref.sgml
index 4aaa7abe1a2..805da81e073 100644
--- a/doc/src/sgml/ref/postgres-ref.sgml
+++ b/doc/src/sgml/ref/postgres-ref.sgml
@@ -298,6 +298,19 @@ PostgreSQL documentation
      </varlistentry>
 
      <varlistentry>
+      <term><option>-R <replaceable class="parameter">file-descriptor</replaceable></option></term>
+      <listitem>
+       <para>
+        Makes <command>postgres</command> prompt for a passphrase or PIN
+        from the specified open numeric file descriptor.  The descriptor
+        is closed after the key is read.  The file descriptor number
+        <literal>-1</literal> duplicates standard error for the terminal;
+        this is useful for single-user mode.
+       </para>
+      </listitem>
+     </varlistentry>
+
+     <varlistentry>
       <term><option>-s</option></term>
       <listitem>
        <para>
diff --git a/doc/src/sgml/storage.sgml b/doc/src/sgml/storage.sgml
index 3234adb639f..cdbc214a510 100644
--- a/doc/src/sgml/storage.sgml
+++ b/doc/src/sgml/storage.sgml
@@ -78,6 +78,11 @@ Item
 </row>
 
 <row>
+ <entry><filename>pg_cryptokeys</filename></entry>
+ <entry>Subdirectory containing file encryption keys</entry>
+</row>
+
+<row>
  <entry><filename>pg_dynshmem</filename></entry>
  <entry>Subdirectory containing files used by the dynamic shared memory
   subsystem</entry>