Add support for file inclusions in HBA and ident configuration files

pg_hba.conf and pg_ident.conf gain support for three record keywords:
- "include", to include a file.
- "include_if_exists", to include a file, ignoring it if missing.
- "include_dir", to include a directory of files.  These are classified
by name (C locale, mostly) and need to be prefixed by ".conf", hence
following the same rules as GUCs.

This commit relies on the refactoring pieces done in efc9816, ad6c528,
783e8c6 and 1b73d0b, adding a small wrapper to build a list of
TokenizedAuthLines (tokenize_include_file), and the code is shaped to
offer some symmetry with what is done for GUCs with the same options.

pg_hba_file_rules and pg_ident_file_mappings gain a new field called
file_name, to track from which file a record is located, taking
advantage of the addition of rule_number in c591300 to offer an
organized view of the HBA or ident records loaded.

Bump catalog version.

Author: Julien Rouhaud
Reviewed-by: Michael Paquier
Discussion: https://postgr.es/m/20220223045959.35ipdsvbxcstrhya@jrouhaud

2 files changed, 92 insertions, 16 deletions

diff --git a/doc/src/sgml/client-auth.sgml b/doc/src/sgml/client-auth.sgml
index 32d5d458638..3b2a0f8fb05 100644
--- a/doc/src/sgml/client-auth.sgml
+++ b/doc/src/sgml/client-auth.sgml
@@ -89,8 +89,8 @@
   </para>
 
   <para>
-   Each record specifies a connection type, a client IP address range
-   (if relevant for the connection type), a database name, a user name,
+   Each authentication record specifies a connection type, a client IP address
+   range (if relevant for the connection type), a database name, a user name,
    and the authentication method to be used for connections matching
    these parameters. The first record with a matching connection type,
    client address, requested database, and user name is used to perform
@@ -101,19 +101,37 @@
   </para>
 
   <para>
+   Each record can be an include directive or an authentication record.
+   Include directives specify files that can be included, that contain
+   additional records. The records will be inserted in place of the
+   include records. These records only contain two fields:
+   <literal>include</literal>, <literal>include_if_exists</literal> or
+   <literal>include_dir</literal> directive and the file or directory to be
+   included. The file or directory can be a relative of absolute path, and can
+   be double-quoted.  For the <literal>include_dir</literal> form, all files
+   not starting with a <literal>.</literal> and ending with
+   <literal>.conf</literal> will be included. Multiple files within an include
+   directory are processed in file name order (according to C locale rules,
+   i.e., numbers before letters, and uppercase letters before lowercase ones).
+  </para>
+
+  <para>
    A record can have several formats:
 <synopsis>
-local         <replaceable>database</replaceable>  <replaceable>user</replaceable>  <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
-host          <replaceable>database</replaceable>  <replaceable>user</replaceable>  <replaceable>address</replaceable>     <replaceable>auth-method</replaceable>  <optional><replaceable>auth-options</replaceable></optional>
-hostssl       <replaceable>database</replaceable>  <replaceable>user</replaceable>  <replaceable>address</replaceable>     <replaceable>auth-method</replaceable>  <optional><replaceable>auth-options</replaceable></optional>
-hostnossl     <replaceable>database</replaceable>  <replaceable>user</replaceable>  <replaceable>address</replaceable>     <replaceable>auth-method</replaceable>  <optional><replaceable>auth-options</replaceable></optional>
-hostgssenc    <replaceable>database</replaceable>  <replaceable>user</replaceable>  <replaceable>address</replaceable>     <replaceable>auth-method</replaceable>  <optional><replaceable>auth-options</replaceable></optional>
-hostnogssenc  <replaceable>database</replaceable>  <replaceable>user</replaceable>  <replaceable>address</replaceable>     <replaceable>auth-method</replaceable>  <optional><replaceable>auth-options</replaceable></optional>
-host          <replaceable>database</replaceable>  <replaceable>user</replaceable>  <replaceable>IP-address</replaceable>  <replaceable>IP-mask</replaceable>      <replaceable>auth-method</replaceable>  <optional><replaceable>auth-options</replaceable></optional>
-hostssl       <replaceable>database</replaceable>  <replaceable>user</replaceable>  <replaceable>IP-address</replaceable>  <replaceable>IP-mask</replaceable>      <replaceable>auth-method</replaceable>  <optional><replaceable>auth-options</replaceable></optional>
-hostnossl     <replaceable>database</replaceable>  <replaceable>user</replaceable>  <replaceable>IP-address</replaceable>  <replaceable>IP-mask</replaceable>      <replaceable>auth-method</replaceable>  <optional><replaceable>auth-options</replaceable></optional>
-hostgssenc    <replaceable>database</replaceable>  <replaceable>user</replaceable>  <replaceable>IP-address</replaceable>  <replaceable>IP-mask</replaceable>      <replaceable>auth-method</replaceable>  <optional><replaceable>auth-options</replaceable></optional>
-hostnogssenc  <replaceable>database</replaceable>  <replaceable>user</replaceable>  <replaceable>IP-address</replaceable>  <replaceable>IP-mask</replaceable>      <replaceable>auth-method</replaceable>  <optional><replaceable>auth-options</replaceable></optional>
+local               <replaceable>database</replaceable>  <replaceable>user</replaceable>  <replaceable>auth-method</replaceable> <optional><replaceable>auth-options</replaceable></optional>
+host                <replaceable>database</replaceable>  <replaceable>user</replaceable>  <replaceable>address</replaceable>     <replaceable>auth-method</replaceable>  <optional><replaceable>auth-options</replaceable></optional>
+hostssl             <replaceable>database</replaceable>  <replaceable>user</replaceable>  <replaceable>address</replaceable>     <replaceable>auth-method</replaceable>  <optional><replaceable>auth-options</replaceable></optional>
+hostnossl           <replaceable>database</replaceable>  <replaceable>user</replaceable>  <replaceable>address</replaceable>     <replaceable>auth-method</replaceable>  <optional><replaceable>auth-options</replaceable></optional>
+hostgssenc          <replaceable>database</replaceable>  <replaceable>user</replaceable>  <replaceable>address</replaceable>     <replaceable>auth-method</replaceable>  <optional><replaceable>auth-options</replaceable></optional>
+hostnogssenc        <replaceable>database</replaceable>  <replaceable>user</replaceable>  <replaceable>address</replaceable>     <replaceable>auth-method</replaceable>  <optional><replaceable>auth-options</replaceable></optional>
+host                <replaceable>database</replaceable>  <replaceable>user</replaceable>  <replaceable>IP-address</replaceable>  <replaceable>IP-mask</replaceable>      <replaceable>auth-method</replaceable>  <optional><replaceable>auth-options</replaceable></optional>
+hostssl             <replaceable>database</replaceable>  <replaceable>user</replaceable>  <replaceable>IP-address</replaceable>  <replaceable>IP-mask</replaceable>      <replaceable>auth-method</replaceable>  <optional><replaceable>auth-options</replaceable></optional>
+hostnossl           <replaceable>database</replaceable>  <replaceable>user</replaceable>  <replaceable>IP-address</replaceable>  <replaceable>IP-mask</replaceable>      <replaceable>auth-method</replaceable>  <optional><replaceable>auth-options</replaceable></optional>
+hostgssenc          <replaceable>database</replaceable>  <replaceable>user</replaceable>  <replaceable>IP-address</replaceable>  <replaceable>IP-mask</replaceable>      <replaceable>auth-method</replaceable>  <optional><replaceable>auth-options</replaceable></optional>
+hostnogssenc        <replaceable>database</replaceable>  <replaceable>user</replaceable>  <replaceable>IP-address</replaceable>  <replaceable>IP-mask</replaceable>      <replaceable>auth-method</replaceable>  <optional><replaceable>auth-options</replaceable></optional>
+include             <replaceable>file</replaceable>
+include_if_exists   <replaceable>file</replaceable>
+include_dir         <replaceable>directory</replaceable>
 </synopsis>
    The meaning of the fields is as follows:
 
@@ -655,6 +673,39 @@ openssl x509 -in myclient.crt -noout --subject -nameopt RFC2253 | sed "s/^subjec
       </para>
      </listitem>
     </varlistentry>
+
+    <varlistentry>
+     <term><literal>include</literal></term>
+     <listitem>
+      <para>
+       This line will be replaced by the contents of the given file.
+      </para>
+     </listitem>
+    </varlistentry>
+
+    <varlistentry>
+     <term><literal>include_if_exists</literal></term>
+     <listitem>
+      <para>
+       This line will be replaced by the content of the given file if the
+       file exists. Otherwise, a message is logged to indicate that the file
+       has been skipped.
+      </para>
+     </listitem>
+    </varlistentry>
+
+    <varlistentry>
+     <term><literal>include_dir</literal></term>
+     <listitem>
+      <para>
+       This line will be replaced by the contents of all the files found in
+       the directory, if they don't start with a <literal>.</literal> and end
+       with <literal>.conf</literal>, processed in file name order (according
+       to C locale rules, i.e., numbers before letters, and uppercase letters
+       before lowercase ones).
+      </para>
+     </listitem>
+    </varlistentry>
    </variablelist>
   </para>
 
@@ -863,9 +914,12 @@ local   db1,db2,@demodbs  all                                   md5
    cluster's data directory.  (It is possible to place the map file
    elsewhere, however; see the <xref linkend="guc-ident-file"/>
    configuration parameter.)
-   The ident map file contains lines of the general form:
+   The ident map file contains lines of the general forms:
 <synopsis>
 <replaceable>map-name</replaceable> <replaceable>system-username</replaceable> <replaceable>database-username</replaceable>
+<replaceable>include</replaceable> <replaceable>file</replaceable>
+<replaceable>include_if_exists</replaceable> <replaceable>file</replaceable>
+<replaceable>include_dir</replaceable> <replaceable>directory</replaceable>
 </synopsis>
    Comments, whitespace and line continuations are handled in the same way as in
    <filename>pg_hba.conf</filename>.  The
@@ -876,6 +930,10 @@ local   db1,db2,@demodbs  all                                   md5
    used repeatedly to specify multiple user-mappings within a single map.
   </para>
   <para>
+   As for <filename>pg_hba.conf</filename>, the lines in this file can
+   be include directives, following the same rules.
+  </para>
+  <para>
    There is no restriction regarding how many database users a given
    operating system user can correspond to, nor vice versa.  Thus, entries
    in a map should be thought of as meaning <quote>this operating system
diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml
index 7c716fe3276..d38b42c5cd5 100644
--- a/doc/src/sgml/system-views.sgml
+++ b/doc/src/sgml/system-views.sgml
@@ -1004,10 +1004,19 @@
 
      <row>
       <entry role="catalog_table_entry"><para role="column_definition">
+       <structfield>file_name</structfield> <type>text</type>
+      </para>
+      <para>
+       Name of the file containing this rule
+      </para></entry>
+     </row>
+
+     <row>
+      <entry role="catalog_table_entry"><para role="column_definition">
        <structfield>line_number</structfield> <type>int4</type>
       </para>
       <para>
-       Line number of this rule in <filename>pg_hba.conf</filename>
+       Line number of this rule in <literal>file_name</literal>
       </para></entry>
      </row>
 
@@ -1154,10 +1163,19 @@
 
      <row>
       <entry role="catalog_table_entry"><para role="column_definition">
+       <structfield>file_name</structfield> <type>text</type>
+      </para>
+      <para>
+       Name of the file containing this map
+      </para></entry>
+     </row>
+
+     <row>
+      <entry role="catalog_table_entry"><para role="column_definition">
        <structfield>line_number</structfield> <type>int4</type>
       </para>
       <para>
-       Line number of this map in <filename>pg_ident.conf</filename>
+       Line number of this map in <literal>file_name</literal>
       </para></entry>
      </row>