Add an enforcement mechanism for global object names in regression tests.

In commit 18555b132 we tentatively established a rule that regression
tests should use names containing "regression" for databases, and names
starting with "regress_" for all other globally-visible object names, so
as to circumscribe the side-effects that "make installcheck" could have
on an existing installation.

This commit adds a simple enforcement mechanism for that rule: if the code
is compiled with ENFORCE_REGRESSION_TEST_NAME_RESTRICTIONS defined, it
will emit a warning (not an error) whenever a database, role, tablespace,
subscription, or replication origin name is created that doesn't obey the
rule.  Running one or more buildfarm members with that symbol defined
should be enough to catch new violations, at least in the regular
regression tests.  Most TAP tests wouldn't notice such warnings, but
that's actually fine because TAP tests don't execute against an existing
server anyway.

Since it's already the case that running src/test/modules/ tests in
installcheck mode is deprecated, we can use that as a home for tests
that seem unsafe to run against an existing server, such as tests that
might have side-effects on existing roles.  Document that (though this
commit doesn't in itself make it any less safe than before).

Update regress.sgml to define these restrictions more clearly, and
to clean up assorted lack-of-up-to-date-ness in its descriptions of
the available regression tests.

Discussion: https://postgr.es/m/16638.1468620817@sss.pgh.pa.us

1 files changed, 86 insertions, 19 deletions

diff --git a/doc/src/sgml/regress.sgml b/doc/src/sgml/regress.sgml
index 448e2d19f72..7b682132668 100644
--- a/doc/src/sgml/regress.sgml
+++ b/doc/src/sgml/regress.sgml
@@ -47,7 +47,7 @@ make check
 <screen>
 <computeroutput>
 =======================
- All 115 tests passed.
+ All 193 tests passed.
 =======================
 </computeroutput>
 </screen>
@@ -98,7 +98,7 @@ make MAX_CONNECTIONS=10 check
 
   <para>
    To run the tests after installation (see <xref linkend="installation"/>),
-   initialize a data area and start the
+   initialize a data directory and start the
    server as explained in <xref linkend="runtime"/>, then type:
 <screen>
 make installcheck
@@ -116,10 +116,10 @@ make installcheck-parallel
 
   <para>
    The tests will also transiently create some cluster-wide objects, such as
-   roles and tablespaces.  These objects will have names beginning with
-   <literal>regress_</literal>.  Beware of using <literal>installcheck</literal>
-   mode in installations that have any actual users or tablespaces named
-   that way.
+   roles, tablespaces, and subscriptions.  These objects will have names
+   beginning with <literal>regress_</literal>.  Beware of
+   using <literal>installcheck</literal> mode with an installation that has
+   any actual global objects named that way.
   </para>
   </sect2>
 
@@ -130,7 +130,7 @@ make installcheck-parallel
    The <literal>make check</literal> and <literal>make installcheck</literal> commands
    run only the <quote>core</quote> regression tests, which test built-in
    functionality of the <productname>PostgreSQL</productname> server.  The source
-   distribution also contains additional test suites, most of them having
+   distribution contains many additional test suites, most of them having
    to do with add-on functionality such as optional procedural languages.
   </para>
 
@@ -146,9 +146,24 @@ make installcheck-world
    already-installed server, respectively, just as previously explained
    for <literal>make check</literal> and <literal>make installcheck</literal>.  Other
    considerations are the same as previously explained for each method.
-   Note that <literal>make check-world</literal> builds a separate temporary
-   installation tree for each tested module, so it requires a great deal
-   more time and disk space than <literal>make installcheck-world</literal>.
+   Note that <literal>make check-world</literal> builds a separate instance
+   (temporary data directory) for each tested module, so it requires more
+   time and disk space than <literal>make installcheck-world</literal>.
+  </para>
+
+  <para>
+   On a modern machine with multiple CPU cores and no tight operating-system
+   limits, you can make things go substantially faster with parallelism.
+   The recipe that most PostgreSQL developers actually use for running all
+   tests is something like
+<screen>
+make check-world -j8 >/dev/null
+</screen>
+   with a <option>-j</option> limit near to or a bit more than the number
+   of available cores.  Discarding <systemitem>stdout</systemitem>
+   eliminates chatter that's not interesting when you just want to verify
+   success.  (In case of failure, the <systemitem>stderr</systemitem>
+   messages are usually enough to determine where to look closer.)
   </para>
 
   <para>
@@ -166,8 +181,7 @@ make installcheck-world
   <itemizedlist>
    <listitem>
     <para>
-     Regression tests for optional procedural languages (other than
-     <application>PL/pgSQL</application>, which is tested by the core tests).
+     Regression tests for optional procedural languages.
      These are located under <filename>src/pl</filename>.
     </para>
    </listitem>
@@ -186,27 +200,49 @@ make installcheck-world
    </listitem>
    <listitem>
     <para>
+     Tests for core-supported authentication methods,
+     located in <filename>src/test/authentication</filename>.
+     (See below for additional authentication-related tests.)
+    </para>
+   </listitem>
+   <listitem>
+    <para>
      Tests stressing behavior of concurrent sessions,
      located in <filename>src/test/isolation</filename>.
     </para>
    </listitem>
    <listitem>
     <para>
-     Tests of client programs under <filename>src/bin</filename>.  See
-     also <xref linkend="regress-tap"/>.
+     Tests for crash recovery and physical replication,
+     located in <filename>src/test/recovery</filename>.
+    </para>
+   </listitem>
+   <listitem>
+    <para>
+     Tests for logical replication,
+     located in <filename>src/test/subscription</filename>.
+    </para>
+   </listitem>
+   <listitem>
+    <para>
+     Tests of client programs, located under <filename>src/bin</filename>.
     </para>
    </listitem>
   </itemizedlist>
 
   <para>
-   When using <literal>installcheck</literal> mode, these tests will destroy any
-   existing databases named <literal>pl_regression</literal>,
-   <literal>contrib_regression</literal>, <literal>isolation_regression</literal>,
-   <literal>ecpg1_regression</literal>, or <literal>ecpg2_regression</literal>, as well as
-   <literal>regression</literal>.
+   When using <literal>installcheck</literal> mode, these tests will create
+   and destroy test databases whose names
+   include <literal>regression</literal>, for
+   example <literal>pl_regression</literal>
+   or <literal>contrib_regression</literal>.  Beware of
+   using <literal>installcheck</literal> mode with an installation that has
+   any non-test databases named that way.
   </para>
 
   <para>
+   Some of these auxiliary test suites use the TAP infrastructure explained
+   in <xref linkend="regress-tap"/>.
    The TAP-based tests are run only when PostgreSQL was configured with the
    option <option>--enable-tap-tests</option>.  This is recommended for
    development, but can be omitted if there is no suitable Perl installation.
@@ -259,6 +295,17 @@ make check-world PG_TEST_EXTRA='kerberos ldap ssl'
    configuration are not run even if they are mentioned in
    <varname>PG_TEST_EXTRA</varname>.
   </para>
+
+  <para>
+   In addition, there are tests in <filename>src/test/modules</filename>
+   which will be run by <literal>make check-world</literal> but not
+   by <literal>make installcheck-world</literal>.  This is because they
+   install non-production extensions or have other side-effects that are
+   considered undesirable for a production installation.  You can
+   use <literal>make install</literal> and <literal>make
+   installcheck</literal> in one of those subdirectories if you wish,
+   but it's not recommended to do so with a non-test server.
+  </para>
   </sect2>
 
   <sect2>
@@ -737,6 +784,26 @@ make check PROVE_TESTS='t/001_test1.pl t/003_test3.pl'
     The TAP tests require the Perl module <literal>IPC::Run</literal>.
     This module is available from CPAN or an operating system package.
    </para>
+
+   <para>
+    Generically speaking, the TAP tests will test the executables in a
+    previously-installed installation tree if you say <literal>make
+    installcheck</literal>, or will build a new local installation tree from
+    current sources if you say <literal>make check</literal>.  In either
+    case they will initialize a local instance (data directory) and
+    transiently run a server in it.  Some of these tests run more than one
+    server.  Thus, these tests can be fairly resource-intensive.
+   </para>
+
+   <para>
+    It's important to realize that the TAP tests will start test server(s)
+    even when you say <literal>make installcheck</literal>; this is unlike
+    the traditional non-TAP testing infrastructure, which expects to use an
+    already-running test server in that case.  Some PostgreSQL
+    subdirectories contain both traditional-style and TAP-style tests,
+    meaning that <literal>make installcheck</literal> will produce a mix of
+    results from temporary servers and the already-running test server.
+   </para>
   </sect1>
 
   <sect1 id="regress-coverage">