aboutsummaryrefslogtreecommitdiff
path: root/src
diff options
context:
space:
mode:
authorPeter Eisentraut <peter_e@gmx.net>2003-11-13 18:03:11 +0000
committerPeter Eisentraut <peter_e@gmx.net>2003-11-13 18:03:11 +0000
commitb9f5c93b75105d3554fbd4e66a1adf0c61cdc162 (patch)
treef3b08d037e99a1e721e747d8780a71d693c95f94 /src
parente873207fddb8bec25f7637f76ddcc4917320b0c0 (diff)
downloadpostgresql-b9f5c93b75105d3554fbd4e66a1adf0c61cdc162.tar.gz
postgresql-b9f5c93b75105d3554fbd4e66a1adf0c61cdc162.zip
Regenerate text files.
Diffstat (limited to 'src')
-rw-r--r--src/test/regress/README379
1 files changed, 208 insertions, 171 deletions
diff --git a/src/test/regress/README b/src/test/regress/README
index 86311be8983..2bebafa3d19 100644
--- a/src/test/regress/README
+++ b/src/test/regress/README
@@ -1,243 +1,280 @@
-Regression Tests
-
-Introduction
+ Regression Tests
The regression tests are a comprehensive set of tests for the SQL
-implementation in PostgreSQL. They test standard SQL operations as well as
-the extended capabilities of PostgreSQL. The test suite was originally
-developed by Jolly Chen and Andrew Yu, and was extensively revised and
-repackaged by Marc Fournier and Thomas Lockhart. From PostgreSQL 6.1 onward
-the regression tests are current for every official release.
+implementation in PostgreSQL. They test standard SQL operations as well as the
+extended capabilities of PostgreSQL. From PostgreSQL 6.1 onward, the regression
+tests are current for every official release.
- ------------------------------------------------------------------------
+-------------------------------------------------------------------------------
Running the Tests
-The regression test can be run against an already installed and running
-server, or using a temporary installation within the build tree.
-Furthermore, there is a "parallel" and a "sequential" mode for running the
-tests. The sequential method runs each test script in turn, whereas the
-parallel method starts up multiple server processes to run groups of tests
-in parallel. Parallel testing gives confidence that interprocess
-communication and locking are working correctly. For historical reasons, the
-sequential test is usually run against an existing installation and the
-parallel method against a temporary installation, but there are no technical
-reasons for this.
+The regression test can be run against an already installed and running server,
+or using a temporary installation within the build tree. Furthermore, there is
+a "parallel" and a "sequential" mode for running the tests. The sequential
+method runs each test script in turn, whereas the parallel method starts up
+multiple server processes to run groups of tests in parallel. Parallel testing
+gives confidence that interprocess communication and locking are working
+correctly. For historical reasons, the sequential test is usually run against
+an existing installation and the parallel method against a temporary
+installation, but there are no technical reasons for this.
To run the regression tests after building but before installation, type
-$ gmake check
+ gmake check
-in the top-level directory. (Or you can change to src/test/regress and run
-the command there.) This will first build several auxiliary files, such as
-platform-dependent "expected" files and some sample user-defined trigger
-functions, and then run the test driver script. At the end you should see
-something like
+in the top-level directory. (Or you can change to "src/test/regress" and run
+the command there.) This will first build several auxiliary files, such as some
+sample user-defined trigger functions, and then run the test driver script. At
+the end you should see something like
-======================
- All 77 tests passed.
-======================
+ ======================
+ All 93 tests passed.
+ ======================
-or otherwise a note about what tests failed. See the Section called Test
+or otherwise a note about which tests failed. See the Section called Test
Evaluation below for more.
- Note: Because this test method runs a temporary server, it will
- not work when you are the root user (the server will not start as
- root). If you already did the build as root, you do not have to
- start all over. Instead, make the regression test directory
- writable by some other user, log in as that user, and restart the
- tests. For example,
+Because this test method runs a temporary server, it will not work when you are
+the root user (since the server will not start as root). If you already did the
+build as root, you do not have to start all over. Instead, make the regression
+test directory writable by some other user, log in as that user, and restart
+the tests. For example
+
+ root# chmod -R a+w src/test/regress
+ root# chmod -R a+w contrib/spi
+ root# su - joeuser
+ joeuser$ cd top-level build directory
+ joeuser$ gmake check
+
+(The only possible "security risk" here is that other users might be able to
+alter the regression test results behind your back. Use common sense when
+managing user permissions.)
+
+Alternatively, run the tests after installation.
+
+The parallel regression test starts quite a few processes under your user ID.
+Presently, the maximum concurrency is twenty parallel test scripts, which means
+sixty processes: there's a server process, a psql, and usually a shell parent
+process for the psql for each test script. So if your system enforces a per-
+user limit on the number of processes, make sure this limit is at least
+seventy-five or so, else you may get random-seeming failures in the parallel
+test. If you are not in a position to raise the limit, you can cut down the
+degree of parallelism by setting the MAX_CONNECTIONS parameter. For example,
- root# chmod -R a+w src/test/regress
- root# chmod -R a+w contrib/spi
- root# su - joeuser
- joeuser$ cd <build top-level directory>
- joeuser$ gmake check
+ gmake MAX_CONNECTIONS=10 check
- (The only possible "security risk" here is that other users might
- be able to alter the regression test results behind your back. Use
- common sense when managing user permissions.)
+runs no more than ten tests concurrently.
- Alternatively, run the tests after installation.
+On some systems, the default Bourne-compatible shell ("/bin/sh") gets confused
+when it has to manage too many child processes in parallel. This may cause the
+parallel test run to lock up or fail. In such cases, specify a different
+Bourne-compatible shell on the command line, for example:
- Tip: On some systems, the default Bourne-compatible shell
- (/bin/sh) gets confused when it has to manage too many child
- processes in parallel. This may cause the parallel test run to
- lock up or fail. In such cases, specify a different
- Bourne-compatible shell on the command line, for example:
+ gmake SHELL=/bin/ksh check
- $ gmake SHELL=/bin/ksh check
+If no non-broken shell is available, you may be able to work around the problem
+by limiting the number of connections, as shown above.
To run the tests after installation, initialize a data area and start the
server, then type
-$ gmake installcheck
+ gmake installcheck
-The tests will expect to contact the server at the local host and the
-default port number, unless directed otherwise by PGHOST and PGPORT
-environment variables.
+The tests will expect to contact the server at the local host and the default
+port number, unless directed otherwise by PGHOST and PGPORT environment
+variables.
- ------------------------------------------------------------------------
+-------------------------------------------------------------------------------
Test Evaluation
Some properly installed and fully functional PostgreSQL installations can
-"fail" some of these regression tests due to platform-specific artifacts
-such as varying floating point representation and time zone support. The
-tests are currently evaluated using a simple diff comparison against the
-outputs generated on a reference system, so the results are sensitive to
-small system differences. When a test is reported as "failed", always
-examine the differences between expected and actual results; you may well
-find that the differences are not significant. Nonetheless, we still strive
-to maintain accurate reference files across all supported platforms, so it
-can be expected that all tests pass.
-
-The actual outputs of the regression tests are in files in the
-src/test/regress/results directory. The test script uses diff to compare
-each output file against the reference outputs stored in the
-src/test/regress/expected directory. Any differences are saved for your
-inspection in src/test/regress/regression.diffs. (Or you can run diff
-yourself, if you prefer.)
-
- ------------------------------------------------------------------------
+"fail" some of these regression tests due to platform-specific artifacts such
+as varying floating-point representation and time zone support. The tests are
+currently evaluated using a simple "diff" comparison against the outputs
+generated on a reference system, so the results are sensitive to small system
+differences. When a test is reported as "failed", always examine the
+differences between expected and actual results; you may well find that the
+differences are not significant. Nonetheless, we still strive to maintain
+accurate reference files across all supported platforms, so it can be expected
+that all tests pass.
+
+The actual outputs of the regression tests are in files in the "src/test/
+regress/results" directory. The test script uses "diff" to compare each output
+file against the reference outputs stored in the "src/test/regress/expected"
+directory. Any differences are saved for your inspection in "src/test/regress/
+regression.diffs". (Or you can run "diff" yourself, if you prefer.)
+
+-------------------------------------------------------------------------------
Error message differences
Some of the regression tests involve intentional invalid input values. Error
messages can come from either the PostgreSQL code or from the host platform
-system routines. In the latter case, the messages may vary between
-platforms, but should reflect similar information. These differences in
-messages will result in a "failed" regression test that can be validated by
-inspection.
+system routines. In the latter case, the messages may vary between platforms,
+but should reflect similar information. These differences in messages will
+result in a "failed" regression test that can be validated by inspection.
- ------------------------------------------------------------------------
+-------------------------------------------------------------------------------
Locale differences
-The tests expect to run in plain "C" locale. This should not cause any
-problems when you run the tests against a temporary installation, since the
-regression test driver takes care to start the server in C locale. However,
-if you run the tests against an already-installed server that is using non-C
-locale settings, you may see differences caused by varying rules for string
-sort order, formatting of numeric and monetary values, and so forth.
-
-In some locales the resulting differences are small and easily checked by
-inspection. However, in a locale that changes the rules for formatting of
-numeric values (typically by swapping the usage of commas and decimal
-points), entry of some data values will fail, resulting in extensive
-differences later in the tests where the missing data values are supposed to
-be used.
-
- ------------------------------------------------------------------------
+If you run the tests against an already-installed server that was initialized
+with a collation-order locale other than C, then there may be differences due
+to sort order and follow-up failures. The regression test suite is set up to
+handle this problem by providing alternative result files that together are
+known to handle a large number of locales. For example, for the char test, the
+expected file "char.out" handles the C and POSIX locales, and the file
+"char_1.out" handles many other locales. The regression test driver will
+automatically pick the best file to match against when checking for success and
+for computing failure differences. (This means that the regression tests cannot
+detect whether the results are appropriate for the configured locale. The tests
+will simply pick the one result file that works best.)
+
+If for some reason the existing expected files do not cover some locale, you
+can add a new file. The naming scheme is testname_digit.out. The actual digit
+is not significant. Remember that the regression test driver will consider all
+such files to be equally valid test results. If the test results are platform-
+specific, the technique described in the Section called Platform-specific
+comparison files should be used instead.
+
+-------------------------------------------------------------------------------
Date and time differences
-Some of the queries in the timestamp test will fail if you run the test on
-the day of a daylight-savings time changeover, or the day before or after
-one. These queries assume that the intervals between midnight yesterday,
-midnight today and midnight tomorrow are exactly twenty-four hours -- which
-is wrong if daylight-savings time went into or out of effect meanwhile.
-
-Most of the date and time results are dependent on the time zone
-environment. The reference files are generated for time zone PST8PDT
-(Berkeley, California) and there will be apparent failures if the tests are
-not run with that time zone setting. The regression test driver sets
-environment variable PGTZ to PST8PDT, which normally ensures proper results.
-However, your system must provide library support for the PST8PDT time zone,
-or the time zone-dependent tests will fail. To verify that your machine does
-have this support, type the following:
-
-$ env TZ=PST8PDT date
-
-The command above should have returned the current system time in the
-PST8PDT time zone. If the PST8PDT database is not available, then your
-system may have returned the time in GMT. If the PST8PDT time zone is not
-available, you can set the time zone rules explicitly:
+A few of the queries in the "horology" test will fail if you run the test on
+the day of a daylight-saving time changeover, or the day after one. These
+queries expect that the intervals between midnight yesterday, midnight today
+and midnight tomorrow are exactly twenty-four hours --- which is wrong if
+daylight-saving time went into or out of effect meanwhile.
-PGTZ='PST8PDT7,M04.01.0,M10.05.03'; export PGTZ
+ Note: Because USA daylight-saving time rules are used, this problem
+ always occurs on the first Sunday of April, the last Sunday of
+ October, and their following Mondays, regardless of when daylight-
+ saving time is in effect where you live. Also note that the problem
+ appears or disappears at midnight Pacific time (UTC-7 or UTC-8), not
+ midnight your local time. Thus the failure may appear late on
+ Saturday or persist through much of Tuesday, depending on where you
+ live.
-There appear to be some systems that do not accept the recommended syntax
-for explicitly setting the local time zone rules; you may need to use a
-different PGTZ setting on such machines.
+Most of the date and time results are dependent on the time zone environment.
+The reference files are generated for time zone PST8PDT (Berkeley, California),
+and there will be apparent failures if the tests are not run with that time
+zone setting. The regression test driver sets environment variable PGTZ to
+PST8PDT, which normally ensures proper results. However, your operating system
+must provide support for the PST8PDT time zone, or the time zone-dependent
+tests will fail. To verify that your machine does have this support, type the
+following:
-Some systems using older time zone libraries fail to apply daylight-savings
-corrections to dates before 1970, causing pre-1970 PDT times to be displayed
-in PST instead. This will result in localized differences in the test
-results.
+ env TZ=PST8PDT date
- ------------------------------------------------------------------------
+The command above should have returned the current system time in the PST8PDT
+time zone. If the PST8PDT time zone is not available, then your system may have
+returned the time in UTC. If the PST8PDT time zone is missing, you can set the
+time zone rules explicitly:
-Floating point differences
+ PGTZ='PST8PDT7,M04.01.0,M10.05.03'; export PGTZ
-Some of the tests involve computing 64-bit (double precision) numbers from
-table columns. Differences in results involving mathematical functions of
-double precision columns have been observed. The float8 and geometry tests
-are particularly prone to small differences across platforms, or even with
-different compiler optimization options. Human eyeball comparison is needed
-to determine the real significance of these differences which are usually 10
-places to the right of the decimal point.
+There appear to be some systems that do not accept the recommended syntax for
+explicitly setting the local time zone rules; you may need to use a different
+PGTZ setting on such machines.
-Some systems signal errors from pow() and exp() differently from the
-mechanism expected by the current PostgreSQL code.
+Some systems using older time-zone libraries fail to apply daylight-saving
+corrections to dates before 1970, causing pre-1970 PDT times to be displayed in
+PST instead. This will result in localized differences in the test results.
- ------------------------------------------------------------------------
+-------------------------------------------------------------------------------
-Polygon differences
+Floating-point differences
-Several of the tests involve operations on geographic data about the
-Oakland/Berkeley, California street map. The map data is expressed as
-polygons whose vertices are represented as pairs of double precision numbers
-(decimal latitude and longitude). Initially, some tables are created and
-loaded with geographic data, then some views are created that join two
-tables using the polygon intersection operator (##), then a select is done
-on the view.
+Some of the tests involve computing 64-bit floating-point numbers (double
+precision) from table columns. Differences in results involving mathematical
+functions of double precision columns have been observed. The float8 and
+geometry tests are particularly prone to small differences across platforms, or
+even with different compiler optimization options. Human eyeball comparison is
+needed to determine the real significance of these differences which are
+usually 10 places to the right of the decimal point.
-When comparing the results from different platforms, differences occur in
-the 2nd or 3rd place to the right of the decimal point. The SQL statements
-where these problems occur are the following:
+Some systems display minus zero as -0, while others just show 0.
-SELECT * from street;
-SELECT * from iexit;
+Some systems signal errors from pow() and exp() differently from the mechanism
+expected by the current PostgreSQL code.
- ------------------------------------------------------------------------
+-------------------------------------------------------------------------------
Row ordering differences
You might see differences in which the same rows are output in a different
order than what appears in the expected file. In most cases this is not,
strictly speaking, a bug. Most of the regression test scripts are not so
-pedantic as to use an ORDER BY for every single SELECT, and so their result
-row orderings are not well-defined according to the letter of the SQL
+pedantic as to use an ORDER BY for every single SELECT, and so their result row
+orderings are not well-defined according to the letter of the SQL
specification. In practice, since we are looking at the same queries being
-executed on the same data by the same software, we usually get the same
-result ordering on all platforms, and so the lack of ORDER BY isn't a
-problem. Some queries do exhibit cross-platform ordering differences,
-however. (Ordering differences can also be triggered by non-C locale
-settings.)
+executed on the same data by the same software, we usually get the same result
+ordering on all platforms, and so the lack of ORDER BY isn't a problem. Some
+queries do exhibit cross-platform ordering differences, however. (Ordering
+differences can also be triggered by non-C locale settings.)
Therefore, if you see an ordering difference, it's not something to worry
about, unless the query does have an ORDER BY that your result is violating.
-But please report it anyway, so that we can add an ORDER BY to that
-particular query and thereby eliminate the bogus "failure" in future
-releases.
+But please report it anyway, so that we can add an ORDER BY to that particular
+query and thereby eliminate the bogus "failure" in future releases.
-You might wonder why we don't order all the regress test queries explicitly
-to get rid of this issue once and for all. The reason is that that would
-make the regression tests less useful, not more, since they'd tend to
-exercise query plan types that produce ordered results to the exclusion of
-those that don't.
+You might wonder why we don't order all the regression test queries explicitly
+to get rid of this issue once and for all. The reason is that that would make
+the regression tests less useful, not more, since they'd tend to exercise query
+plan types that produce ordered results to the exclusion of those that don't.
- ------------------------------------------------------------------------
+-------------------------------------------------------------------------------
The "random" test
-There is at least one case in the "random" test script that is intended to
-produce random results. This causes random to fail the regression test once
-in a while (perhaps once in every five to ten trials). Typing
+There is at least one case in the random test script that is intended to
+produce random results. This causes random to fail the regression test once in
+a while (perhaps once in every five to ten trials). Typing
-diff results/random.out expected/random.out
+ diff results/random.out expected/random.out
should produce only one or a few lines of differences. You need not worry
-unless the random test always fails in repeated attempts. (On the other
-hand, if the random test is never reported to fail even in many trials of
-the regression tests, you probably should worry.)
+unless the random test always fails in repeated attempts. (On the other hand,
+if the random test is *never* reported to fail even in many trials of the
+regression tests, you probably *should* worry.)
+
+-------------------------------------------------------------------------------
+
+Platform-specific comparison files
+
+Since some of the tests inherently produce platform-specific results, we have
+provided a way to supply platform-specific result comparison files. Frequently,
+the same variation applies to multiple platforms; rather than supplying a
+separate comparison file for every platform, there is a mapping file that
+defines which comparison file to use. So, to eliminate bogus test "failures"
+for a particular platform, you must choose or make a variant result file, and
+then add a line to the mapping file, which is "src/test/regress/resultmap".
+
+Each line in the mapping file is of the form
+
+ testname/platformpattern=comparisonfilename
+
+The test name is just the name of the particular regression test module. The
+platform pattern is a pattern in the style of the Unix tool "expr" (that is, a
+regular expression with an implicit ^ anchor at the start). It is matched
+against the platform name as printed by "config.guess" followed by :gcc or :cc,
+depending on whether you use the GNU compiler or the system's native compiler
+(on systems where there is a difference). The comparison file name is the name
+of the substitute result comparison file.
+
+For example: some systems using older time zone libraries fail to apply
+daylight-saving corrections to dates before 1970, causing pre-1970 PDT times to
+be displayed in PST instead. This causes a few differences in the "horology"
+regression test. Therefore, we provide a variant comparison file, "horology-no-
+DST-before-1970.out", which includes the results to be expected on these
+systems. To silence the bogus "failure" message on HPUX platforms, "resultmap"
+includes
+
+ horology/.*-hpux=horology-no-DST-before-1970
+
+which will trigger on any machine for which the output of "config.guess"
+includes -hpux. Other lines in "resultmap" select the variant comparison file
+for other platforms where it's appropriate.