diff options
Diffstat (limited to 'src')
| -rw-r--r-- | src/test/regress/README | 243 |
1 files changed, 243 insertions, 0 deletions
diff --git a/src/test/regress/README b/src/test/regress/README new file mode 100644 index 00000000000..86311be8983 --- /dev/null +++ b/src/test/regress/README @@ -0,0 +1,243 @@ +Regression Tests + +Introduction + +The regression tests are a comprehensive set of tests for the SQL +implementation in PostgreSQL. They test standard SQL operations as well as +the extended capabilities of PostgreSQL. The test suite was originally +developed by Jolly Chen and Andrew Yu, and was extensively revised and +repackaged by Marc Fournier and Thomas Lockhart. From PostgreSQL 6.1 onward +the regression tests are current for every official release. + + ------------------------------------------------------------------------ + +Running the Tests + +The regression test can be run against an already installed and running +server, or using a temporary installation within the build tree. +Furthermore, there is a "parallel" and a "sequential" mode for running the +tests. The sequential method runs each test script in turn, whereas the +parallel method starts up multiple server processes to run groups of tests +in parallel. Parallel testing gives confidence that interprocess +communication and locking are working correctly. For historical reasons, the +sequential test is usually run against an existing installation and the +parallel method against a temporary installation, but there are no technical +reasons for this. + +To run the regression tests after building but before installation, type + +$ gmake check + +in the top-level directory. (Or you can change to src/test/regress and run +the command there.) This will first build several auxiliary files, such as +platform-dependent "expected" files and some sample user-defined trigger +functions, and then run the test driver script. At the end you should see +something like + +====================== + All 77 tests passed. +====================== + +or otherwise a note about what tests failed. See the Section called Test +Evaluation below for more. + + Note: Because this test method runs a temporary server, it will + not work when you are the root user (the server will not start as + root). If you already did the build as root, you do not have to + start all over. Instead, make the regression test directory + writable by some other user, log in as that user, and restart the + tests. For example, + + root# chmod -R a+w src/test/regress + root# chmod -R a+w contrib/spi + root# su - joeuser + joeuser$ cd <build top-level directory> + joeuser$ gmake check + + (The only possible "security risk" here is that other users might + be able to alter the regression test results behind your back. Use + common sense when managing user permissions.) + + Alternatively, run the tests after installation. + + Tip: On some systems, the default Bourne-compatible shell + (/bin/sh) gets confused when it has to manage too many child + processes in parallel. This may cause the parallel test run to + lock up or fail. In such cases, specify a different + Bourne-compatible shell on the command line, for example: + + $ gmake SHELL=/bin/ksh check + +To run the tests after installation, initialize a data area and start the +server, then type + +$ gmake installcheck + +The tests will expect to contact the server at the local host and the +default port number, unless directed otherwise by PGHOST and PGPORT +environment variables. + + ------------------------------------------------------------------------ + +Test Evaluation + +Some properly installed and fully functional PostgreSQL installations can +"fail" some of these regression tests due to platform-specific artifacts +such as varying floating point representation and time zone support. The +tests are currently evaluated using a simple diff comparison against the +outputs generated on a reference system, so the results are sensitive to +small system differences. When a test is reported as "failed", always +examine the differences between expected and actual results; you may well +find that the differences are not significant. Nonetheless, we still strive +to maintain accurate reference files across all supported platforms, so it +can be expected that all tests pass. + +The actual outputs of the regression tests are in files in the +src/test/regress/results directory. The test script uses diff to compare +each output file against the reference outputs stored in the +src/test/regress/expected directory. Any differences are saved for your +inspection in src/test/regress/regression.diffs. (Or you can run diff +yourself, if you prefer.) + + ------------------------------------------------------------------------ + +Error message differences + +Some of the regression tests involve intentional invalid input values. Error +messages can come from either the PostgreSQL code or from the host platform +system routines. In the latter case, the messages may vary between +platforms, but should reflect similar information. These differences in +messages will result in a "failed" regression test that can be validated by +inspection. + + ------------------------------------------------------------------------ + +Locale differences + +The tests expect to run in plain "C" locale. This should not cause any +problems when you run the tests against a temporary installation, since the +regression test driver takes care to start the server in C locale. However, +if you run the tests against an already-installed server that is using non-C +locale settings, you may see differences caused by varying rules for string +sort order, formatting of numeric and monetary values, and so forth. + +In some locales the resulting differences are small and easily checked by +inspection. However, in a locale that changes the rules for formatting of +numeric values (typically by swapping the usage of commas and decimal +points), entry of some data values will fail, resulting in extensive +differences later in the tests where the missing data values are supposed to +be used. + + ------------------------------------------------------------------------ + +Date and time differences + +Some of the queries in the timestamp test will fail if you run the test on +the day of a daylight-savings time changeover, or the day before or after +one. These queries assume that the intervals between midnight yesterday, +midnight today and midnight tomorrow are exactly twenty-four hours -- which +is wrong if daylight-savings time went into or out of effect meanwhile. + +Most of the date and time results are dependent on the time zone +environment. The reference files are generated for time zone PST8PDT +(Berkeley, California) and there will be apparent failures if the tests are +not run with that time zone setting. The regression test driver sets +environment variable PGTZ to PST8PDT, which normally ensures proper results. +However, your system must provide library support for the PST8PDT time zone, +or the time zone-dependent tests will fail. To verify that your machine does +have this support, type the following: + +$ env TZ=PST8PDT date + +The command above should have returned the current system time in the +PST8PDT time zone. If the PST8PDT database is not available, then your +system may have returned the time in GMT. If the PST8PDT time zone is not +available, you can set the time zone rules explicitly: + +PGTZ='PST8PDT7,M04.01.0,M10.05.03'; export PGTZ + +There appear to be some systems that do not accept the recommended syntax +for explicitly setting the local time zone rules; you may need to use a +different PGTZ setting on such machines. + +Some systems using older time zone libraries fail to apply daylight-savings +corrections to dates before 1970, causing pre-1970 PDT times to be displayed +in PST instead. This will result in localized differences in the test +results. + + ------------------------------------------------------------------------ + +Floating point differences + +Some of the tests involve computing 64-bit (double precision) numbers from +table columns. Differences in results involving mathematical functions of +double precision columns have been observed. The float8 and geometry tests +are particularly prone to small differences across platforms, or even with +different compiler optimization options. Human eyeball comparison is needed +to determine the real significance of these differences which are usually 10 +places to the right of the decimal point. + +Some systems signal errors from pow() and exp() differently from the +mechanism expected by the current PostgreSQL code. + + ------------------------------------------------------------------------ + +Polygon differences + +Several of the tests involve operations on geographic data about the +Oakland/Berkeley, California street map. The map data is expressed as +polygons whose vertices are represented as pairs of double precision numbers +(decimal latitude and longitude). Initially, some tables are created and +loaded with geographic data, then some views are created that join two +tables using the polygon intersection operator (##), then a select is done +on the view. + +When comparing the results from different platforms, differences occur in +the 2nd or 3rd place to the right of the decimal point. The SQL statements +where these problems occur are the following: + +SELECT * from street; +SELECT * from iexit; + + ------------------------------------------------------------------------ + +Row ordering differences + +You might see differences in which the same rows are output in a different +order than what appears in the expected file. In most cases this is not, +strictly speaking, a bug. Most of the regression test scripts are not so +pedantic as to use an ORDER BY for every single SELECT, and so their result +row orderings are not well-defined according to the letter of the SQL +specification. In practice, since we are looking at the same queries being +executed on the same data by the same software, we usually get the same +result ordering on all platforms, and so the lack of ORDER BY isn't a +problem. Some queries do exhibit cross-platform ordering differences, +however. (Ordering differences can also be triggered by non-C locale +settings.) + +Therefore, if you see an ordering difference, it's not something to worry +about, unless the query does have an ORDER BY that your result is violating. +But please report it anyway, so that we can add an ORDER BY to that +particular query and thereby eliminate the bogus "failure" in future +releases. + +You might wonder why we don't order all the regress test queries explicitly +to get rid of this issue once and for all. The reason is that that would +make the regression tests less useful, not more, since they'd tend to +exercise query plan types that produce ordered results to the exclusion of +those that don't. + + ------------------------------------------------------------------------ + +The "random" test + +There is at least one case in the "random" test script that is intended to +produce random results. This causes random to fail the regression test once +in a while (perhaps once in every five to ten trials). Typing + +diff results/random.out expected/random.out + +should produce only one or a few lines of differences. You need not worry +unless the random test always fails in repeated attempts. (On the other +hand, if the random test is never reported to fail even in many trials of +the regression tests, you probably should worry.) |