From 40e2e5e92b7da358fb45802b53c735d25a51d23a Mon Sep 17 00:00:00 2001 From: Nathan Bossart Date: Mon, 16 Sep 2024 16:10:33 -0500 Subject: Introduce framework for parallelizing various pg_upgrade tasks. A number of pg_upgrade steps require connecting to every database in the cluster and running the same query in each one. When there are many databases, these steps are particularly time-consuming, especially since they are performed sequentially, i.e., we connect to a database, run the query, and process the results before moving on to the next database. This commit introduces a new framework that makes it easy to parallelize most of these once-in-each-database tasks by processing multiple databases concurrently. This framework manages a set of slots that follow a simple state machine, and it uses libpq's asynchronous APIs to establish the connections and run the queries. The --jobs option is used to determine the number of slots to use. To use this new task framework, callers simply need to provide the query and a callback function to process its results, and the framework takes care of the rest. A more complete description is provided at the top of the new task.c file. None of the eligible once-in-each-database tasks are converted to use this new framework in this commit. That will be done via several follow-up commits. Reviewed-by: Jeff Davis, Robert Haas, Daniel Gustafsson, Ilya Gladyshev, Corey Huinker Discussion: https://postgr.es/m/20240516211638.GA1688936%40nathanxps13 --- doc/src/sgml/ref/pgupgrade.sgml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) (limited to 'doc/src') diff --git a/doc/src/sgml/ref/pgupgrade.sgml b/doc/src/sgml/ref/pgupgrade.sgml index 9877f2f01c6..fc2d0ff8451 100644 --- a/doc/src/sgml/ref/pgupgrade.sgml +++ b/doc/src/sgml/ref/pgupgrade.sgml @@ -118,7 +118,7 @@ PostgreSQL documentation - number of simultaneous processes or threads to use + number of simultaneous connections and processes/threads to use @@ -587,8 +587,8 @@ NET STOP postgresql-&majorversion; The option allows multiple CPU cores to be used - for copying/linking of files and to dump and restore database schemas - in parallel; a good place to start is the maximum of the number of + for copying/linking of files, dumping and restoring database schemas + in parallel, etc.; a good place to start is the maximum of the number of CPU cores and tablespaces. This option can dramatically reduce the time to upgrade a multi-database server running on a multiprocessor machine. -- cgit v1.2.3