diff options
| author | Jeff Davis <jdavis@postgresql.org> | 2020-03-18 15:42:02 -0700 |
|---|---|---|
| committer | Jeff Davis <jdavis@postgresql.org> | 2020-03-18 15:42:02 -0700 |
| commit | 1f39bce021540fde00990af55b4432c55ef4b3c7 (patch) | |
| tree | c2403fb61234d93408b23350a82ad429b3625af3 /src/backend/optimizer/plan/planner.c | |
| parent | e00912e11a9ec2d29274ed8a6465e81385906dc2 (diff) | |
| download | postgresql-1f39bce021540fde00990af55b4432c55ef4b3c7.tar.gz postgresql-1f39bce021540fde00990af55b4432c55ef4b3c7.zip | |
Disk-based Hash Aggregation.
While performing hash aggregation, track memory usage when adding new
groups to a hash table. If the memory usage exceeds work_mem, enter
"spill mode".
In spill mode, new groups are not created in the hash table(s), but
existing groups continue to be advanced if input tuples match. Tuples
that would cause a new group to be created are instead spilled to a
logical tape to be processed later.
The tuples are spilled in a partitioned fashion. When all tuples from
the outer plan are processed (either by advancing the group or
spilling the tuple), finalize and emit the groups from the hash
table. Then, create new batches of work from the spilled partitions,
and select one of the saved batches and process it (possibly spilling
recursively).
Author: Jeff Davis
Reviewed-by: Tomas Vondra, Adam Lee, Justin Pryzby, Taylor Vesely, Melanie Plageman
Discussion: https://postgr.es/m/507ac540ec7c20136364b5272acbcd4574aa76ef.camel@j-davis.com
Diffstat (limited to 'src/backend/optimizer/plan/planner.c')
| -rw-r--r-- | src/backend/optimizer/plan/planner.c | 19 |
1 files changed, 11 insertions, 8 deletions
diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index b44efd6314c..eb25c2f4707 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -4258,11 +4258,12 @@ consider_groupingsets_paths(PlannerInfo *root, dNumGroups - exclude_groups); /* - * gd->rollups is empty if we have only unsortable columns to work - * with. Override work_mem in that case; otherwise, we'll rely on the - * sorted-input case to generate usable mixed paths. + * If we have sortable columns to work with (gd->rollups is non-empty) + * and enable_groupingsets_hash_disk is disabled, don't generate + * hash-based paths that will exceed work_mem. */ - if (hashsize > work_mem * 1024L && gd->rollups) + if (!enable_groupingsets_hash_disk && + hashsize > work_mem * 1024L && gd->rollups) return; /* nope, won't fit */ /* @@ -6528,7 +6529,8 @@ add_paths_to_grouping_rel(PlannerInfo *root, RelOptInfo *input_rel, * were unable to sort above, then we'd better generate a Path, so * that we at least have one. */ - if (hashaggtablesize < work_mem * 1024L || + if (enable_hashagg_disk || + hashaggtablesize < work_mem * 1024L || grouped_rel->pathlist == NIL) { /* @@ -6561,7 +6563,8 @@ add_paths_to_grouping_rel(PlannerInfo *root, RelOptInfo *input_rel, agg_final_costs, dNumGroups); - if (hashaggtablesize < work_mem * 1024L) + if (enable_hashagg_disk || + hashaggtablesize < work_mem * 1024L) add_path(grouped_rel, (Path *) create_agg_path(root, grouped_rel, @@ -6830,7 +6833,7 @@ create_partial_grouping_paths(PlannerInfo *root, * Tentatively produce a partial HashAgg Path, depending on if it * looks as if the hash table will fit in work_mem. */ - if (hashaggtablesize < work_mem * 1024L && + if ((enable_hashagg_disk || hashaggtablesize < work_mem * 1024L) && cheapest_total_path != NULL) { add_path(partially_grouped_rel, (Path *) @@ -6857,7 +6860,7 @@ create_partial_grouping_paths(PlannerInfo *root, dNumPartialPartialGroups); /* Do the same for partial paths. */ - if (hashaggtablesize < work_mem * 1024L && + if ((enable_hashagg_disk || hashaggtablesize < work_mem * 1024L) && cheapest_partial_path != NULL) { add_partial_path(partially_grouped_rel, (Path *) |