Set join_collapse_limit default value to 13

[andr-sokolov]

The default value of join_collapse_limit was 20. When this value is set and
the query contains about 20 joins (see added test), Postgres query optimizer
cannot build a plan during hours and consumes a lot of memory, because the
planner checks a lot of possible ways to join the tables.
When join_collapse_limit is 13, the query plan is built in reasonable time.

[andr-sokolov]

I suggest join_collapse_limit=8 to reduce the difference from PostgreSQL 14, but you can consider slightly larger values. In the table below there are time and memory consumption to process the query from the test.

join_collapse_limit	Query processing time, ms	Backend memory usage
8	96	52 MB
9	230	88 MB
10	653	210 MB
11	1269	394 MB
12	2994	889 MB
13	9003	2.4 GB
14	27245	7.3 GB
15	80904	22 GB
16	670578	66 GB
17	4158314	200 GB

Query processing time was got using \timing on in psql. Backend memory usage was measured at the end of the planner function

[leborchuk]

I suggest join_collapse_limit=8 to reduce the difference from PostgreSQL 14, but you can consider slightly larger values. In the table below there are time and memory consumption to process the query from the test.

join_collapse_limit Query processing time, ms Backend memory usage
8 96 52 MB
9 230 88 MB
10 653 210 MB
11 1269 394 MB
12 2994 889 MB
13 9003 2.4 GB
14 27245 7.3 GB
15 80904 22 GB
16 670578 66 GB
17 4158314 200 GB
Query processing time was got using \timing on in psql. Backend memory usage was measured at the end of the planner function

Thank you such interesting research!

I would suggest setting join_collapse_limit to 13 or 14 for your choice.

I agree, 20 is too much, but 8 is also too small. The reason is to exchange execution time for planning time and free memory (on the master). However, the master has its own limitations. Join_collapse_limit should be set to a value that prevents exhaustion of resources by a single user. Typically, even test/development installations have 8 GB of free memory space. I'm not sure about 22 GB; it seems quite large. I have seen many dev/test instances with 10 GB to 16 GB of free memory space.

[andr-sokolov]

I would suggest setting join_collapse_limit to 13 or 14 for your choice.

I agree, 20 is too much, but 8 is also too small. The reason is to exchange execution time for planning time and free memory (on the master). However, the master has its own limitations. Join_collapse_limit should be set to a value that prevents exhaustion of resources by a single user. Typically, even test/development installations have 8 GB of free memory space. I'm not sure about 22 GB; it seems quite large. I have seen many dev/test instances with 10 GB to 16 GB of free memory space.

I have replaced 8 with 13. 14 is too high, because in this case I get ERROR: Canceling query because of high VMEM usage. Used: 7376MB, available 816MB, red zone: 7372MB (runaway_cleaner.c:189) on demo cluster with default gp_vmem_protect_limit value

[Smyatkin-Maxim]

+1 for anything between 12 and 13