My objective is to make Postgres choose a faster plan after a large table update. Before the table update, the plan PG choose is what I expect (the faster plan). However, after the update, PG chooses a slower plan (it ignores an index).
Setup
I have a single-column table of type jsonb.
CREATE SCHEMA icn;
CREATE TABLE icn.jsonbs (dat jsonb);
So, I loaded the table with random data.
DO
$do$
BEGIN
FOR i IN 1..10000 LOOP
EXECUTE format('INSERT INTO icn.jsonbs VALUES (''{"name": "%s", "role": "%s", "addr": {"city": "%s"}}'')',
(SELECT ('[0:2]={dog,cat,bird}'::text[])[floor(random()*3)]),
(SELECT ('[0:1]={true,false}'::text[])[floor(random()*2)]),
(SELECT ('[0:11]={toronto,vancouver,montreal,dhaka,sylhet,alberta,a,b,c,d,e,f}'::text[])[floor(random()*12)])
);
END LOOP;
END
$do$;
(pg-14.9) agedev=# SELECT * FROM icn.jsonbs LIMIT 5;
dat
----------------------------------------------------------------
{"addr": {"city": "vancouver"}, "name": "cat", "role": "true"}
{"addr": {"city": "sylhet"}, "name": "dog", "role": "true"}
{"addr": {"city": "dhaka"}, "name": "cat", "role": "false"}
{"addr": {"city": "toronto"}, "name": "bird", "role": "false"}
{"addr": {"city": "b"}, "name": "bird", "role": "false"}
(5 rows)
Then, I created one GIN and one Btree index.
CREATE INDEX btree_name ON icn.jsonbs USING btree ((UPPER((dat -> 'name')::text)));
CREATE INDEX gin_dat ON icn.jsonbs USING gin(dat);
Next, I run this query.:
EXPLAIN ANALYZE
SELECT * FROM icn.jsonbs
WHERE
UPPER((dat -> 'name')::text) = '"DOG"' AND dat @> '{"addr": {"city": "toronto"}}'
;
I expect PG to use this plan. So, for this time it chose the correct plan.
QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------
Bitmap Heap Scan on jsonbs (cost=29.66..33.69 rows=1 width=32) (actual time=2.929..4.634 rows=281 loops=1)
Recheck Cond: ((upper(((dat -> 'name'::text))::text) = '"DOG"'::text) AND (dat @> '{"addr": {"city": "toronto"}}'::jsonb))
Heap Blocks: exact=115
-> BitmapAnd (cost=29.66..29.66 rows=1 width=0) (actual time=2.833..2.834 rows=0 loops=1)
-> Bitmap Index Scan on btree_name (cost=0.00..4.66 rows=50 width=0) (actual time=0.876..0.876 rows=3297 loops=1)
Index Cond: (upper(((dat -> 'name'::text))::text) = '"DOG"'::text)
-> Bitmap Index Scan on gin_dat (cost=0.00..24.75 rows=100 width=0) (actual time=1.929..1.929 rows=861 loops=1)
Index Cond: (dat @> '{"addr": {"city": "toronto"}}'::jsonb)
Planning Time: 0.470 ms
Execution Time: 4.720 ms
(10 rows)
Then, I update some random row:
DO
$do$
BEGIN
FOR i IN 1..5000 LOOP
UPDATE icn.jsonbs SET dat = dat || '{"role": "mouse"}' WHERE ctid = (SELECT ctid FROM icn.jsonbs OFFSET floor(random()*10000) LIMIT 1);
END LOOP;
END
$do$;
Then, I run the previous query again:
QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------
Bitmap Heap Scan on jsonbs (cost=4.74..119.63 rows=1 width=32) (actual time=1.267..11.013 rows=281 loops=1)
Recheck Cond: (upper(((dat -> 'name'::text))::text) = '"DOG"'::text)
Filter: (dat @> '{"addr": {"city": "toronto"}}'::jsonb)
Rows Removed by Filter: 3016
Heap Blocks: exact=150
-> Bitmap Index Scan on btree_name (cost=0.00..4.74 rows=60 width=0) (actual time=1.095..1.095 rows=3968 loops=1)
Index Cond: (upper(((dat -> 'name'::text))::text) = '"DOG"'::text)
Planning Time: 0.209 ms
Execution Time: 11.087 ms
(9 rows)
Problem
After the table update, Postgres stops using the BitmapAnd
(the combination of the GIN and Btree index). It uses the btree index only. As a result, the query runs slower.
I expect it to keep using the plan with both indices.
Expectation
Dropping and recreating the indices solves the problem. Also, this problem does not appear if number of random row update is lower (for example, less than 1000).
I also tried VACUUM ANALYZE
. It doesn’t solve the issue.
I expect it to choose the faster plan, without having to recreating the indices.
2
Answers
It can depend upon how the database is configured.
The typical reason is that if random_page_cost has been left as the default, which is best for mechanical hard drives but not as appropriate for SSDs.
So accessing 2 indexes would be perceived as more expensive than one, especially if the second index was a large GIN index.
So if that’s applicable, then maybe tuning this setting would give a different result, although there are other settings which can have an influence, e.g.
effective_cache_size
, which determines if it’s worth using more RAM for the sake of better performance.In this case, the change in plans is due to the GIN index pending list which gets bloated during the update. The planner knows that it is large (once the update is done), and penalizes the index’s usage for it.
If you are not interested in a pending list, you could turn it off by specifying
with (fastupdate=off)
when creating the index. Or you could manually clean it usingselect gin_clean_pending_list('icn.gin_dat')
after the update is done; but VACUUM also does this cleaning, so you could instead just let vacuum do its job.Note that cleaning the pending list doesn’t just make the BitmapAnd query appear faster, it also makes it be faster. So we don’t know that it is choosing the wrong plan, perhaps it is choosing the right plan given the true state of the pending list at that time. It is hard to force it to use the BitmapAnd when it doesn’t want to, there are no
enable_*
parameters that addresses that situation. Maybe you could use pg_hint_plan to do it, I don’t know. Or you could compile an experimental server with some hacks, but since I don’t have your weird hardware this wouldn’t be very useful for me to do.And it is all probably a trivial pursuit anyway. Would fixing an artificial problem caused in an artificial benchmark carry over to the real world in any meaningful way? It is not impossible, but I rather doubt it.