skip to Main Content

Postgres version: 13.2

We have a complex query to get a list of decks in production and we experience slow response time.

Here is the content of the function. I have hardcoded the 4 arguments of the function to get a detailed query plan

SELECT community_deck.approved_at,
        community_deck.approval_status,
        community_deck.is_public,
        community_deck.owner_sub,
        decks.id,
        decks.title,
        decks.owner,
        decks.share_id,
        decks.objective,
        decks.description,
        decks.updated_at,
        decks.frontend_id,
        af.field,
        users.avatar,
        users.quote,
        users.nickname,
        users.picture,
        users.profile_picture,
        ud.background_color,
        ud.text_color,
        count(DISTINCT dc.card_id) as number_of_cards,
        count(DISTINCT dr.user_id) as number_of_likes,
        count(DISTINCT dd.user_id) as number_of_unique_downloads
 from community_deck
          left join decks on decks.id = community_deck.deck_id
          left join decks_cards dc on decks.id = dc.deck_id
          left join deck_reaction dr on decks.id = dr.deck_id
          left join academic_fields af on af.id = decks.category
          left join users on users.sub = decks.owner
          left join user_deck ud on decks.id = ud.deck_id AND ud.user_id = decks.owner
          left join deck_download dd on decks.id = dd.deck_id
 WHERE community_deck.is_public = true
   AND community_deck.approval_status = 'approved'
   AND (null is null OR af.field = null)
 group by decks.id, af.field, users.avatar, users.quote, users.nickname, users.picture,
          users.profile_picture,
          ud.background_color, ud.text_color, decks.created_at, community_deck.approved_at,
          community_deck.approval_status, community_deck.is_public, community_deck.owner_sub
 ORDER BY approved_at DESC;

There might be multiple factors as why the reponse time is slow but I wanted to know if a query like this could become problematic and if there were obvious improvements that could be added.

I ran a query plan using explain (analyze, buffers) SELECT and here is the result I got:

GroupAggregate  (cost=314.33..396.91 rows=1573 width=410) (actual time=517.521..945.153 rows=49 loops=1)
"  Group Key: community_deck.approved_at, decks.id, af.field, users.avatar, users.quote, users.nickname, users.picture, users.profile_picture, ud.background_color, ud.text_color, community_deck.approval_status, community_deck.is_public, community_deck.owner_sub"
"  Buffers: shared hit=14643, temp read=5777 written=5793"
  ->  Sort  (cost=314.33..318.26 rows=1573 width=474) (actual time=517.373..770.206 rows=81653 loops=1)
"        Sort Key: community_deck.approved_at DESC, decks.id, af.field, users.avatar, users.quote, users.nickname, users.picture, users.profile_picture, ud.background_color, ud.text_color, community_deck.is_public, community_deck.owner_sub"
        Sort Method: external merge  Disk: 39152kB
"        Buffers: shared hit=14643, temp read=5777 written=5793"
        ->  Nested Loop Left Join  (cost=102.01..230.81 rows=1573 width=474) (actual time=3.146..78.871 rows=81653 loops=1)
              Buffers: shared hit=14629
              ->  Nested Loop Left Join  (cost=101.72..140.56 rows=42 width=470) (actual time=3.111..19.517 rows=801 loops=1)
                    Buffers: shared hit=4874
                    ->  Nested Loop Left Join  (cost=101.44..124.22 rows=42 width=457) (actual time=3.058..15.053 rows=801 loops=1)
                          Buffers: shared hit=2471
                          ->  Hash Left Join  (cost=101.16..106.63 rows=42 width=303) (actual time=2.998..5.190 rows=801 loops=1)
                                Hash Cond: (decks.category = af.id)
                                Buffers: shared hit=68
                                ->  Hash Right Join  (cost=99.56..104.91 rows=42 width=307) (actual time=2.873..4.179 rows=801 loops=1)
                                      Hash Cond: (dd.deck_id = decks.id)
                                      Buffers: shared hit=64
                                      ->  Seq Scan on deck_download dd  (cost=0.00..4.69 rows=169 width=46) (actual time=0.011..0.163 rows=194 loops=1)
                                            Buffers: shared hit=3
                                      ->  Hash  (cost=99.03..99.03 rows=42 width=265) (actual time=2.844..2.851 rows=139 loops=1)
                                            Buckets: 1024  Batches: 1  Memory Usage: 50kB
                                            Buffers: shared hit=61
                                            ->  Hash Right Join  (cost=95.29..99.03 rows=42 width=265) (actual time=2.502..2.712 rows=139 loops=1)
                                                  Hash Cond: (dr.deck_id = decks.id)
                                                  Buffers: shared hit=61
                                                  ->  Seq Scan on deck_reaction dr  (cost=0.00..3.25 rows=125 width=46) (actual time=0.011..0.108 rows=138 loops=1)
                                                        Buffers: shared hit=2
                                                  ->  Hash  (cost=94.77..94.77 rows=42 width=223) (actual time=2.473..2.477 rows=49 loops=1)
                                                        Buckets: 1024  Batches: 1  Memory Usage: 21kB
                                                        Buffers: shared hit=59
                                                        ->  Hash Right Join  (cost=2.12..94.77 rows=42 width=223) (actual time=0.271..2.404 rows=49 loops=1)
                                                              Hash Cond: (decks.id = community_deck.deck_id)
                                                              Buffers: shared hit=59
                                                              ->  Seq Scan on decks  (cost=0.00..85.43 rows=2743 width=162) (actual time=0.010..1.763 rows=2780 loops=1)
                                                                    Buffers: shared hit=58
                                                              ->  Hash  (cost=1.60..1.60 rows=42 width=65) (actual time=0.070..0.072 rows=49 loops=1)
                                                                    Buckets: 1024  Batches: 1  Memory Usage: 13kB
                                                                    Buffers: shared hit=1
                                                                    ->  Seq Scan on community_deck  (cost=0.00..1.60 rows=42 width=65) (actual time=0.019..0.041 rows=49 loops=1)
                                                                          Filter: (is_public AND (approval_status = 'approved'::text))
                                                                          Rows Removed by Filter: 4
                                                                          Buffers: shared hit=1
                                ->  Hash  (cost=1.27..1.27 rows=27 width=28) (actual time=0.042..0.043 rows=27 loops=1)
                                      Buckets: 1024  Batches: 1  Memory Usage: 10kB
                                      Buffers: shared hit=1
                                      ->  Seq Scan on academic_fields af  (cost=0.00..1.27 rows=27 width=28) (actual time=0.009..0.015 rows=27 loops=1)
                                            Buffers: shared hit=1
                          ->  Index Scan using users_sub_uindex on users  (cost=0.28..0.42 rows=1 width=196) (actual time=0.011..0.011 rows=1 loops=801)
                                Index Cond: (sub = decks.owner)
                                Buffers: shared hit=2403
                    ->  Index Scan using user_deck_pkey on user_deck ud  (cost=0.28..0.39 rows=1 width=58) (actual time=0.004..0.004 rows=1 loops=801)
                          Index Cond: ((deck_id = decks.id) AND (user_id = decks.owner))
                          Buffers: shared hit=2403
              ->  Index Only Scan using decks_cards_pkey on decks_cards dc  (cost=0.29..1.72 rows=43 width=8) (actual time=0.006..0.035 rows=102 loops=801)
                    Index Cond: (deck_id = decks.id)
                    Heap Fetches: 8672
                    Buffers: shared hit=9755
Planning:
  Buffers: shared hit=589
Planning Time: 12.864 ms
Execution Time: 955.913 ms

2

Answers


  1. Chosen as BEST ANSWER

    After some trials, it seems that removing the joins in favour of sub-queries greatly impacts performances. Especially removing the decks_cards join (~100 000 rows)

    here is the new query:

    SELECT community_deck.approved_at,
           community_deck.approval_status,
           community_deck.is_public,
           community_deck.owner_sub,
           decks.id,
           decks.title,
           decks.owner,
           decks.share_id,
           decks.objective,
           decks.description,
           decks.updated_at,
           decks.frontend_id,
           af.field,
           users.avatar,
           users.quote,
           users.nickname,
           users.picture,
           users.profile_picture,
           ud.background_color,
           ud.text_color,
           (SELECT (count(DISTINCT card_id)) from decks_cards where deck_id = decks.id)   as number_of_cards,
           (SELECT (count(DISTINCT user_id)) from deck_reaction where deck_id = decks.id) as number_of_likes,
           (SELECT (count(DISTINCT user_id)) from deck_download where deck_id = decks.id) as number_of_unique_downloads
    from community_deck
             left join decks on decks.id = community_deck.deck_id
             left join academic_fields af on af.id = decks.category
             left join users on users.sub = decks.owner
             left join user_deck ud on decks.id = ud.deck_id AND ud.user_id = decks.owner
    WHERE community_deck.is_public = true
      AND community_deck.approval_status = 'approved'
      AND (null is null
        OR af.field = null)
    group by decks.id, af.field, users.avatar, users.quote, users.nickname, users.picture,
             users.profile_picture,
             ud.background_color, ud.text_color, decks.created_at, community_deck.approved_at,
             community_deck.approval_status, community_deck.is_public, community_deck.owner_sub
    ORDER BY approved_at DESC;
    

    And here is the new query plan:

    Group  (cost=131.55..1596.98 rows=42 width=410) (actual time=3.969..11.180 rows=49 loops=1)
    "  Group Key: community_deck.approved_at, decks.id, af.field, users.avatar, users.quote, users.nickname, users.picture, users.profile_picture, ud.background_color, ud.text_color, community_deck.approval_status, community_deck.is_public, community_deck.owner_sub"
      Buffers: shared hit=1449
      ->  Sort  (cost=131.55..131.65 rows=42 width=386) (actual time=3.770..3.794 rows=49 loops=1)
    "        Sort Key: community_deck.approved_at DESC, decks.id, af.field, users.avatar, users.quote, users.nickname, users.picture, users.profile_picture, ud.background_color, ud.text_color, community_deck.is_public, community_deck.owner_sub"
            Sort Method: quicksort  Memory: 50kB
            Buffers: shared hit=354
            ->  Nested Loop Left Join  (cost=4.29..130.42 rows=42 width=386) (actual time=0.426..3.671 rows=49 loops=1)
                  Buffers: shared hit=354
                  ->  Nested Loop Left Join  (cost=4.01..114.08 rows=42 width=373) (actual time=0.398..3.246 rows=49 loops=1)
                        Buffers: shared hit=207
                        ->  Hash Left Join  (cost=3.73..96.48 rows=42 width=219) (actual time=0.319..2.523 rows=49 loops=1)
                              Hash Cond: (decks.category = af.id)
                              Buffers: shared hit=60
                              ->  Hash Right Join  (cost=2.12..94.77 rows=42 width=223) (actual time=0.269..2.415 rows=49 loops=1)
                                    Hash Cond: (decks.id = community_deck.deck_id)
                                    Buffers: shared hit=59
                                    ->  Seq Scan on decks  (cost=0.00..85.43 rows=2743 width=162) (actual time=0.010..1.756 rows=2783 loops=1)
                                          Buffers: shared hit=58
                                    ->  Hash  (cost=1.60..1.60 rows=42 width=65) (actual time=0.078..0.080 rows=49 loops=1)
                                          Buckets: 1024  Batches: 1  Memory Usage: 13kB
                                          Buffers: shared hit=1
                                          ->  Seq Scan on community_deck  (cost=0.00..1.60 rows=42 width=65) (actual time=0.026..0.051 rows=49 loops=1)
                                                Filter: (is_public AND (approval_status = 'approved'::text))
                                                Rows Removed by Filter: 4
                                                Buffers: shared hit=1
                              ->  Hash  (cost=1.27..1.27 rows=27 width=28) (actual time=0.040..0.041 rows=27 loops=1)
                                    Buckets: 1024  Batches: 1  Memory Usage: 10kB
                                    Buffers: shared hit=1
                                    ->  Seq Scan on academic_fields af  (cost=0.00..1.27 rows=27 width=28) (actual time=0.010..0.018 rows=27 loops=1)
                                          Buffers: shared hit=1
                        ->  Index Scan using users_sub_uindex on users  (cost=0.28..0.42 rows=1 width=196) (actual time=0.013..0.013 rows=1 loops=49)
                              Index Cond: (sub = decks.owner)
                              Buffers: shared hit=147
                  ->  Index Scan using user_deck_pkey on user_deck ud  (cost=0.28..0.39 rows=1 width=58) (actual time=0.006..0.006 rows=1 loops=49)
                        Index Cond: ((deck_id = decks.id) AND (user_id = decks.owner))
                        Buffers: shared hit=147
      SubPlan 1
        ->  Aggregate  (cost=26.13..26.14 rows=1 width=8) (actual time=0.064..0.064 rows=1 loops=49)
              Buffers: shared hit=850
              ->  Index Only Scan using decks_cards_pkey on decks_cards  (cost=0.29..26.02 rows=43 width=4) (actual time=0.012..0.043 rows=82 loops=49)
                    Index Cond: (deck_id = decks.id)
                    Heap Fetches: 961
                    Buffers: shared hit=850
      SubPlan 2
        ->  Aggregate  (cost=3.57..3.58 rows=1 width=8) (actual time=0.035..0.035 rows=1 loops=49)
              Buffers: shared hit=98
              ->  Seq Scan on deck_reaction  (cost=0.00..3.56 rows=3 width=42) (actual time=0.015..0.027 rows=3 loops=49)
                    Filter: (deck_id = decks.id)
                    Rows Removed by Filter: 136
                    Buffers: shared hit=98
      SubPlan 3
        ->  Aggregate  (cost=5.13..5.14 rows=1 width=8) (actual time=0.045..0.045 rows=1 loops=49)
              Buffers: shared hit=147
              ->  Seq Scan on deck_download  (cost=0.00..5.11 rows=5 width=42) (actual time=0.024..0.037 rows=4 loops=49)
                    Filter: (deck_id = decks.id)
                    Rows Removed by Filter: 191
                    Buffers: shared hit=147
    Planning:
      Buffers: shared hit=42
    Planning Time: 2.867 ms
    Execution Time: 11.441 ms
    

  2. And by pushing doown details about decks ? like :

    WITH T AS
    (
    SELECT community_deck.approved_at,
            community_deck.approval_status,
            community_deck.is_public,
            community_deck.owner_sub,
            decks.id,
            af.field,
            users.avatar,
            users.quote,
            users.nickname,
            users.picture,
            users.profile_picture,
            ud.background_color,
            ud.text_color,
            count(DISTINCT dc.card_id) as number_of_cards,
            count(DISTINCT dr.user_id) as number_of_likes,
            count(DISTINCT dd.user_id) as number_of_unique_downloads
     from community_deck
              left join decks on decks.id = community_deck.deck_id
              left join decks_cards dc on decks.id = dc.deck_id
              left join deck_reaction dr on decks.id = dr.deck_id
              left join academic_fields af on af.id = decks.category
              left join users on users.sub = decks.owner
              left join user_deck ud on decks.id = ud.deck_id AND ud.user_id = decks.owner
              left join deck_download dd on decks.id = dd.deck_id
     WHERE community_deck.is_public = true
       AND community_deck.approval_status = 'approved'
       AND (null is null OR af.field = null)
     group by decks.id, af.field, users.avatar, users.quote, users.nickname, users.picture,
              users.profile_picture,
              ud.background_color, ud.text_color, decks.created_at, community_deck.approved_at,
              community_deck.approval_status, community_deck.is_public, community_deck.owner_sub
    )
    SELECT  T.*, 
            decks.title,
            decks.owner,
            decks.share_id,
            decks.objective,
            decks.description,
            decks.updated_at,
            decks.frontend_id
    FROM    T
            JOIN decks ON T.id = decks.id
    ORDER BY approved_at DESC;
    

    You can also join with users and user_decks at the final query, like this :

    WITH T AS
    (
    SELECT community_deck.approved_at,
            community_deck.approval_status,
            community_deck.is_public,
            community_deck.owner_sub,
            decks.id,
            af.field,
            count(DISTINCT dc.card_id) as number_of_cards,
            count(DISTINCT dr.user_id) as number_of_likes,
            count(DISTINCT dd.user_id) as number_of_unique_downloads
     from community_deck
              left join decks on decks.id = community_deck.deck_id
                 left join decks_cards dc on decks.id = dc.deck_id
                 left join deck_reaction dr on decks.id = dr.deck_id
                 left join academic_fields af on af.id = decks.category
    
                 left join deck_download dd on decks.id = dd.deck_id
     WHERE community_deck.is_public = true
       AND community_deck.approval_status = 'approved'
       AND (null is null OR af.field = null)
     group by decks.id, af.field, users.avatar, users.quote, users.nickname, users.picture,
              users.profile_picture,
              ud.background_color, ud.text_color, decks.created_at, community_deck.approved_at,
              community_deck.approval_status, community_deck.is_public, community_deck.owner_sub
    )
    SELECT  T.*, 
            decks.title,
            decks.owner,
            decks.share_id,
            decks.objective,
            decks.description,
            decks.updated_at,
            decks.frontend_id,
            users.avatar,
            users.quote,
            users.nickname,
            users.picture,
            users.profile_picture,
            ud.background_color,
            ud.text_color
    FROM    T
            JOIN decks ON T.id = decks.id
            left join users on users.sub = decks.owner
            left join user_deck ud on decks.id = ud.deck_id AND ud.user_id = decks.owner
    ORDER BY approved_at DESC;
    
    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search