feat(tasks/queue-size): Implement queue_size

[gagantrivedi]

Thanks for submitting a PR! Please check the boxes below:

• I have run pre-commit to check linting
• I have filled in the "Changes" section below?
• I have filled in the "How did you test this code" section below?
• I have used a Conventional Commit title for this Pull Request

Changes

Implement queue size to limit low priority tasks from overwhelming the queue

Plan and SQL:

explain analyze SELECT COUNT(*) AS "__count" FROM "task_processor_task" WHERE (NOT "task_processor_task"."completed" AND "task_processor_task"."num_failures" < 3 AND "task_processor_task"."task_identifier" = 'edge_request_forwarder.forward_identity_request');
                                                                       QUERY PLAN                                                                        
---------------------------------------------------------------------------------------------------------------------------------------------------------
 Aggregate  (cost=69541.34..69541.35 rows=1 width=8) (actual time=5.017..5.018 rows=1 loops=1)
   ->  Index Scan using incomplete_tasks_idx on task_processor_task  (cost=0.38..68634.33 rows=362804 width=0) (actual time=0.018..5.013 rows=4 loops=1)
         Filter: ((task_identifier)::text = 'edge_request_forwarder.forward_identity_request'::text)
         Rows Removed by Filter: 1
 Planning Time: 0.305 ms
 Execution Time: 5.037 ms
(6 rows)

How did you test this code?

Adds unit test cases

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 4, 2023 2:21pm
flagsmith-frontend-preview	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 4, 2023 2:21pm
flagsmith-frontend-staging	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 4, 2023 2:21pm

[github-actions]

Uffizzi Preview deployment-37576 was deleted.

[novakzaballa]

@gagantrivedi , can you please add to the description what will happen when the queue size limit is reached, as you explained in the Eng Meeting, just to have it documented here?

[matthewelwell]

Added one minor comment about the implementation but I do wonder if we might be better off implementing multiple queues (with prioritisation). I don't know if it would be much more work to do that or not?

[matthewelwell]

Think this should be queue_size: Optional[int] ?

[matthewelwell]

This should be a temporary measure until we can deprecate these tasks entirely.

[codecov-commenter]

Codecov Report

Attention: 5 lines in your changes are missing coverage. Please review.

Comparison is base (929afeb) 95.53% compared to head (1d15e17) 95.53%.
Report is 9 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2826   +/-   ##
=======================================
  Coverage   95.53%   95.53%           
=======================================
  Files         994      994           
  Lines       28072    28095   +23     
=======================================
+ Hits        26818    26840   +22     
- Misses       1254     1255    +1     

Files	Coverage Δ
api/edge_api/identities/edge_request_forwarder.py	100.00% <100.00%> (ø)
api/task_processor/exceptions.py	100.00% <100.00%> (ø)
api/task_processor/models.py	95.41% <100.00%> (+2.96%)	⬆️
.../task_processor/test_unit_task_processor_models.py	100.00% <100.00%> (ø)
api/task_processor/decorators.py	88.40% <28.57%> (-5.45%)	⬇️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.