feat: Add backoff to webhooks, and call multiple webhooks async

[tushar5526]

Closes #1654
Thanks for submitting a PR! Please check the boxes below:

• I have run pre-commit to check linting
• I have filled in the "Changes" section below?
• I have filled in the "How did you test this code" section below?
• I have used a Conventional Commit title for this Pull Request

Changes

Significant changes include calling call_environment_webhooks and call_organisation_webhooks using the task processor and making _call_webhook_email_on_error async so that any unhealthy webhook in the list does not block sending payload to other webhooks.

I have also added backoff and jitter to _call_webhook with 3 retries and using a max_time of 2s so that threads are not marked unhealthy by the task processor's healthcheck.

How did you test this code?

I ran unit tests and set up two endpoints as webhooks for testing. Next, I created a feature flag and added those webhooks to it. I then called those webhooks by toggling the feature and tested for various combinations.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Sep 12, 2023 2:23pm
flagsmith-frontend-staging	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Sep 12, 2023 2:23pm

[vercel]

@tushar5526 is attempting to deploy a commit to the Flagsmith Team on Vercel.

A member of the Team first needs to authorize it.

[github-actions]

Uffizzi Preview deployment-32279 was deleted.

[matthewelwell]

Thanks for submitting this @tushar5526. I have added a couple of specific comments, but I'd also love to see some tests written for these too. It seems like perhaps we don't have any currently (otherwise I would have expected them to fail), but if you are able to add some, that would be really great.

[tushar5526]

Hi @matthewelwell, I modified the logic to use id instead of environment or organisation model object. Still, it fails the test case where organistaion object is deleted and associated webhooks are called later, so doing a Organisation.objects.get(id) errors out. Should the organisation webhooks be referenced, even if the organisation model object is deleted?

https://github.com/Flagsmith/flagsmith/pull/2559/files#diff-86f132934c5b75bd30cb37ce30cc88b07b738d2b09d772cbeea42e2186809cbaR73

[matthewelwell]

Hi @matthewelwell, I modified the logic to use id instead of environment or organisation model object. Still, it fails the test case where organistaion object is deleted and associated webhooks are called later, so doing a Organisation.objects.get(id) errors out. Should the organisation webhooks be referenced, even if the organisation model object is deleted?

https://github.com/Flagsmith/flagsmith/pull/2559/files#diff-86f132934c5b75bd30cb37ce30cc88b07b738d2b09d772cbeea42e2186809cbaR73

No, we can ignore the case where the organisation is deleted.

[codecov-commenter]

Codecov Report

Patch coverage: 94.73% and no project coverage change.

Comparison is base (ecb7fd2) 95.51% compared to head (72b1676) 95.51%.
Report is 9 commits behind head on main.

❗ Current head 72b1676 differs from pull request most recent head fdba296. Consider uploading reports for the commit fdba296 to get more accurate results

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2559   +/-   ##
=======================================
  Coverage   95.51%   95.51%           
=======================================
  Files         993      993           
  Lines       27971    28023   +52     
=======================================
+ Hits        26717    26767   +50     
- Misses       1254     1256    +2     

Files Changed	Coverage Δ
api/tests/unit/audit/test_unit_audit_tasks.py	100.00% <ø> (ø)
api/webhooks/webhooks.py	93.06% <90.32%> (-2.06%)	⬇️
...k_processor/test_unit_task_processor_decorators.py	94.02% <90.47%> (-1.63%)	⬇️
api/audit/signals.py	82.60% <100.00%> (ø)
api/edge_api/identities/tasks.py	97.72% <100.00%> (ø)
api/features/tasks.py	96.15% <100.00%> (-0.15%)	⬇️
api/features/tests/test_tasks.py	100.00% <100.00%> (ø)
api/task_processor/decorators.py	93.93% <100.00%> (+7.40%)	⬆️
api/task_processor/exceptions.py	100.00% <100.00%> (ø)
api/tests/unit/audit/test_unit_audit_models.py	98.78% <100.00%> (ø)
... and 2 more

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[matthewelwell]

I've created this PR to validate the arguments that are passed to task processor functions both when called directly, or when executed synchronously. This should help to ensure that all new tasks adhere to the serialization rules.

Based on this, in terms of this PR, I don't think there is any additional logic required. I will review for a final time and then, once my separate PR is merged, we can merge it here and re run the tests as a final check.

Hopefully that all makes sense?

[matthewelwell]

This looks good to me, but I'd appreciate another pair of eyes from @gagantrivedi or @khvn26 before merging.

[gagantrivedi]

LGTM, apart from Minor test improvements