feat: Create split testing for multivariate

[zachaysan]

Thanks for submitting a PR! Please check the boxes below:

• I have run pre-commit to check linting
• [ ] I have added information to docs/ if required so people know about the feature!
• I have filled in the "Changes" section below?
• I have filled in the "How did you test this code" section below?
• I have used a Conventional Commit title for this Pull Request

Changes

Edit: These changes are out of date after multiple rounds of review and now with ongoing work to pull the split testing code into it's own repository.

Overall, the changes introduced here support adding split testing to Flagsmith for the customers of ours that use the settings.USE_POSTGRES_FOR_ANALYTICS option, so this does not support our current SaaS method. The basic idea is that each of our multivariate options are linked into a SplitTest model and serve as the source of identification as to which split test is represented. In detail changes include:

1. A modification to FeatureEvaluationRaw to accept a new identifier so we can tell when a user has encountered a feature. This has been implemented as a query param since the body of the request just has feature names. Also, FeatureEvaluationRaw needed faster lookups so an index on feature_name was added concurrently as well.
2. An introduction of the chi2_contingency summary statistic that follows the G-Test version of split testing which was verified as current on Wikipedia as well as elsewhere. This comes with all of scipy which may be a drawback on some systems, but if there is significant issues with it we can always extract the one function we need out into its own helper.
3. The creation of a new tracking metric called ConversionEvent which can be called for events like new user signup, just as a normal analytics app would. This is used to track the actual split test results.
4. After a recurring task has processed the latest SplitTest results they are available via a SplitTestViewSet which serializes the related attributes of feature and multivariate_feature_option which reused serializers from existing areas of the codebase.

How did you test this code?

Around seven or eight new tests, mostly on the new functionality. I wasn't quite happy with the split test task test since it is not deterministic and there didn't seem to be a clean way to force some identities to some multivariate options and other ones to others. Also, it would have been nice to test it with more objects, but with the size of the objects that would have been necessary, the statistics part was tested with raw numbers instead.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jan 31, 2024 7:42pm
flagsmith-frontend-preview	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jan 31, 2024 7:42pm
flagsmith-frontend-staging	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jan 31, 2024 7:42pm

[matthewelwell]

@zachaysan on the whole, this is really impressive. I have gone through with quite a fine-toothed comb so there are a fair amount of comments but most of them are my lack of knowledge in the world of statistics, or a little bit nit-picky :)