Upgrade to 2.100.0 can fail when app_analytics_featureevaluationraw table is large

[matthewelwell]

When the app_analytics_featureevaluationraw table is large, this migration added in 2.100.0 can be problematic.

When the migration runs, it adds the new columns and then starts to add the new index (concurrently). If this migration times out (we may need to look at whether there is a timeout configured on the migrate-db init container in our helm chart for example) then it will never write to the django_migrations table to tell it that the migration completed.

It may or may not also be relevant that, in order to support adding the index concurrently, the migration is configured with atomic = False. This means (I think, although I'm not certain that atomic=False is the issue here) that the columns which were added are not rolled back and hence, when it tries to run again it complains that the identity_identifier column already exists.

My train of thought at the moment is such that there are 2 things that we need to investigate first:

1. Is there a timeout on the migrate-db init container? If so, should we remove / increase it?
2. Should we move the creation of the index into it's own separate migration with an IF NOT EXISTS clause?

If we implement (2) only, this might mean that the migration will still fail at least the first time in these scenarios but it would perhaps mean that it would work on retry. Is there a better solution to this?