Populate flagsmith_environments_v2 from flagsmith_identities

[matthewelwell]

We need a way to migrate the data in the identity_features attribute on each identity in an environment into the new flagsmith_overrides table.

This should be a one off migration process which we can create a Task in the task processor for.

[khvn26]

1. Have a DDB CDC stream consumer that filters identity documents by given environments, maps them to overrides and writes them to a new table.
2. When migration prompted, add an ephemeral "_migrate_requested": timestamp() attribute to all of the project's identity documents (how do we get those?). This will produce the update events in the stream.
3. Flip the switch in the project settings so the subsequent identity overrides are written to the new table. (make sure that deletions are imdepotent there).
4. Run the consumer for all the affected environments until it starves out (what would be the criteria here?)

[khvn26]

This is now going to be a permanent worker process that:

• consumes all updates in the flagsmith_identities table.
• skips identities with identity_features = [].
• writes identity features to flagsmith_environments_v2 with a correct partition/sort key. Refer to schema defined in https://github.com/Flagsmith/infrastructure/issues/18.

[khvn26]

This is solved as a task processor task in the Core API that's triggered on EdgeIdentity.save.

[khvn26]

The task processor code is merged.

For the backfill/migration part, we need to call the task with the changeset that updates all the identity overrides in the project.