ci(build): Reusable private cloud build

[khvn26]

Thanks for submitting a PR! Please check the boxes below:

• I have run pre-commit to check linting
• I have added information to docs/ if required so people know about the feature!
• I have filled in the "Changes" section below?
• I have filled in the "How did you test this code" section below?
• I have used a Conventional Commit title for this Pull Request

Changes

Contributes to #3718.

This pull request improves our build toolchain to clarify our shipping targets and make building them more accessible. It integrates Private Cloud builds into our platform PR workflow, allowing us to run the E2E test suite against it. It streamlines our deployment workflows, allowing deploy jobs to benefit from Depot caching and cleanly segregates deployment configurations between GitHub environments. It sets up Trivy Docker image scan result uploads to the repo's Security tab, enabling us to address detected vulnerabilities ahead of releases. This PR is large, so I'll break down the changes below.

Single Dockerfile for all targets

I consolidated multiple Dockerfiles into a single Dockerfile containing all build stages. We ship multiple Docker targets:

• Private cloud: Unified
• SaaS: API
• Open Source: API, Frontend, Unified

Previously, Private Cloud and SaaS were awkwardly distributed between API and Unified Dockerfiles and corresponding GHA workflows. The private backend code was manually pulled by the workflow, and a special build argument controlled whether the dependencies for it should be installed. SaaS deployments required specifying private backend code versions for each environment. Changes to the private side required touching two Dockerfiles and 2-3 workflows and hoping the build worked since it was only reproducible in GHA.

Now, every target, including Private Cloud and SaaS, can be built using docker build, provided the required secrets are mounted. The final stages are minimal, assembled from building blocks factored out of our former Dockerfiles and GHA workflows. All stages are documented for easy navigation.

The CI_COMMIT_SHA, .versions.json, and edition markers are now explicitly requested by the Docker build instead of being added by GHA workflows. Permanent caching for the backend's get_version_info utility has been added.

To demonstrate improved build sanity, I've added a Private cloud API target. The stage's code is just 5 lines. I've also created private-cloud-api private repositories in our GitHub Container registry and Docker Hub organization.

The docker build -t flagsmith:local . command remains fully backward compatible due to the oss-unified stage being the last one.

Private cloud images built for PRs and on merge to main

The "Platform Pull Requests" workflow now builds a Unified Private cloud image, enabling faster testing of enterprise features. E2E tests are now run against both Open Source and Private cloud API image versions. We can also deliver improvements and bug fixes to enterprise clients ahead of releases by offering a PR or a staging image.

ECS deployments with less fat

Deployment configuration data values have been moved to their respective GHA environments, allowing a reusable ECS deployment workflow for both staging and production. The workflow uses Depot and the new Dockerfile to leverage cached build stages. The change also results in less infrastructure-specific data in our workflow code.

Platform workflow publishes all Docker images

All Docker image building and publication have been consolidated in .github/workflows/platform-docker-build-test-publish.yml, triggered on merges to main and releases. Images are built for every merge and release, and published on release only if an E2E run is successful. After successful image publication, a PR to flagsmith-charts is created. Individual Docker image publication workflows have been removed.

CodeQL reports

CodeQL has been enabled for the repository. In addition to free code scanning, Docker image scan results are now available as part of CodeQL reports. Alerts/CI failures will occur on newly detected critical issues.

Bits and Pieces

• Added the install-private-modules Makefile target for use outside of Docker builds, i.e., by the "Run API tests with private package" workflow.
• Removed .ubi Dockerfiles as they are no longer needed.
• Removed E2E image publication to Docker Hub; those needing it can pull from ghcr.io/flagsmith/flagsmith-e2e.

To do / to think on

• Consider making the SSE PGP key a runtime dependency to build the SaaS API target once per deployment, not per environment.
• Avoid invoking docker build on release trigger; re-tag the main branch images instead. To do that, ensure release jobs run after the latest main branch builds complete, e.g. using a concurrency group.
• Reevaluate Uffizzi's reusable workflow to remove boilerplate.
• Consider adding the E2E image to the mega-Dockerfile targets.
• Avoid duplicating the Private cloud build for the "Run API tests with private package" workflow.

How did you test this code?

Docker builds were tested locally with docker build.

We need a plan for testing the ECS deployment workflow.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
flagsmith-frontend-preview	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jun 20, 2024 4:41pm
flagsmith-frontend-staging	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jun 20, 2024 4:41pm

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
docs	⬜️ Ignored (Inspect)	Visit Preview		Jun 20, 2024 4:41pm

[github-actions]

Uffizzi Preview deployment-53072 was deleted.