Context
In the earlier model, every ML repository created its own storage buckets and encryption keys, and fixes to pipeline or promotion logic had to be repeated project by project.
MLOps v3, the next version of the platform, answers that with a hub-and-spoke architecture built on AWS CDK (Python). Platform concerns are built once in a shared hub, and every ML project, called a spoke, reuses them. I am designing and building it this year as Senior AI Engineer, building on CarePath, the multi-account platform behind 37 clinical prediction models.
What I built
- Shared data tiers per account and region, each encrypted with its own customer-managed KMS key. Each project writes only under its own prefix, so many projects share one set of buckets instead of creating their own.
- Convention-based contracts instead of code coupling: projects discover shared resources through published parameters and follow naming rules the hub already trusts, so onboarding a new model needs no hub change.
- Least-privilege security by default: a role per project scoped to its own prefix, a permission boundary on promotion roles, and cross-account grants that only let approved artifacts move forward, from dev to stage to prod.
- Self-mutating CDK Pipelines in a central Shared Services account, one per region, that deploy every project to dev, stage, and prod, with a manual approval before production.
- Registry-first model promotion: advancing a version's lifecycle stage in the dev model registry emits an event that an orchestrator Lambda checks for eligibility. It then writes a promotion manifest that starts the target environment's pipeline, which copies the artifact into approved storage, waits for human approval in prod, registers the version idempotently, and retires the previous release.
- Multi-region disaster recovery: cross-region replication of approved artifacts, an event-driven model registry mirror with scheduled reconciliation, secondary-region pipelines that wait for the primary, and an operator-controlled failover flag kept outside infrastructure code so a redeploy cannot reset it mid-incident.
- One configuration file per project, seeded with organization-wide values, that drives accounts, networking, approvals, and the tags required by Shared Services tag enforcement.
Design principles
- 01Dev is the only place humans decideStage and prod registries are projections that only automation writes.
- 02Approval is not movementApproving makes a version eligible. Advancing its lifecycle stage is what moves it.
- 03Prod keeps a human gateAn artifact can reach prod storage, but it becomes servable only after approval.
- 04Everything is idempotentDuplicate events, retries, and bursts converge on the same result, and batching coalesces bursts.
- 05Rollback is a forward promotionRolling back means promoting an earlier good version through the same path.
Results
- New ML projects start from a generated, ready-to-deploy repository that is already wired to the shared platform.
- One set of buckets and keys per account replaces per-repository storage and key sprawl.
- Pipeline and promotion fixes ship once through coe-utils, and template fixes reach existing projects through copier update.
- Model versions move between environments without redeploying code, and prod cannot receive a model that has not reached stage.