Repository navigation
fix(deploy): run the commit's image digest and fail on ECS rollback - #187
Conversation
Every deploy registers a task definition revision pinned to the digest docker-build pushed, instead of forcing a deployment of mutable :latest (which tf-apply could start first with the old image, and which made ecs-deploy skip the force). ecs-deploy then requires the deployment's rolloutState COMPLETED on that revision and prints stopped-task reasons, container exit codes and the log group on rollback, failure or timeout. The ecs-service module creates the service with 0 tasks and ignores task_definition and desired_count afterwards; ecs-deploy sets the count from the javabin:desired-count tag, so a new app's first deploy no longer starts tasks before its image exists. latest/date tags follow the repo's default branch instead of a literal main/master. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Terraform Plan🚧 Changes detected — Plan: 0 to add, 4 to change, 0 to destroy. Plan outputLLM ReviewRisk: 🟡 MEDIUM Plan updates 4 CI deploy IAM role inline policies (ecs-deploy) for the cupcake, javazoneweb, jzwatch and testteam roles to add ECS read/tag/task-definition permissions; no creates or destroys.
|
…holds The deploy credentials last 900 s but ecs-deploy waited up to 1200 s, so a slow rollout died on ExpiredToken without diagnostics. The timeout is now 720 s from the start of the step, and an AWS error while polling fails the job with the last known service events, stopped tasks and log group. Revisions take the service's tags when the base revision has none, so one untagged revision no longer makes every later one untagged, and carry managed-by=ecs-deploy instead of the base's managed-by=terraform. budget-enforcer sets javabin:desired-count to 0 on the services it scales down, so app pushes keep them at 0 until a reviewed tf-apply writes the count back. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Review follow-up in c689b48. I fixed all four findings, and none of them turned out to be wrong.
Verification:
Rollout order: the iam/lambdas apply has to land before the scripts take effect, as before. Until then the tag reads fail, and revisions are registered untagged but self-heal through the service-tag fallback. 🤖 Generated with Claude Code |
Terraform Plan🚧 Changes detected — Plan: 0 to add, 6 to change, 0 to destroy. Plan outputLLM ReviewRisk: 🟢 LOW Plan updates 6 existing resources in place (0 add, 0 destroy): five IAM policy changes for CI deploy roles and the budget enforcer, plus a Lambda code update; no creations, deletions, or public exposure.
|
Combine the suspension marker with main's javabin:desired-count tag (#187): ecs-deploy now scales to the tag and Terraform ignores desired_count, so the ecs-service module plans the tag as "0" while the team is suspended. The enforcer pins the tag to 0 on suspend and reconcile, records the previous value in the marker, and restores it on lift. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Problem
Three deploy-correctness findings from the DX audit:
:latest. docker-build runs after tf-apply, so when tf-apply changed the task definition ECS started a deployment that resolved the old:latest.ecs-deploy.shthen saw more than one deployment, skipped--force-new-deploymentand only waited, so the new image never ran. Separately,latestandmain-*tags were pushed only when the branch was literallymainormaster, althoughjavabin.ymldeploys from whatever the default branch is.aws ecs wait services-stablepasses after a circuit-breaker rollback. The 2027.javazone.no run 41025435178 is green, but the service hasrolloutState FAILEDand 0/1 tasks running.SERVICE_TASK_START_IMPAIREDalert to Slack) before docker-build pushed anything.Change
scripts/ecs-deploy.sh(rewritten): takes the family's latest ACTIVE revision, which is Terraform's newest or the last one this script registered from it. It registers a copy with the app container pinned toIMAGE(repo@sha256:…) and copies the base revision's tags. Then it updates the service to that revision, with the desired count from the service tagjavabin:desired-count, and polls until that deployment isCOMPLETEDon the new revision (timeout 20 min). If the deployment rolls back, endsFAILEDor times out, the script prints the service events, the stopped tasks'stopCode/stoppedReason, each container's exit code and reason, and the log group, then fails the job. If a later run's newer revision replaced the deployment, the script passes with a warning.docker-build.yml: newimageoutput ({repo}@{digest}from build-push-action).latestandmain-*tags now followgithub.event.repository.default_branch. Themain-prefix stays so the ECR lifecycle rule still applies.ecs-deploy.yml/javabin.yml: passimagethrough.image_tagis still accepted but unused.terraform/modules/ecs-service: the service is created withdesired_count = 0, haslifecycle { ignore_changes = [task_definition, desired_count] }, and is taggedjavabin:desired-count = var.desired_count. A new service has nothing to start until ecs-deploy gives it a pinned image and its count. tf-apply never starts a deployment by itself any more.terraform/platform/iam(ci_deploy_ecs): grants the deploy roleecs:ListTagsForResourceandecs:DescribeTasks(team-tag scoped),ecs:ListTasks(scoped to the platform cluster), andecs:TagResourceonly whenecs:CreateAction = RegisterTaskDefinition. Tag-resource authorization isonin the account.docs/reusable-workflows.md,docs/reusable-modules.md,docs/app-yaml-reference.md,docs/cost-management.mdandCLAUDE.mdare updated. The generator gets a comment on the image line; its output is unchanged.The plan-review and apply-gate flow is unchanged. ecs-deploy still runs after tf-apply and only when docker-build succeeds.
Transition for existing apps
Live state (read-only check): cupcake-cupcake:4, cupcake-cupcake-backend:1, jzwatch-jz-watch:1 and javazoneweb-javazone-2027:1 all run the generated module with the untagged repo URL. testteam-platform-test-app:1 runs
:latestfrom legacy inline TF.ignore_changeskeeps the live count and revision, so there is no redeploy from tf-apply. The following ecs-deploy registers the first pinned revision from the current one. The jq transform was checked against the live cupcake-cupcake:4: every remaining key is a valid RegisterTaskDefinition parameter.@main), but the IAM change waits for the platform pipeline. Until then the script registers untagged revisions, keeps the current desired count, and reports that it cannot list stopped tasks. The deploy itself still works with today's permissions (DescribeTaskDefinition, RegisterTaskDefinition, UpdateService, PassRole).ignore_changes. Each of its plans will show the service drifting back to its own revision until it is regenerated onto the modules. It still works, but expect a plan diff on every run.Verification
python3 -m unittest discover -s scripts/tests: 149 tests OK. The 16 new tests intest_ecs_deploy.pyrun the script against a fakeawsand cover:python3 -m unittest discover -s terraform/lambda-src/tests: 219 OKterraform fmt -recursive -check, plusterraform validateformodules/ecs-serviceandterraform/platformshellcheck -s sh scripts/ecs-deploy.shandactionlinton the three workflows: cleanNot exercised end to end: no AWS writes from here. The first app run after merge and platform apply is the real test.
🤖 Generated with Claude Code