Rollback

Scope

• All services and jobs deployed via Cloud Deploy.
• Cloud Run rollbacks must be executed through Cloud Deploy (console or gcloud), not by redeploying Cloud Run directly.
• Spanner schema rollbacks must be executed via Liquibase, not by manual DDL.

Procedure

1. Identify the last known good release in Cloud Deploy.
2. Trigger a rollback in Cloud Deploy (GCP console or equivalent gcloud command).
3. Monitor the rollout until it completes.
4. Validate service health and key workflows.
5. Announce in #eng that rollback is in progress and when it completes.
6. Open or update an incident if user impact continues.

Database rollback (Spanner)

1. Identify the Liquibase changeset, tag, or count to roll back from db/liquibase/changelogs/spanner/nest/.
2. Execute a Liquibase rollback using the approved workflow (db/liquibase/run-liquibase.sh rollback ... or the CI pipeline that runs Liquibase updates).
3. Re-validate the application after the schema rollback completes.
4. If the service rollback depends on schema compatibility, coordinate the Cloud Deploy rollback and Liquibase rollback together.

Validation

• Confirm health endpoints and error rates return to normal.
• Verify integration-specific behavior (partner APIs, Spanner reads/writes).
• Check logs for residual errors from the failed release.

Notes

• Cloud Deploy is the source of truth for releases and rollouts.
• If schema changes are involved, verify backward compatibility before rollback.
• Liquibase is the source of truth for Spanner schema changes and rollbacks.