Runbooks

Stalled pipeline warnings

Symptoms

• Logs show warnings that pipelines may be stalled.

Checks

1. Verify Stream and Handler are healthy.
2. Inspect the affected pipeline in Spanner (Pipelines.last_modified).
3. Confirm expected delays for that pipeline and PIMS.

Spanner access failures

Symptoms

• Job fails to start or read from Spanner.

Checks

1. Verify SPANNER_INSTANCE_ID and SPANNER_DATABASE_ID.
2. Ensure ADC is configured for the service account.

Missing Sentry DSN

Symptoms

• Startup failure with Sentry DSN error.

Checks

1. Ensure WATCHER_SENTRY_DSN exists in Secret Manager.