Runbooks

Missing pipeline executions

Symptoms

• Downstream data stops updating for a PIMS integration.

Checks

1. Verify Stream is scheduling Cloud Tasks for the pipeline.
2. Inspect Cloud Tasks queues for pending/failed tasks.
3. Check Handler logs for pipeline execution errors.
4. Confirm pipeline status updates in Spanner.
5. Ensure DEBUG is not enabled in production (status updates are disabled).

Partner credentials failures

Symptoms

• Pipeline fails with auth or token errors.

Checks

1. Ensure PartnerCreds is reachable and healthy.
2. Validate Secret Manager entries for the organization (ezyVet secrets).
3. Confirm PARTNERCREDS_SERVICE_BASE_URL is correct.

Pipeline timeouts

Symptoms

• Tasks fail after ~PIPELINE_MAX_RUNTIME_SECONDS.

Checks

1. Confirm the timeout value for the environment.
2. Inspect Spanner write volume for oversized batches.
3. Reduce pipeline scope (smaller last_modified window) to unblock.

Spanner write errors

Symptoms

• 5xx errors and Spanner exceptions during pipeline load.

Checks

1. Verify service account access to Spanner.
2. Check Spanner schema compatibility with pipeline outputs.
3. Inspect logs for failed mutations or constraint violations.

Bubble webhook processing failures

Symptoms

• Bubble-triggered updates or invoice forwarding stalls.

Checks

1. Confirm Webhooks is delivering tasks to Handler.
2. Validate BUBBLE_API_TOKEN and BUBBLE_INVOICES_ENDPOINT.
3. Review pipeline logs for webhooks.* errors.

Duplicate Cloud Tasks

Symptoms

• Task invocations repeat for the same payload.

Checks

1. Check Cloud Tasks retry settings in the queue.
2. Confirm idempotency keys are set for invoice forwarding tasks.
3. Verify pipeline logic tolerates duplicates (upserts vs. inserts).