Runbooks

Missing tasks after change stream event

Symptoms

• Pipelines updates do not trigger downstream Handler jobs.

Checks

1. Verify Dataflow is publishing change stream events to Pub/Sub.
2. Confirm Pub/Sub push subscription is delivering to /api/v1/pipeline/stream.
3. Inspect Stream logs for validation errors or scheduling failures.
4. Check Cloud Tasks queues for scheduled tasks.

Spanner lookup failures

Symptoms

• Stream returns 5xx or logs mention Spanner errors.

Checks

1. Validate SPANNER_INSTANCE_ID and SPANNER_DATABASE_ID.
2. Confirm service account has Spanner access.
3. Ensure the Pipelines table and related views exist.

Cloud Tasks errors

Symptoms

• Task creation fails or tasks are repeatedly retried.

Checks

1. Validate CLOUD_TASKS_* settings and per-PIMS queue ids.
2. Confirm queues exist and IAM permissions are correct.
3. Verify HANDLER_SERVICE_BASE_URL is reachable.

Duplicate tasks

Symptoms

• Logs show ALREADY_EXISTS warnings.

Checks

• This is expected under retries; idempotency is based on commit timestamps.
• Investigate if Dataflow is duplicating events excessively.