Runbooks

API key failures (401/500)

Symptoms

• Clients receive 401 Unauthorized or {"detail": "API key is missing"}.
• Service returns 500 with "API key error" on every request.

Checks

1. Confirm the request includes X-API-Key.
2. Verify the WEBHOOKS_API_KEY secret exists in Secret Manager.
3. Ensure the service account has secretmanager.versions.access.
4. Restart the service to reload secrets.

Task scheduling failures

Symptoms

• Webhook requests return 500 with enqueue errors.
• Cloud Tasks show missing tasks or repeated failures.

Checks

1. Verify CLOUD_TASKS_* settings and HANDLER_SERVICE_BASE_URL.
2. Confirm the Cloud Tasks queue exists in the configured location.
3. Ensure the service account has permission to create tasks.
4. Check Handler availability for target endpoints.

Payload validation errors

Symptoms

• 422 responses or logs referencing validation errors.
• 413 Request payload too large.

Checks

1. Ensure the webhook payload matches the expected schema.
2. Confirm invoice counts do not exceed 100 per request.
3. Keep payload size below 750 KiB.

Bubble sync issues

Symptoms

• Bubble-triggered updates do not appear in Spanner.

Checks

1. Verify Bubble triggers are sending RawEvent payloads.
2. Confirm tasks are created and dispatched to Handler.
3. Inspect Handler logs for processing failures.