Runbooks

Token fetch failures

Symptoms

• 5xx responses from /auth/pims/token.
• Logs show partner token request errors or timeouts.

Checks

1. Verify Secret Manager entries exist for the organization identifier.
2. Confirm partner API endpoints and credentials are valid.
3. Check for network/TLS errors in logs.

Organization not found

Symptoms

• 404 with "organization could not be found."

Checks

1. Confirm organization_id exists in Spanner.
2. Verify Spanner credentials and environment variables.

Cache churn or high latency

Symptoms

• Repeated token fetches for the same org.
• Increased latency on token calls.

Checks

1. Verify token TTLs for the provider (ezyVet ~12h, Bitwerx ~1h).
2. Check for frequent instance restarts or autoscaling.

Startup failures

Symptoms

• Service fails to start; logs show missing Sentry DSN or Spanner errors.

Checks

1. Ensure PARTNERCREDS_SENTRY_DSN secret exists.
2. Validate Spanner instance/database settings and ADC.

Health checks

• GET /api/v1/health returns basic service status.