Stream
Overview
Python 3.13 internal ingress for Spanner change streams
Stream is an internal Python 3.13 FastAPI service that receives Pub/Sub push
messages containing Spanner Change Stream events (via Dataflow). It translates
changes in the Pipelines table into Cloud Tasks for the Handler service.
Service profile
| Field | Value |
|---|---|
| Code | services/stream/ |
| Package | stream |
| Runtime | Python 3.13 (FastAPI) |
| Status | Active |
| Primary owner | Joe Pardi |
| Secondary owner | Akansh Divker |
| Ingress | Internal only (Pub/Sub push) |
| Data source | Spanner Change Streams (Dataflow) |
Responsibilities
- Accept Spanner Change Stream events from Pub/Sub.
- Enrich pipeline metadata by querying Spanner.
- Schedule Cloud Tasks for Handler based on PIMS integration and pipeline rules.
Non-goals
- Serving public APIs or direct customer traffic.
- Performing the ETL itself (delegated to Handler).
Dependencies
- Upstream: Pub/Sub push from Dataflow change stream pipeline.
- Downstream: Cloud Tasks; Handler service.
- External: Cloud Spanner; Secret Manager; Sentry.
Related pages
Last updated on