Architecture overview

This section summarizes the platform at a high level. Detailed service, job, and app documentation lives in the dedicated Services, Jobs, and Apps sections.

Platform summary

• Monorepo for data ingestion, tooling, and client experiences.
• Core runtime is Google Cloud Run backed by Cloud Spanner.
• Data ingestion is orchestrated via Cloud Tasks with per-vendor pipelines.
• Client access is provided by public ingress services (Interface/Auth) and the Raven ConnectRPC backend.

Core data flows

Spanner Change Streams -> Dataflow -> Pub/Sub -> Stream -> Cloud Tasks
                                                    |
                                                    v
                                             Handler (ETL)
                                                    |
                                                    v
                                               Cloud Spanner

Bubble triggers -> Webhooks -> Cloud Tasks -> Handler
Raven Desktop -> Raven (ConnectRPC) -> Spanner

Service boundaries

• Internal ingress: Stream, Handler, Webhooks (Cloud Tasks).
• Public ingress: Interface (legacy read-only), Auth (partner auth).
• Client backend: Raven (ConnectRPC over HTTP/2).

Shared foundations

• packages/python/common: Spanner client, models, secrets, shared utilities.
• packages/python/nest_protos: generated protobuf/Connect stubs.
• proto/: Buf workspace for shared proto definitions.

Where to go next

• Infrastructure (Terraform)
• CI/CD and Deployments
• Services
• Jobs
• Apps