Platform overview

This page provides the high-level view of the Nest Engineering platform. For service and pipeline specifics, see the Services and Jobs sections.

Runtime and infrastructure

• Cloud Run hosts the backend services.
• Cloud Spanner is the system of record.
• Cloud Tasks drives internal ETL execution.
• Pub/Sub and Dataflow deliver change stream events to Stream.
• Secret Manager stores service credentials and API tokens.

Core data flows

Spanner Change Streams -> Dataflow -> Pub/Sub -> Stream -> Cloud Tasks
                                                    |
                                                    v
                                             Handler (ETL)
                                                    |
                                                    v
                                               Cloud Spanner

Bubble triggers -> Webhooks -> Cloud Tasks -> Handler
Raven Desktop -> Raven (ConnectRPC) -> Spanner

Service boundaries

• Internal ingress: Stream, Handler, Webhooks (Cloud Tasks).
• Public ingress: Interface (legacy read-only), Auth (partner auth).
• Client backend: Raven (ConnectRPC over HTTP/2).

Deployment model

• Each service has a Dockerfile under services/<name>/.
• Cloud Build builds images using ci/cloudbuild/*.yaml.
• Cloud Deploy applies manifests under services/*/deploy/ and jobs/*/deploy/.

Observability

• JSON structured logs from common.logging_utils.
• Optional Sentry integration per service (when DSN is configured).

Where to go next

• Architecture
• Infrastructure (Terraform)
• CI/CD and Deployments
• Services
• Jobs