Source connector

Google Cloud Storage connector

Google Cloud Storage integration for databases, warehouses and BI tools

Object storage for files and data lake exports on GCP. Datrise captures Google Cloud Storage entities such as buckets, objects, prefixes, object metadata, generations and storage classes, then lands normalized and typed models for warehouse reporting, attribution, and AI workflows.

The Cloud Storage JSON API v1 authenticates with service account, OAuth 2.0, incremental runs key on updated (object metadata timestamp) or generation number, and Google Cloud Storage caps calls at buckets initially support roughly 1,000 object writes/sec and 5,000 object reads/sec before autoscaling further, but writes to any single object name are capped at 1 write per second.

Of the 6 objects, 6 sync as upserts keyed on the primary key and 0 as append-only streams, so each incremental run moves only what changed since the last watermark. Every object lands in its own table (for example `google_cloud_storage_f_buckets`) with a schema derived from the Google Cloud Storage API rather than a flattened CSV, in any of the 43 supported destinations.

Datrise is in early access: pick a destination below to open the Google Cloud Storage pipeline page, and join the waitlist to get the Google Cloud Storage connector with guided onboarding.

Incremental syncSchema-aware mapping43 destinations available

What Google Cloud Storage data Datrise extracts

Each object lands in its own typed table. Sync mode is derived from the object: history-style objects append, everything else upserts.

Object	Landing table	Sync mode
buckets	google_cloud_storage_f_buckets	Upsert by primary key, incremental on updated-at
objects	google_cloud_storage_f_objects	Upsert by primary key, incremental on updated-at
prefixes	google_cloud_storage_f_prefixes	Upsert by primary key, incremental on updated-at
object metadata	google_cloud_storage_f_object_metadata	Upsert by primary key, incremental on updated-at
generations	google_cloud_storage_f_generations	Upsert by primary key, incremental on updated-at
storage classes	google_cloud_storage_f_storage_classes	Upsert by primary key, incremental on updated-at

Google Cloud Storage API: setup facts for the connector

Verified against the vendor documentation on 2026-09-02. Unknown values are left blank rather than guessed.

API	Cloud Storage JSON API v1
Authentication	service account, OAuth 2.0 — Requests carry an OAuth 2.0 Bearer access token, typically minted from an attached or key-based service account; HMAC keys are also available for S3-compatible XML API access.
Pagination	token — `objects.list` returns up to 1,000 combined `items`/`prefixes` per page (`maxResults`); follow the returned `nextPageToken` until it is absent to get the full listing.
Rate limit	buckets initially support roughly 1,000 object writes/sec and 5,000 object reads/sec before autoscaling further, but writes to any single object name are capped at 1 write per second
Incremental cursor	updated (object metadata timestamp) or generation number — There is no server-side 'modified since' filter on `objects.list`; incremental pulls filter the per-object `updated` timestamp client-side, or consume Pub/Sub notifications for near-real-time capture instead of relisting.
Webhooks	Yes — Pub/Sub notifications (OBJECT_FINALIZE, OBJECT_METADATA_UPDATE, OBJECT_DELETE, OBJECT_ARCHIVE, OBJECT_INITIALIZE)
Bulk export	Yes — Storage Transfer Service / `gcloud storage` bulk copy-out
Deletes detectable	Yes — requires Object Versioning enabled; each object change is tracked by its `generation` number and a real DELETE is observable via a Pub/Sub `OBJECT_DELETE` (or `OBJECT_ARCHIVE` for a version becoming noncurrent) notification

Official API reference

What bites when replicating Google Cloud Storage

`objects.list` caps each page at 1,000 combined `items`/`prefixes` (`maxResults`) and requires following `nextPageToken` until it is absent, or a listing silently truncates.
Writes to the same object name are limited to 1 write per second and 1 metadata update per second, so a connector that repeatedly overwrites a single rolling key (e.g. a 'latest.json' manifest) gets throttled well before the bucket-wide ~1,000 writes/sec ramp matters.
Pub/Sub notifications guarantee at-least-once delivery but explicitly not ordering, and duplicate messages for the same `OBJECT_FINALIZE` event are possible, so a consumer must dedupe rather than trust event order or count.
Delete visibility depends on Object Versioning being turned on first: `OBJECT_DELETE`/`OBJECT_ARCHIVE` notifications and any noncurrent-version history only exist for versioned buckets, tracked per-object by a `generation` number rather than a single mutable revision.

Compare this against every other documented API

Popular destinations for Google Cloud Storage

Choose a destination below to open the ready pipeline page.

A
Airtable
Relational spreadsheet destination for ops and go-to-market teams.
View pipeline
A
Amazon Athena
Serverless SQL over S3 data lake tables.
View pipeline
A
Amazon DynamoDB
Serverless key-value and document store on AWS.
View pipeline
A
Amazon QuickSight
AWS serverless BI with SPICE and embedded analytics.
View pipeline
A
Amazon Redshift
AWS petabyte-scale warehouse with Spectrum.
View pipeline
A
Amazon S3 Data Lake
Object storage landing zone for parquet and snapshots.
View pipeline
A
Apache Superset
Open-source BI for SQL exploration, charts, and dashboard publishing.
View pipeline
A
Azure Data Lake Storage
ADLS Gen2 object storage for analytics workloads.
View pipeline
A
Azure Synapse
Microsoft analytics workspace with SQL pools.
View pipeline

Google Cloud Storage connector FAQ

What Google Cloud Storage data does the connector extract?: 6 objects: buckets, objects, prefixes, object metadata, generations and storage classes. Each lands in its own table with types preserved; history-style objects (0) load as append-only streams.
Is the Google Cloud Storage integration incremental?: Yes. After the initial full load, every run reads only records changed since the last watermark (updated-at or event timestamp) and applies an upsert or an append depending on the object.
Where can I load Google Cloud Storage data?: Into 43 destinations: relational databases, cloud warehouses, data lakes, spreadsheets and BI tools. Each Google Cloud Storage → destination pair has its own page with the object-to-table mapping.

Related sources

Other sources in the same category, each with its own connector page.

Early access

Launch your Google Cloud Storage pipeline with guided onboarding

Join the waitlist to get priority access for Google Cloud Storage integrations with guided setup, schema-aware mapping, and production-grade incremental sync.