:: g3

Google Drive Storage Gmail Metadata SQLite / PostgreSQL Prometheus Metrics OpenTelemetry Tracing

S3 gateway backed by Google Drive and Gmail

A Go service that presents an S3-compatible HTTP API and stores objects using Google’s free storage. Object data lives in Google Drive files (no size limit). Gmail emails serve as metadata pointers. A local SQLite index eliminates API calls for metadata-only operations like HeadObject and ListObjects. Designed for write-once/read-rarely workloads like offsite backups.

Any S3 client works – AWS CLI, s3cmd, SDKs, or s3-orchestrator as a backend
Drive hybrid storage eliminates Gmail’s 25 MB attachment limit – no chunking needed
Metadata index (SQLite or PostgreSQL) makes HeadObject and ListObjects instant with zero API calls
Dual API quota pools – Drive and Gmail operate on separate rate limits
Full observability with Prometheus metrics, OpenTelemetry traces, and structured JSON logging

Key Features

S3-Compatible API

PUT, GET, HEAD, DELETE, ListObjectsV2, multipart uploads, and bucket operations.

Implements the S3 API surface needed for backup workloads. Works with the AWS CLI, any S3 SDK, and s3-orchestrator as a backend target. SigV4 request authentication with per-bucket credentials.

Drive + Gmail Hybrid Storage

Object data in Google Drive, metadata pointers in Gmail emails.

Drive files store the actual object data via resumable upload with no size limit (up to 5TB). Gmail emails contain JSON metadata with the Drive file ID, ETag, and user metadata. Separate API quota pools nearly double total throughput.

Metadata Index

Local or shared database eliminates API calls for metadata-only operations.

HeadObject and ListObjects resolve entirely from the metadata index with zero API calls. Supports SQLite (single-node, persistent volume) or PostgreSQL (multi-node, shared database). Index is populated automatically on writes and recoverable via g3 sync.

Multipart Upload

Standard S3 multipart protocol for large file uploads from any client.

Parts are buffered individually in memory. On CompleteMultipartUpload, parts are streamed in order into the Drive upload without assembling into a single buffer. Abandoned uploads are cleaned up after 1 hour.

Prometheus Metrics

Request counts, latency histograms, Gmail/Drive API metrics, and operational gauges.

11 metric families prefixed with g3_ covering HTTP requests, API calls to both Gmail and Drive, inflight tracking, storage estimates, and build info. Exportable to any Prometheus-compatible monitoring stack.

OpenTelemetry Tracing

Server spans for S3 requests with client spans for every Gmail and Drive API call.

Exports traces via OTLP gRPC to Tempo or any OpenTelemetry-compatible backend. Custom g3.* attributes on every span. trace_id and span_id injected into structured logs for correlation in Grafana.