Time Series Database in Go

I wanted to understand how time series databases actually work.

Building this taught me:

• How write-ahead logs provide crash recovery guarantees
• Why segment-based storage is perfect for time-bounded data
• How delta encoding exploits time series patterns to save 30-50% storage
• The tradeoffs between write durability and query performance

What I Actually Built

My TSDB includes these core components:

• Write-Ahead Log (WAL) - Every write goes to WAL first with sync before storage. Crash recovery replays uncommitted writes.
• Segment-based Storage - Time-bounded files that auto-rotate at 1MB. Each segment knows its time range for fast filtering.
• Delta Compression - Timestamps use delta encoding, values use XOR encoding. Small deltas = big savings.
• Query Language - PromQL-like syntax with time ranges and aggregations: cpu.usage[1h], avg(memory[30m])
• HTTP API - REST endpoints for writes and queries
• Inverted Index - Maps metric names to segments for faster lookups

How It Works

Write Path

When a metric comes in, it flows through multiple layers:

First, the metric gets encoded to pipe-delimited format: cpu.usage|45.2|1000. Simple but works.

Then it hits the WAL. This is critical for durability:

1. Encode the metric with a 4-byte length prefix
2. Append to WAL file
3. Call sync to force disk flush
4. Only then write to storage

If the process crashes between WAL sync and storage write, recovery replays the WAL on startup. Nothing lost.

The WAL file looks like this:

[length][data][length][data][length][data]...
[0x18 0x00 0x00 0x00][cpu.usage|45.2|1000]...

Length prefix lets recovery parse record boundaries even in corrupted files.

After WAL sync, the metric goes to storage. Segments are append-only files organized by time. When a segment hits 1MB, it rotates to a new file.

Storage Architecture

Segments are the heart of storage. Each one covers a time range:

segment_abc123.dat  [Jan 1 10:00 - Jan 1 10:15]  1.2MB
segment_def456.dat  [Jan 1 10:15 - Jan 1 10:30]  1.1MB
segment_ghi789.dat  [Jan 1 10:30 - Jan 1 10:45]  0.8MB (active)

Metrics in each segment are separated by newlines:

cpu.usage|45.200000|1000
cpu.usage|52.100000|1060
memory.usage|1024.500000|1100

When querying cpu.usage[1h], the engine:

1. Calculates time range: now-3600 to now
2. Scans segments, skipping ones outside the range
3. Reads matching segments line by line
4. Filters by metric name and timestamp
5. Returns sorted results

Only segments that overlap the query window get scanned. If you query last 5 minutes but have 24 hours of data, most segments are skipped.

Compression

Time series data has patterns you can exploit.

Timestamps increase steadily. Instead of storing [1000, 1060, 1120, 1180], store [1000][60][60][60]. Delta encoding.

With varint encoding for the deltas, you can shrink timestamps by 50-70%.

Values change slowly. For floats like [45.2, 45.3, 45.1], convert to bits and XOR consecutive values:

bits1 = 0x4046999999999999
bits2 = 0x404699999999999a
XOR   = 0x0000000000000003  (tiny!)

Similar floats have similar bit patterns. XOR produces small numbers that compress well.

Query Language

Parser supports PromQL-like syntax:

cpu.usage[1h]           - Last hour
cpu.usage[2h:1h]        - 2 hours ago to 1 hour ago
avg(cpu.usage)[30m]     - Average over 30 minutes
max(memory.usage)[1d]   - Max in last day

Time units: s (seconds), m (minutes), h (hours), d (days)

Aggregations: avg, max, min, sum

The parser uses regex to extract metric name, time range, and aggregation function. Then the executor calls the engine's Query method and applies aggregation to results.

API Usage

Write Metrics

curl -X POST http://localhost:8080/api/v1/write \
  -H "Content-Type: application/json" \
  -d '{"name":"cpu.usage","value":45.2,"timestamp":1782048900}'

Query Metrics

# Last hour
curl "http://localhost:8080/api/v1/query?q=cpu.usage%5B1h%5D"

# With aggregation
curl "http://localhost:8080/api/v1/query?q=avg(cpu.usage)%5B30m%5D"

Or use --data-urlencode to avoid manual URL encoding:

curl -G http://localhost:8080/api/v1/query \
  --data-urlencode "q=cpu.usage[1h]"

Health Check

curl http://localhost:8080/health

Configuration

Create a .env file:

HOST=localhost
PORT=8080

Defaults to localhost:8080 if not specified.

Running

# Start server
go run cmd/main.go

Server starts with:

• WAL recovery (replays uncommitted writes from previous session)
• Test metrics insertion (writes a few sample metrics)
• HTTP API on configured port

Press Ctrl+C for graceful shutdown.

Project Structure

internal/
├── api/          - HTTP server and handlers
├── compression/  - Delta and XOR encoding
├── config/       - Environment variable loading
├── engine/       - Coordinates WAL + storage
├── index/        - Metric name and time indexes
├── ingestion/    - Batch writer with buffering
├── model/        - Metric struct and encoding
├── query/        - Query parser and executor
├── storage/      - Segment-based storage
└── wal/          - Write-ahead log

cmd/
└── main.go       - Entry point

data/
├── wal/          - WAL files
└── segments/     - Segment files

What I Learned

Durability is expensive. Every write needs WAL sync, which means disk I/O. You can batch writes to amortize the cost, but single writes will always be slow.

Indexing matters. Sequential scans don't scale. Even with time-based filtering, scanning segments is slow with lots of data. Inverted indexes help but aren't enough - you need bloom filters or bitmap indexes for production.

Compression works. Delta encoding cut storage by 30-50% on real time series data. The pattern is predictable: timestamps increase steadily, values change slowly.

Recovery is critical. Crashes happen. WAL recovery needs to handle partial writes, corrupted records, and missing data. Skip bad records, keep processing.

Simple encodings win. Pipe-delimited strings are easy to debug and parse. For learning, they beat protobuf or custom binary formats. Performance comes later.

Built this to understand databases at the code level. No abstractions, no magic, just file I/O and data structures.