burneh

The platform every other cluster runs on.

Isolation, limits, and observability, the unglamorous parts that decide whether a system holds up.

platform · ops console
load
healthy

Overview

live · steady state
throughput
4.2k
req/s · +3%
p95 latency
182ms
well under SLO
error rate
0.3%
steady
queue depth
128
0 in DLQ
request throughput · last 60snominal
SLO
subsystemstatesaturation
rate limiter4.2k rps · 0.3% 42942%
job queue128 depth · 8 workers24%
traces · p95182ms · 0 slow spans30%
tenant RLS3 tenants · sealed18%
deploysv128 · canary ok12%
webhooks100% acked10%
event log
info
canary v128 promoted to 100%
2m ago
ok
indexer backfill complete
6m ago
info
nightly restore drill passed
1h ago
the boring state is the product
push the load → Quiet is the goal. The boring state is the product.

Push the load. Watch it hold.

A traffic spike gets shed by the limits. An incident degrades one subsystem, and isolation plus auto-rollback keep the rest safe. The unglamorous parts, doing their job.

Every cluster stands on the same floor.

A product, a bot, a contract, a trading system: they're all just apps, and they all run on the same platform underneath. Isolation, limits, queues, observability, and safe deploys aren't six projects, they're one floor that every cluster shares, built once and built to hold.

ProductBotsCryptoTradingthe platformisolationrate limitsqueuesobservabilitydeployswebhooksone region · one bill · one pane of glass

When one thing fails, the rest don't.

The incident you just triggered degraded one subsystem and left the others standing. That's not luck, it's design: limits, bulkheads, circuit breakers, and auto-rollback declared up front, so a failure is contained to its corner instead of taking the whole platform with it.

limits.yml
1# limits.yml — how the platform contains a blast
2per_tenant:
3 rate: 1000/min # one noisy tenant can't starve the rest
4 cpu_ms: 2000 # runaway work is killed, not tolerated
5timeouts: { upstream: "3s", retry: exponential }
6circuit_breaker: { trip_at: "50% errors / 30s", cool_off: "60s" }
7deploy: { canary: 10%, on_breach: rollback }
Bulkheaded, not shared-fate

One tenant or one service is walled off, so its failure can't drain the pool the others depend on.

Circuit breakers

A failing dependency is tripped and given time to recover, instead of every request piling onto it.

Timeouts and backoff

Nothing waits forever. A slow call is abandoned and a flaky one retried, so a stall doesn't spread.

Auto-rollback and kill switch

A bad deploy pulls itself at the canary, and one command can halt a runaway before it compounds.

The parts you only notice when they fail.

Nobody buys isolation or rate limiting for the demo. They matter on the one bad day: the customer who tried to read another's data, the client that hammered the API, the deploy that would have taken everyone down. This is the least glamorous work we do, and it's the first thing that decides whether a system holds up. We build it like the rest depends on it, because it does.

Tell us what has to stay up.

Send the system and the load it has to hold. We come back with a scope, a plan, and where a demo proves it.