9 articles tagged Data Engineering
Structured logging: searchable at scale
Why free-text logs get slow and expensive at scale, and how a stable schema, shared logger, and value-based retention fix both problems.
20 min read · 12 November 2026
DuckDB in production: when it replaces a warehouse
When an embedded columnar database beats a cloud warehouse for analytics, how to size it, and the failure modes that decide the boundary.
14 min read · 30 September 2026
CDC pipeline death spiral: one bad LLM call can crash…
How unconstrained LLM bursts stall CDC pipelines, and how to add backpressure at each layer with metrics, configs, and a worked failure example.
12 min read · 2 August 2026
High-Volume Log Pipelines: Shipping 50k Events Per Second
Compare a columnar-SQL log pipeline against a search-cluster pipeline at 50k EPS: architecture, failure modes, cost drivers, and how to measure both.
11 min read · 3 July 2026
Log audit costs without breaking compliance
Compare PostgreSQL and ClickHouse for audit logging: write amplification, query latency, retention, cost modelling, and a decision checklist.
13 min read · 22 June 2026
7 patterns for systems that won’t die when networks
Concrete patterns for building services that tolerate network partitions: retries, idempotency, outbox, sagas, and how to measure each trade-off.
18 min read · 14 June 2026
Observability 2026: ditch the three pillars
Why the logs-metrics-traces split breaks under load, and how to design a single event store with schemas, queries and migration steps.
14 min read · 10 June 2026
Event sourcing: pay the complexity tax only here
When an append-only event log earns its cost, when a temporal table is enough, and how to measure replay and projection lag before committing.
10 min read · 9 June 2026
RAG in prod: what tutorials won’t tell you
Why nightly embedding refreshes break RAG latency, and how a hot/cold retrieval split with partial vector updates fixes it.
14 min read · 8 June 2026