Kubai Kevin

9 articles tagged Data Engineering

Structured logging: searchable at scale

Why free-text logs get slow and expensive at scale, and how a stable schema, shared logger, and value-based retention fix both problems.

20 min read · 12 November 2026

DuckDB in production: when it replaces a warehouse

When an embedded columnar database beats a cloud warehouse for analytics, how to size it, and the failure modes that decide the boundary.

14 min read · 30 September 2026

CDC pipeline death spiral: one bad LLM call can crash…

How unconstrained LLM bursts stall CDC pipelines, and how to add backpressure at each layer with metrics, configs, and a worked failure example.

12 min read · 2 August 2026

High-Volume Log Pipelines: Shipping 50k Events Per Second

Compare a columnar-SQL log pipeline against a search-cluster pipeline at 50k EPS: architecture, failure modes, cost drivers, and how to measure both.

11 min read · 3 July 2026

Log audit costs without breaking compliance

Compare PostgreSQL and ClickHouse for audit logging: write amplification, query latency, retention, cost modelling, and a decision checklist.

13 min read · 22 June 2026

7 patterns for systems that won’t die when networks

Concrete patterns for building services that tolerate network partitions: retries, idempotency, outbox, sagas, and how to measure each trade-off.

18 min read · 14 June 2026

Observability 2026: ditch the three pillars

Why the logs-metrics-traces split breaks under load, and how to design a single event store with schemas, queries and migration steps.

14 min read · 10 June 2026

Event sourcing: pay the complexity tax only here

When an append-only event log earns its cost, when a temporal table is enough, and how to measure replay and projection lag before committing.

10 min read · 9 June 2026

RAG in prod: what tutorials won’t tell you

Why nightly embedding refreshes break RAG latency, and how a hot/cold retrieval split with partial vector updates fixes it.

14 min read · 8 June 2026