Kubai Kevin

97 articles tagged Distributed Systems

Human-in-the-loop: when agents need a human veto

Design agent approval gates that live in the executor, not the prompt: signed single-use tokens, per-action scope, audit trails and anti-rubber-stamp checks.

14 min read · 16 November 2026

Multi-region active-active: the hidden costs

Active-active across AWS, Azure and Google Cloud is a distributed-systems build, not a product. Here are its real costs and failure modes.

17 min read · 15 November 2026

Python deps: uv vs Poetry vs pip-tools

Compare uv, Poetry and pip-tools by resolver, lockfile and environment model, with a worked example, failure modes and a decision checklist.

12 min read · 15 November 2026

Distributed locks: safe only with fencing

A lease can expire while its holder is still running. Fencing tokens, enforced by the protected resource, are what make a distributed lock safe.

17 min read · 13 November 2026

Saga pattern: orchestration vs choreography in payments

How payment sagas coordinate steps, why refunds are not charge reversals, and a decision framework for choosing orchestration or choreography.

16 min read · 10 November 2026

WAF and bot management: what they miss

WAFs and bot managers filter known-bad requests, not business logic. Map blind spots, tune rules safely, and add application-layer controls.

16 min read · 9 November 2026

Redis vs Valkey vs Dragonfly: which in-memory store?

A mechanism-first guide to picking Redis, Valkey or Dragonfly: fork history, single-thread limits, expiry semantics, migration tests and metrics.

17 min read · 7 November 2026

USSD/SMS fallback: the session trap

Why USSD sessions and SMS receipts mislead fallback designs, and how to make the ledger the source of truth with idempotent state transitions.

16 min read · 6 November 2026

Audit trails for AI agents: what breaks first

Design append-only decision records for AI agents: write-ahead ordering, policy snapshots, idempotency keys, and reconstruction drills that expose gaps.

15 min read · 5 November 2026

Dependency pinning: the lockfile lie

A lockfile records a resolution, not a guarantee. Learn where determinism breaks, how to enforce lockfiles in CI, and how pinning interacts with SCA.

16 min read · 31 October 2026

AI answers first, judgment later

AI assistants give juniors fast, confident answers without your system's context. Here is how to move mentoring from giving answers to reviewing judgment.

14 min read · 29 October 2026

Drift detection: the plan that lies

Terraform plan only reports drift a provider reads back. Build observe-classify-remediate pipelines that catch out-of-band changes safely.

16 min read · 26 October 2026

Agentic systems: architecture principles that hold

How to enforce deadlines, token budgets, idempotency keys and bounded fan-out around multi-model agent graphs, with Python code and tests.

15 min read · 23 October 2026

Agent code review: 7 traps that slip through

Agent-written code passes its own tests and reads as deliberate. Here are seven recurring traps, the mechanical checks that catch them, and how to measure.

17 min read · 23 October 2026

Remote premium rates: skills AI can't copy

Four drills for remote engineers: latency budgets, evidence-based diagnosis, decision records and handoff artifacts that outlast the work.

19 min read · 21 October 2026

Property-based testing vs example tests: the bug gap

How example-based and property-based tests differ in what they catch, what each costs to maintain, and how to split work between them in financial code.

18 min read · 20 October 2026

Open banking APIs: the integration traps

Open banking integration failures come from semantic gaps between markets. Here is how consent, idempotency and errors differ, and how to test.

15 min read · 17 October 2026

Why leader election fails on cloud networks

Cloud networks break the timing assumptions consensus algorithms rely on. See the failure modes, a corrected etcd lease example, and fencing tokens.

13 min read · 17 October 2026

Agent audit trails: event log vs state snapshots

Event sourcing and state snapshots solve different audit problems for AI agents. Compare causal reconstruction, debugging fidelity, cost, and replay limits.

13 min read · 12 October 2026

Rate limiters: burst vs abuse, two designs

Two rate limiter designs compared: token bucket with concurrency cap versus sliding window with penalty box, plus how to tune and measure each.

15 min read · 11 October 2026

Webhook delivery that survives outages

How to build durable webhook delivery with a transactional outbox, at-least-once queues, idempotency keys, and a retry schedule that outlasts outages.

13 min read · 2 October 2026

Double-entry ledger: concurrent write errors

Why concurrent writes corrupt double-entry ledgers, and how to enforce the debit-equals-credit invariant with isolation, locks, and triggers.

15 min read · 1 October 2026

Agentic workflows: surviving network partitions

How to keep agentic workflows correct when networks drop: idempotency keys, an outbox with stable action IDs, and reconciliation on reconnect.

18 min read · 1 October 2026

DuckDB in production: when it replaces a warehouse

When an embedded columnar database beats a cloud warehouse for analytics, how to size it, and the failure modes that decide the boundary.

14 min read · 30 September 2026

Settlement vs clearing: backend engineer's guide

Clearing produces obligations; settlement moves money. How to model both ledgers, reconcile them, and avoid the schema mistakes that break finance.

15 min read · 28 September 2026

AI features: M-Pesa, Paystack, Flutterwave traps

Payment webhooks from M-Pesa, Paystack and Flutterwave disagree on timing, idempotency and error semantics. Here is how to normalize them before an AI feature reads them.

12 min read · 23 September 2026

Tool calls: why retries stampede

Why per-client backoff fails under fan-out tool use, and how shared token buckets, single-flight and one-layer retries fix it.

12 min read · 22 September 2026

CQRS: when it pays off and when it breaks

CQRS trades strong consistency for independent read and write scaling. Learn the failure modes, how to measure projection lag, and when a read replica is enough.

12 min read · 21 September 2026

Rust rewrites: where they pay off, where they don't

A practical framework for deciding whether a Rust backend rewrite is worth it, with code, failure modes, and measurement steps.

12 min read · 21 September 2026

Idempotency keys: what they fix, what they don't

How idempotency keys work, where they fail, and the failure modes they don't cover: concurrent retries, key reuse, expiry, and partial failures.

14 min read · 21 September 2026

Agent access for non-AI engineers

A backend-first guide to exposing internal tools to LLM agents: typed schemas, dry-run defaults, idempotency, budgets, and replayable traces.

16 min read · 17 September 2026

Which agent stack survives 3G?

How edge-deployed and centralised agent stacks behave on high-latency, lossy mobile links, and how to measure reconnection storms before choosing one.

13 min read · 16 September 2026

Agent platform abstractions: a case study

Why agent failures are platform failures: typed tool contracts, durable execution, and structured traces that make runs replayable and retries safe.

11 min read · 16 September 2026

Memory Systems for Production Agents

How in-memory stores, caches, and stateful services leak context in production agents, plus code, failure modes, and instrumentation to measure it yourself.

11 min read · 15 September 2026

Stop trusting agents for core logic

Agent orchestration hides coupling in YAML and makes review harder. Learn contract-first design, policy-as-code, and how to measure the trade-off.

11 min read · 14 September 2026

Designing a Schema That Survives Embedding Model Upgrades

Why embedding columns break on model upgrades, how to version vectors, migrate legacy JSON payloads, and when to offload to a vector store.

11 min read · 13 September 2026

Agent Systems: Circuit Breakers Are Not Enough

Circuit breakers and bulkheads were built for stateless services. Agent systems need semantic error handling, externalized state, and recursion guards.

13 min read · 12 September 2026

Thundering Herd: The Hidden Scaling Trap

Why synchronized cache misses and naive retries collapse services, and how coalescing, jittered backoff, and locks prevent self-inflicted overload.

12 min read · 10 September 2026

Hybrid Cloud Routing: Cost-Speed Balance for SEA

How SEA teams route latency-sensitive traffic locally and bursty workloads to the cloud, with Nginx fallback config, failure modes, and cost checks.

13 min read · 4 September 2026

Portfolio projects that hire remote senior devs

Build a portfolio that proves operational rigor: caching, rate limiting, observability, and load tests that survive a senior remote technical screen.

14 min read · 3 September 2026

IDP for AI: 2026 stack evolution

How to wire embedding model changes into an internal developer platform so dimension drift fails loudly instead of silently shipping broken indexes.

14 min read · 2 September 2026

Ship evals before your agent ships

Agent evals fail when they only test the happy path. Build outcome-based evals from shadow traffic, wire them into CI, and catch regressions before users do.

16 min read · 31 August 2026

Always-on agents cost more than you think

Always-on agents bill you while idle; on-demand agents bill you at wake-up. A cost model, failure modes, and how to measure both.

15 min read · 30 August 2026

Agent tracing: when logs lie and metrics mislead

Async agents break synchronous tracing assumptions. Learn how to model queue spans, propagate context across brokers, and read traces honestly.

10 min read · 28 August 2026

AI on slow networks: Nairobi teams' 2026 playbook

How to cut perceived AI latency with multi-tier caching, stampede guards, and edge orchestration, plus how to measure each layer yourself.

13 min read · 26 August 2026

AI payments break in M-Pesa first

Payment webhooks arrive duplicated and out of order. Here is how to normalize provider events, dedupe them, and reconcile an AI feature store safely.

14 min read · 23 August 2026

Evolve your monolith without rewriting in 2026

A three-layer FastAPI layout (API, domain, infra) that keeps a monolith evolvable: worked code, failure modes, cache stampede fixes, and how to measure.

15 min read · 19 August 2026

Orchestration tax kills multi-agent systems

Why coordination layers dominate multi-agent latency and memory, with patterns that remove the scheduler and how to measure your own orchestration tax.

13 min read · 11 August 2026

Observability Agents as Application Code

Why telemetry agents break at scale, how to budget their resources, and when the standard install-and-forget playbook is actually fine.

13 min read · 11 August 2026

Agent frameworks: stop rewriting the same glue code

Agent glue code — retries, credentials, state — is mostly boilerplate. Here are three abstractions, worked examples, and how to measure the savings.

13 min read · 10 August 2026

Offline-first money apps with DynamoDB Streams

Design an offline-first mobile-money sync pipeline on DynamoDB Streams: idempotency keys, conditional balance writes, DLQ handling, and how to measure it.

15 min read · 8 August 2026

Stopping a runaway polling agent from flooding your API

Diagnose and fix runaway agent API calls: bounded retries, capped backoff, idempotency keys, and volume alerts that catch spikes before the bill does.

15 min read · 5 August 2026

Supervisor vs swarm vs debate vs pipeline

Compare four multi-agent orchestration patterns: supervisor, swarm, debate, and pipeline. Covers failure modes, backpressure, retries, and how to measure each.

19 min read · 3 August 2026

Boring AWS Lambda layers that don’t lock…

How to version, publish and reference Lambda layers so runtime upgrades, region moves and architecture changes don't break dependent functions.

12 min read · 31 July 2026

Agent permissions: the mistake that breaks production

How to build desktop agents that survive no-admin Windows environments: COM automation, per-user installs, heartbeats, and failure design, with code.

17 min read · 30 July 2026

Event-driven systems avoid agent-driven contention

How to tell when scheduled jobs and polling loops quietly turn an event-driven system into an agent-driven one, and how to fix it without a rewrite.

15 min read · 30 July 2026

Context windows live longer than you think

Long-running agents exhaust context windows through unbounded history growth. Covers pruning, summarisation, retrieval, and retention policies with code.

13 min read · 25 July 2026

Cache stampedes: the patterns that scale and the ones…

How cache stampedes happen, why lazy TTLs fail under load, and how to implement coordinated refresh with locks, stale fallbacks, and jitter.

16 min read · 21 July 2026

AI features as tech debt or golden paths

Why wrapper-plus-cache AI features fail under network drops and model drift, and how to design state-machine fallbacks you can actually measure.

12 min read · 17 July 2026

Designing AI-Assisted Sales for Developer Tools

How to build an AI-assisted sales pipeline for developer tools: intent routing, code generation, handoff rules, failure modes, and what to instrument.

16 min read · 14 July 2026

Agent context: short-term vs long-term memory

Compare Redis-style short-term agent context with Postgres event logs: latency, durability, auditability, cost, and how to measure each.

15 min read · 9 July 2026

AI in local currency: costs that surprise…

How to choose where to run AI inference: decouple variable compute cost from fixed per-region compliance cost, and measure the trade-off.

15 min read · 7 July 2026

Audit logs: ship them fast, keep them forever

Design an audit log pipeline that separates durability from searchability, with tiered retention, integrity hashing, and a cost model you can verify.

14 min read · 6 July 2026

7 skills senior engineers need now agents are live

Long-running agents break assumptions built for short-lived processes. Seven skills, how to measure each one, and failure modes to watch for.

15 min read · 5 July 2026

Always-On vs On-Demand Agents: Real Cost Trade-offs

Always-on containers bill for idle time; on-demand functions bill per invocation. Compare latency, retries, cold starts, and the failure modes that decide which fits.

12 min read · 5 July 2026

Evaluate agents with Spark vs custom harness

A practical comparison of Spark-based and custom Python agent evaluation harnesses, with failure modes, cost arithmetic, and a decision checklist.

12 min read · 3 July 2026

High-Volume Log Pipelines: Shipping 50k Events Per Second

Compare a columnar-SQL log pipeline against a search-cluster pipeline at 50k EPS: architecture, failure modes, cost drivers, and how to measure both.

11 min read · 3 July 2026

Serve 4G-as-baseline apps in 2026

Design APIs for 300 ms RTT, packet loss and data caps: payload budgets, stale-while-revalidate caching, adaptive fetch and honest measurement.

14 min read · 3 July 2026

Stop the writes — zero-downtime DB migrations

Zero-downtime migrations fail when writes keep flowing during the copy. A drain-copy-flip state machine, with instrumentation and a decision checklist.

13 min read · 2 July 2026

Meter dev-tool pricing in 2026

Build usage-metered pricing for a dev tool: Stripe metered billing, Redis counters, idempotent usage events, and the edge cases that break first.

11 min read · 2 July 2026

5 ways to find hidden latency before production

Five practical ways to surface hidden latency before production: traffic replay, synthetic probes, profiling diffs, query plan diffs, and dependency latency budgets.

12 min read · 30 June 2026

Bank APIs broke under African 2G

Why aggressive caching and remote idempotency stores fail on flaky 2G/3G links, and how to instrument, measure, and redesign balance endpoints for it.

14 min read · 28 June 2026

Migrations? Stop running them live

Why live schema migrations cause outages, and how versioning the schema as a deployable artifact with expand/contract steps reduces that risk.

11 min read · 28 June 2026

Monolith to services without a rewrite

Most monolith-to-services migrations fail on data coupling, not code structure. Here is how to audit coupling and measure the latency tax before extracting.

14 min read · 27 June 2026

Tool calling patterns: patterns that survive load vs

A practical comparison of naive synchronous tool calls versus retry-budget, backoff, circuit-breaker and bulkhead patterns, with code and metrics.

14 min read · 27 June 2026

NDPR 2026 broke fintech APIs

How regulatory transport rules and intermittent mobile connectivity change fintech API design, with worked latency arithmetic and a concrete measurement plan.

12 min read · 26 June 2026

Webhooks done right: retries that don’t explode in 2026

A practical guide to webhook delivery: idempotency, retry budgets, silent rate limits, poison-message triage, and what to instrument before you scale.

10 min read · 25 June 2026

Store data once, serve everywhere: cheaper

How to serve reads from regional caches instead of replicating your database everywhere, with worked cost math, invalidation code, and failure modes.

12 min read · 25 June 2026

API abuse patterns rising in 2026

A layered architecture for stopping credential stuffing and scraping: client integrity tokens, cohort-based rate limiting, and behavioral scoring at the CDN edge.

12 min read · 24 June 2026

AI-native apps break when you copy old rules

Model calls are distributed workloads, not function calls. Here is how to design task queues, cache keys, token budgets and retry policy for AI features.

13 min read · 23 June 2026

Vibe code: works great, fails forever

Ten failure modes that make vibe-coded MVPs unmaintainable, with code fixes, measurement recipes, and a hardening checklist for each.

13 min read · 20 June 2026

Defending APIs Against Amplification, Stuffing and Smuggling

How cache-stampede amplification, credential stuffing and HTTP/2 request smuggling work, plus edge admission, idempotency keys and circuit breakers that blunt them.

13 min read · 20 June 2026

Run APIs in 50 spots: what breaks first

Edge APIs fail at the data layer, not the network. Covers sharding, bounded staleness, cache invalidation, clock drift, and how to measure each.

12 min read · 19 June 2026

Webhook Delivery: Retries, Idempotency, and Durability

Design reliable webhook ingestion with durable buffering, idempotency keys, bounded retries, and circuit breakers, plus how to measure each guarantee.

11 min read · 19 June 2026

Portfolio that hires: show constraints, not code

A portfolio that proves engineering judgement: latency budgets, load tests, failure-mode notes and honest trade-offs recruiters can verify.

14 min read · 16 June 2026

7 patterns for systems that won’t die when networks

Concrete patterns for building services that tolerate network partitions: retries, idempotency, outbox, sagas, and how to measure each trade-off.

18 min read · 14 June 2026

Skip M-Pesa: build one API for 3 African markets

How to serve three African mobile-money markets through one orchestrator plus thin provider adapters, with metrics and reconciliation that hold up.

15 min read · 14 June 2026

AI Interviews: Four Resilience Questions That Matter

AI screening interviews grade production resilience, not algorithm puzzles. Learn the four fault-injection questions and how to measure candidate fixes.

13 min read · 14 June 2026

AI freelance rates 2026: which stacks still pay

AI code generation shifts effort from writing components to compliance, observability and edge cases. Here is how to price and instrument that hidden work.

12 min read · 13 June 2026

Reconciling mobile money: edge cases banks ignore

Why provider webhooks are only approximate signals, and how to reconcile them against MNO ledgers with tolerance windows, dedup keys and batch jobs.

17 min read · 12 June 2026

AI interviews broke system design: 3 patterns that

AI design tools produce complete-looking architectures that fail under real load. Three patterns, why they happen, and how to catch them before staging.

15 min read · 11 June 2026

Designing USSD fintech flows that survive live networks

A practical guide to building reliable USSD fintech flows on AWS: state machines, 160-character screens, carrier SLAs, idempotency, and latency instrumentation.

17 min read · 11 June 2026

Cutting cloud egress costs: architecture that works

Egress costs come from small, frequent cross-boundary calls. Learn the architecture patterns that reduce data transfer and how to verify savings.

13 min read · 11 June 2026

Personal AI assistants: avoid vendor lock-in traps

How to structure a personal AI assistant as replaceable services: contracts, self-hosted embeddings, retries, failure modes, and a checklist to start.

15 min read · 11 June 2026

Neon vs PlanetScale vs Turso: which broke first?

Diagnose connection drops, replica lag and proxy limits in Neon, PlanetScale and Turso, with reproducible tests and fixes that do not rely on guesswork.

13 min read · 10 June 2026

AI interviews now ask for this

AI-assisted coding has changed what engineering interviews probe: flaky tests, observability overhead, secret rotation, and auditing generated code.

16 min read · 10 June 2026

Event sourcing: pay the complexity tax only here

When an append-only event log earns its cost, when a temporal table is enough, and how to measure replay and projection lag before committing.

10 min read · 9 June 2026