97 articles tagged Distributed Systems
Human-in-the-loop: when agents need a human veto
Design agent approval gates that live in the executor, not the prompt: signed single-use tokens, per-action scope, audit trails and anti-rubber-stamp checks.
14 min read · 16 November 2026
Multi-region active-active: the hidden costs
Active-active across AWS, Azure and Google Cloud is a distributed-systems build, not a product. Here are its real costs and failure modes.
17 min read · 15 November 2026
Python deps: uv vs Poetry vs pip-tools
Compare uv, Poetry and pip-tools by resolver, lockfile and environment model, with a worked example, failure modes and a decision checklist.
12 min read · 15 November 2026
Distributed locks: safe only with fencing
A lease can expire while its holder is still running. Fencing tokens, enforced by the protected resource, are what make a distributed lock safe.
17 min read · 13 November 2026
Saga pattern: orchestration vs choreography in payments
How payment sagas coordinate steps, why refunds are not charge reversals, and a decision framework for choosing orchestration or choreography.
16 min read · 10 November 2026
WAF and bot management: what they miss
WAFs and bot managers filter known-bad requests, not business logic. Map blind spots, tune rules safely, and add application-layer controls.
16 min read · 9 November 2026
Redis vs Valkey vs Dragonfly: which in-memory store?
A mechanism-first guide to picking Redis, Valkey or Dragonfly: fork history, single-thread limits, expiry semantics, migration tests and metrics.
17 min read · 7 November 2026
USSD/SMS fallback: the session trap
Why USSD sessions and SMS receipts mislead fallback designs, and how to make the ledger the source of truth with idempotent state transitions.
16 min read · 6 November 2026
Audit trails for AI agents: what breaks first
Design append-only decision records for AI agents: write-ahead ordering, policy snapshots, idempotency keys, and reconstruction drills that expose gaps.
15 min read · 5 November 2026
Dependency pinning: the lockfile lie
A lockfile records a resolution, not a guarantee. Learn where determinism breaks, how to enforce lockfiles in CI, and how pinning interacts with SCA.
16 min read · 31 October 2026
AI answers first, judgment later
AI assistants give juniors fast, confident answers without your system's context. Here is how to move mentoring from giving answers to reviewing judgment.
14 min read · 29 October 2026
Drift detection: the plan that lies
Terraform plan only reports drift a provider reads back. Build observe-classify-remediate pipelines that catch out-of-band changes safely.
16 min read · 26 October 2026
Agentic systems: architecture principles that hold
How to enforce deadlines, token budgets, idempotency keys and bounded fan-out around multi-model agent graphs, with Python code and tests.
15 min read · 23 October 2026
Agent code review: 7 traps that slip through
Agent-written code passes its own tests and reads as deliberate. Here are seven recurring traps, the mechanical checks that catch them, and how to measure.
17 min read · 23 October 2026
Remote premium rates: skills AI can't copy
Four drills for remote engineers: latency budgets, evidence-based diagnosis, decision records and handoff artifacts that outlast the work.
19 min read · 21 October 2026
Property-based testing vs example tests: the bug gap
How example-based and property-based tests differ in what they catch, what each costs to maintain, and how to split work between them in financial code.
18 min read · 20 October 2026
Open banking APIs: the integration traps
Open banking integration failures come from semantic gaps between markets. Here is how consent, idempotency and errors differ, and how to test.
15 min read · 17 October 2026
Why leader election fails on cloud networks
Cloud networks break the timing assumptions consensus algorithms rely on. See the failure modes, a corrected etcd lease example, and fencing tokens.
13 min read · 17 October 2026
Agent audit trails: event log vs state snapshots
Event sourcing and state snapshots solve different audit problems for AI agents. Compare causal reconstruction, debugging fidelity, cost, and replay limits.
13 min read · 12 October 2026
Rate limiters: burst vs abuse, two designs
Two rate limiter designs compared: token bucket with concurrency cap versus sliding window with penalty box, plus how to tune and measure each.
15 min read · 11 October 2026
Webhook delivery that survives outages
How to build durable webhook delivery with a transactional outbox, at-least-once queues, idempotency keys, and a retry schedule that outlasts outages.
13 min read · 2 October 2026
Double-entry ledger: concurrent write errors
Why concurrent writes corrupt double-entry ledgers, and how to enforce the debit-equals-credit invariant with isolation, locks, and triggers.
15 min read · 1 October 2026
Agentic workflows: surviving network partitions
How to keep agentic workflows correct when networks drop: idempotency keys, an outbox with stable action IDs, and reconciliation on reconnect.
18 min read · 1 October 2026
DuckDB in production: when it replaces a warehouse
When an embedded columnar database beats a cloud warehouse for analytics, how to size it, and the failure modes that decide the boundary.
14 min read · 30 September 2026
Settlement vs clearing: backend engineer's guide
Clearing produces obligations; settlement moves money. How to model both ledgers, reconcile them, and avoid the schema mistakes that break finance.
15 min read · 28 September 2026
AI features: M-Pesa, Paystack, Flutterwave traps
Payment webhooks from M-Pesa, Paystack and Flutterwave disagree on timing, idempotency and error semantics. Here is how to normalize them before an AI feature reads them.
12 min read · 23 September 2026
Tool calls: why retries stampede
Why per-client backoff fails under fan-out tool use, and how shared token buckets, single-flight and one-layer retries fix it.
12 min read · 22 September 2026
CQRS: when it pays off and when it breaks
CQRS trades strong consistency for independent read and write scaling. Learn the failure modes, how to measure projection lag, and when a read replica is enough.
12 min read · 21 September 2026
Rust rewrites: where they pay off, where they don't
A practical framework for deciding whether a Rust backend rewrite is worth it, with code, failure modes, and measurement steps.
12 min read · 21 September 2026
Idempotency keys: what they fix, what they don't
How idempotency keys work, where they fail, and the failure modes they don't cover: concurrent retries, key reuse, expiry, and partial failures.
14 min read · 21 September 2026
Agent access for non-AI engineers
A backend-first guide to exposing internal tools to LLM agents: typed schemas, dry-run defaults, idempotency, budgets, and replayable traces.
16 min read · 17 September 2026
Which agent stack survives 3G?
How edge-deployed and centralised agent stacks behave on high-latency, lossy mobile links, and how to measure reconnection storms before choosing one.
13 min read · 16 September 2026
Agent platform abstractions: a case study
Why agent failures are platform failures: typed tool contracts, durable execution, and structured traces that make runs replayable and retries safe.
11 min read · 16 September 2026
Memory Systems for Production Agents
How in-memory stores, caches, and stateful services leak context in production agents, plus code, failure modes, and instrumentation to measure it yourself.
11 min read · 15 September 2026
Stop trusting agents for core logic
Agent orchestration hides coupling in YAML and makes review harder. Learn contract-first design, policy-as-code, and how to measure the trade-off.
11 min read · 14 September 2026
Designing a Schema That Survives Embedding Model Upgrades
Why embedding columns break on model upgrades, how to version vectors, migrate legacy JSON payloads, and when to offload to a vector store.
11 min read · 13 September 2026
Agent Systems: Circuit Breakers Are Not Enough
Circuit breakers and bulkheads were built for stateless services. Agent systems need semantic error handling, externalized state, and recursion guards.
13 min read · 12 September 2026
Thundering Herd: The Hidden Scaling Trap
Why synchronized cache misses and naive retries collapse services, and how coalescing, jittered backoff, and locks prevent self-inflicted overload.
12 min read · 10 September 2026
Hybrid Cloud Routing: Cost-Speed Balance for SEA
How SEA teams route latency-sensitive traffic locally and bursty workloads to the cloud, with Nginx fallback config, failure modes, and cost checks.
13 min read · 4 September 2026
Portfolio projects that hire remote senior devs
Build a portfolio that proves operational rigor: caching, rate limiting, observability, and load tests that survive a senior remote technical screen.
14 min read · 3 September 2026
IDP for AI: 2026 stack evolution
How to wire embedding model changes into an internal developer platform so dimension drift fails loudly instead of silently shipping broken indexes.
14 min read · 2 September 2026
Ship evals before your agent ships
Agent evals fail when they only test the happy path. Build outcome-based evals from shadow traffic, wire them into CI, and catch regressions before users do.
16 min read · 31 August 2026
Always-on agents cost more than you think
Always-on agents bill you while idle; on-demand agents bill you at wake-up. A cost model, failure modes, and how to measure both.
15 min read · 30 August 2026
Agent tracing: when logs lie and metrics mislead
Async agents break synchronous tracing assumptions. Learn how to model queue spans, propagate context across brokers, and read traces honestly.
10 min read · 28 August 2026
AI on slow networks: Nairobi teams' 2026 playbook
How to cut perceived AI latency with multi-tier caching, stampede guards, and edge orchestration, plus how to measure each layer yourself.
13 min read · 26 August 2026
AI payments break in M-Pesa first
Payment webhooks arrive duplicated and out of order. Here is how to normalize provider events, dedupe them, and reconcile an AI feature store safely.
14 min read · 23 August 2026
Evolve your monolith without rewriting in 2026
A three-layer FastAPI layout (API, domain, infra) that keeps a monolith evolvable: worked code, failure modes, cache stampede fixes, and how to measure.
15 min read · 19 August 2026
Orchestration tax kills multi-agent systems
Why coordination layers dominate multi-agent latency and memory, with patterns that remove the scheduler and how to measure your own orchestration tax.
13 min read · 11 August 2026
Observability Agents as Application Code
Why telemetry agents break at scale, how to budget their resources, and when the standard install-and-forget playbook is actually fine.
13 min read · 11 August 2026
Agent frameworks: stop rewriting the same glue code
Agent glue code — retries, credentials, state — is mostly boilerplate. Here are three abstractions, worked examples, and how to measure the savings.
13 min read · 10 August 2026
Offline-first money apps with DynamoDB Streams
Design an offline-first mobile-money sync pipeline on DynamoDB Streams: idempotency keys, conditional balance writes, DLQ handling, and how to measure it.
15 min read · 8 August 2026
Stopping a runaway polling agent from flooding your API
Diagnose and fix runaway agent API calls: bounded retries, capped backoff, idempotency keys, and volume alerts that catch spikes before the bill does.
15 min read · 5 August 2026
Supervisor vs swarm vs debate vs pipeline
Compare four multi-agent orchestration patterns: supervisor, swarm, debate, and pipeline. Covers failure modes, backpressure, retries, and how to measure each.
19 min read · 3 August 2026
Boring AWS Lambda layers that don’t lock…
How to version, publish and reference Lambda layers so runtime upgrades, region moves and architecture changes don't break dependent functions.
12 min read · 31 July 2026
Agent permissions: the mistake that breaks production
How to build desktop agents that survive no-admin Windows environments: COM automation, per-user installs, heartbeats, and failure design, with code.
17 min read · 30 July 2026
Event-driven systems avoid agent-driven contention
How to tell when scheduled jobs and polling loops quietly turn an event-driven system into an agent-driven one, and how to fix it without a rewrite.
15 min read · 30 July 2026
Context windows live longer than you think
Long-running agents exhaust context windows through unbounded history growth. Covers pruning, summarisation, retrieval, and retention policies with code.
13 min read · 25 July 2026
Cache stampedes: the patterns that scale and the ones…
How cache stampedes happen, why lazy TTLs fail under load, and how to implement coordinated refresh with locks, stale fallbacks, and jitter.
16 min read · 21 July 2026
AI features as tech debt or golden paths
Why wrapper-plus-cache AI features fail under network drops and model drift, and how to design state-machine fallbacks you can actually measure.
12 min read · 17 July 2026
Designing AI-Assisted Sales for Developer Tools
How to build an AI-assisted sales pipeline for developer tools: intent routing, code generation, handoff rules, failure modes, and what to instrument.
16 min read · 14 July 2026
Agent context: short-term vs long-term memory
Compare Redis-style short-term agent context with Postgres event logs: latency, durability, auditability, cost, and how to measure each.
15 min read · 9 July 2026
AI in local currency: costs that surprise…
How to choose where to run AI inference: decouple variable compute cost from fixed per-region compliance cost, and measure the trade-off.
15 min read · 7 July 2026
Audit logs: ship them fast, keep them forever
Design an audit log pipeline that separates durability from searchability, with tiered retention, integrity hashing, and a cost model you can verify.
14 min read · 6 July 2026
7 skills senior engineers need now agents are live
Long-running agents break assumptions built for short-lived processes. Seven skills, how to measure each one, and failure modes to watch for.
15 min read · 5 July 2026
Always-On vs On-Demand Agents: Real Cost Trade-offs
Always-on containers bill for idle time; on-demand functions bill per invocation. Compare latency, retries, cold starts, and the failure modes that decide which fits.
12 min read · 5 July 2026
Evaluate agents with Spark vs custom harness
A practical comparison of Spark-based and custom Python agent evaluation harnesses, with failure modes, cost arithmetic, and a decision checklist.
12 min read · 3 July 2026
High-Volume Log Pipelines: Shipping 50k Events Per Second
Compare a columnar-SQL log pipeline against a search-cluster pipeline at 50k EPS: architecture, failure modes, cost drivers, and how to measure both.
11 min read · 3 July 2026
Serve 4G-as-baseline apps in 2026
Design APIs for 300 ms RTT, packet loss and data caps: payload budgets, stale-while-revalidate caching, adaptive fetch and honest measurement.
14 min read · 3 July 2026
Stop the writes — zero-downtime DB migrations
Zero-downtime migrations fail when writes keep flowing during the copy. A drain-copy-flip state machine, with instrumentation and a decision checklist.
13 min read · 2 July 2026
Meter dev-tool pricing in 2026
Build usage-metered pricing for a dev tool: Stripe metered billing, Redis counters, idempotent usage events, and the edge cases that break first.
11 min read · 2 July 2026
5 ways to find hidden latency before production
Five practical ways to surface hidden latency before production: traffic replay, synthetic probes, profiling diffs, query plan diffs, and dependency latency budgets.
12 min read · 30 June 2026
Bank APIs broke under African 2G
Why aggressive caching and remote idempotency stores fail on flaky 2G/3G links, and how to instrument, measure, and redesign balance endpoints for it.
14 min read · 28 June 2026
Migrations? Stop running them live
Why live schema migrations cause outages, and how versioning the schema as a deployable artifact with expand/contract steps reduces that risk.
11 min read · 28 June 2026
Monolith to services without a rewrite
Most monolith-to-services migrations fail on data coupling, not code structure. Here is how to audit coupling and measure the latency tax before extracting.
14 min read · 27 June 2026
Tool calling patterns: patterns that survive load vs
A practical comparison of naive synchronous tool calls versus retry-budget, backoff, circuit-breaker and bulkhead patterns, with code and metrics.
14 min read · 27 June 2026
NDPR 2026 broke fintech APIs
How regulatory transport rules and intermittent mobile connectivity change fintech API design, with worked latency arithmetic and a concrete measurement plan.
12 min read · 26 June 2026
Webhooks done right: retries that don’t explode in 2026
A practical guide to webhook delivery: idempotency, retry budgets, silent rate limits, poison-message triage, and what to instrument before you scale.
10 min read · 25 June 2026
Store data once, serve everywhere: cheaper
How to serve reads from regional caches instead of replicating your database everywhere, with worked cost math, invalidation code, and failure modes.
12 min read · 25 June 2026
API abuse patterns rising in 2026
A layered architecture for stopping credential stuffing and scraping: client integrity tokens, cohort-based rate limiting, and behavioral scoring at the CDN edge.
12 min read · 24 June 2026
AI-native apps break when you copy old rules
Model calls are distributed workloads, not function calls. Here is how to design task queues, cache keys, token budgets and retry policy for AI features.
13 min read · 23 June 2026
Vibe code: works great, fails forever
Ten failure modes that make vibe-coded MVPs unmaintainable, with code fixes, measurement recipes, and a hardening checklist for each.
13 min read · 20 June 2026
Defending APIs Against Amplification, Stuffing and Smuggling
How cache-stampede amplification, credential stuffing and HTTP/2 request smuggling work, plus edge admission, idempotency keys and circuit breakers that blunt them.
13 min read · 20 June 2026
Run APIs in 50 spots: what breaks first
Edge APIs fail at the data layer, not the network. Covers sharding, bounded staleness, cache invalidation, clock drift, and how to measure each.
12 min read · 19 June 2026
Webhook Delivery: Retries, Idempotency, and Durability
Design reliable webhook ingestion with durable buffering, idempotency keys, bounded retries, and circuit breakers, plus how to measure each guarantee.
11 min read · 19 June 2026
Portfolio that hires: show constraints, not code
A portfolio that proves engineering judgement: latency budgets, load tests, failure-mode notes and honest trade-offs recruiters can verify.
14 min read · 16 June 2026
7 patterns for systems that won’t die when networks
Concrete patterns for building services that tolerate network partitions: retries, idempotency, outbox, sagas, and how to measure each trade-off.
18 min read · 14 June 2026
Skip M-Pesa: build one API for 3 African markets
How to serve three African mobile-money markets through one orchestrator plus thin provider adapters, with metrics and reconciliation that hold up.
15 min read · 14 June 2026
AI Interviews: Four Resilience Questions That Matter
AI screening interviews grade production resilience, not algorithm puzzles. Learn the four fault-injection questions and how to measure candidate fixes.
13 min read · 14 June 2026
AI freelance rates 2026: which stacks still pay
AI code generation shifts effort from writing components to compliance, observability and edge cases. Here is how to price and instrument that hidden work.
12 min read · 13 June 2026
Reconciling mobile money: edge cases banks ignore
Why provider webhooks are only approximate signals, and how to reconcile them against MNO ledgers with tolerance windows, dedup keys and batch jobs.
17 min read · 12 June 2026
AI interviews broke system design: 3 patterns that
AI design tools produce complete-looking architectures that fail under real load. Three patterns, why they happen, and how to catch them before staging.
15 min read · 11 June 2026
Designing USSD fintech flows that survive live networks
A practical guide to building reliable USSD fintech flows on AWS: state machines, 160-character screens, carrier SLAs, idempotency, and latency instrumentation.
17 min read · 11 June 2026
Cutting cloud egress costs: architecture that works
Egress costs come from small, frequent cross-boundary calls. Learn the architecture patterns that reduce data transfer and how to verify savings.
13 min read · 11 June 2026
Personal AI assistants: avoid vendor lock-in traps
How to structure a personal AI assistant as replaceable services: contracts, self-hosted embeddings, retries, failure modes, and a checklist to start.
15 min read · 11 June 2026
Neon vs PlanetScale vs Turso: which broke first?
Diagnose connection drops, replica lag and proxy limits in Neon, PlanetScale and Turso, with reproducible tests and fixes that do not rely on guesswork.
13 min read · 10 June 2026
AI interviews now ask for this
AI-assisted coding has changed what engineering interviews probe: flaky tests, observability overhead, secret rotation, and auditing generated code.
16 min read · 10 June 2026
Event sourcing: pay the complexity tax only here
When an append-only event log earns its cost, when a temporal table is enough, and how to measure replay and projection lag before committing.
10 min read · 9 June 2026