Kubai Kevin

77 articles tagged Observability

Structured logging: searchable at scale

Why free-text logs get slow and expensive at scale, and how a stable schema, shared logger, and value-based retention fix both problems.

20 min read · 12 November 2026

Canary rollbacks: automate the decision, not the deploy

Design canary analysis that compares against a live baseline, guards on sample size, and avoids the false positives that erode trust in automation.

13 min read · 11 November 2026

Queueing theory for backend capacity

Why doubling traffic can quadruple tail latency, how to model each constrained resource as a queue, and how to measure utilisation before you hit the cliff.

14 min read · 8 November 2026

Redis vs Valkey vs Dragonfly: which in-memory store?

A mechanism-first guide to picking Redis, Valkey or Dragonfly: fork history, single-thread limits, expiry semantics, migration tests and metrics.

17 min read · 7 November 2026

Small language models: the classifier trap

Why compact classifiers fail on rare inputs, and how per-label thresholds, guardrails and score monitoring contain the damage.

15 min read · 31 October 2026

OpenTelemetry collector pipelines: sampling, batching…

How receivers, processors, and exporters interact under load, with worked memory arithmetic, sampling trade-offs, routing isolation, and metrics to watch.

13 min read · 30 October 2026

Multi-agent DX: LangGraph vs CrewAI gaps

Where LangGraph and CrewAI leave you on your own when debugging multi-agent systems, the failure modes that recur, and how to instrument for causal traces.

16 min read · 28 October 2026

Remote premium rates: skills AI can't copy

Four drills for remote engineers: latency budgets, evidence-based diagnosis, decision records and handoff artifacts that outlast the work.

19 min read · 21 October 2026

Agent reliability: 7 metrics users trust

Seven agent reliability metrics that reflect user trust, how to compute each from existing logs, and which to instrument first.

16 min read · 14 October 2026

Detecting silent agent failures

A layered quality gate that catches well-formed but wrong agent output: schema checks, PII scans, citation grounding, and a model judge, plus how to measure each.

15 min read · 13 October 2026

SLOs for agentic features: beyond latency

Define agent SLOs on task success, tool-call validity and clean termination, instrument them with OpenTelemetry, and alert on error-budget burn.

15 min read · 6 October 2026

Fraud detection: rules vs scoring, where they meet

How to split fraud decisions between a deterministic rules gate and a calibrated scoring model, with a worked three-layer architecture and audit trail.

17 min read · 4 October 2026

LLM evals: LangSmith vs Promptfoo in prod

How to choose between hosted trace-based eval platforms and local CLI eval runners, with latency, cost, CI and data-residency tradeoffs.

15 min read · 3 October 2026

Chaos engineering without an SRE team

How to run controlled fault injection in CI without a dedicated SRE team: scoping, kill switches, instrumentation, and failure modes to avoid.

14 min read · 30 September 2026

Tech debt: when half your stack is unfamiliar

A repeatable process for debugging, verifying and governing tools you didn't write, with baselines, contracts, environment measurement and tool cards.

14 min read · 26 September 2026

APM is blind to your AI agents

Traditional APM monitors transport, not semantics. See how to instrument retrieval quality, token cost, and answer correctness in AI agent pipelines.

12 min read · 25 September 2026

Multi-agent systems: when they quietly degrade

Multi-agent LLM systems drift into slow, costly behaviour without failing loudly. Here is what to instrument, what to budget, and how to catch the slide.

12 min read · 24 September 2026

Agentic AI cost control: 7 tools, one backfire

How to control agentic AI spend: ground-truth billing data first, attribution second, and why LLM optimizers over billing data backfire.

15 min read · 19 September 2026

Platform engineering ROI: metrics that fund teams

How to connect platform work to business outcomes with flow metrics, honest attribution, and instrumentation you can build in an afternoon.

13 min read · 19 September 2026

Most LLM cost dashboards lie…

Per-request token attribution, prompt caching, model routing and retry caps: how to instrument LLM spend so the numbers match the invoice.

13 min read · 19 September 2026

Multi-agent systems: why failures hide

Why multi-agent LLM pipelines fail silently, and how per-agent tracing, schema validation, budget enforcement, and loop detection make those failures loud.

13 min read · 18 September 2026

Platform Teams: Agentic World Struggle

Platform teams must adapt identity, tooling, and observability for AI agents that call APIs instead of using dashboards, with concrete patterns and failure modes.

13 min read · 12 September 2026

Why scaling agents kills plain logs

Why plain-text logs and unpinned models break down past a handful of agents, and how to add structured logging plus hash verification.

11 min read · 11 September 2026

Agent drift: why it hurts UX

Agent drift degrades latency and error rates without obvious alarms. Learn the four drift classes, detection methods, and a 30-minute audit.

12 min read · 9 September 2026

Ship evals before your agent ships

Agent evals fail when they only test the happy path. Build outcome-based evals from shadow traffic, wire them into CI, and catch regressions before users do.

16 min read · 31 August 2026

LLM risks underestimated in OWASP Top 10

The OWASP LLM Top 10 covers prompt injection, but production failures cluster in tool use, latency budgets and supply chain. How to measure and mitigate each.

9 min read · 30 August 2026

Prevent ownership drift when velocity explodes

How to keep clear ownership of caches, SLOs and services when teams ship faster: deployment boundaries, error budgets, stampede locks and useful telemetry.

15 min read · 29 August 2026

Agent tracing: when logs lie and metrics mislead

Async agents break synchronous tracing assumptions. Learn how to model queue spans, propagate context across brokers, and read traces honestly.

10 min read · 28 August 2026

Agentic FinOps is a trap

Standard FinOps tracks per-call cost and latency. Agents with retries, growing state and human hand-offs leak cost that never shows on the dashboard.

12 min read · 22 August 2026

AI errors hide in plain latency

Why token counts and GPU utilization miss user-visible AI degradation, and how to instrument end-to-end p95/p99 latency and error rate instead.

10 min read · 21 August 2026

AI unit costs: the metric both teams fight over

Why per-call AI cost estimates break down, and how to instrument a shared cost-per-outcome metric that product and finance both accept.

14 min read · 18 August 2026

Agent SLOs: measure what agents actually do

Latency and 5xx SLOs miss stuck loops. Define outcome SLOs on completion rate, with schema, PromQL, and failure handling.

13 min read · 11 August 2026

Observability Agents as Application Code

Why telemetry agents break at scale, how to budget their resources, and when the standard install-and-forget playbook is actually fine.

13 min read · 11 August 2026

eBPF observability in 2026: what actually works

A practical guide to production eBPF observability and runtime security: kernel requirements, verifier failures, ring buffers, pinning, and how to measure overhead.

14 min read · 9 August 2026

Why your multi-agent system degraded slowly

Multi-agent systems can lose p99 responsiveness while every agent stays healthy. Here is how payload creep, queueing and retries compound, and how to measure it.

14 min read · 9 August 2026

AI cost attribution without the spreadsheet hell

How to attribute AI inference costs to users, features and models: instrumentation, cardinality limits, sampling error, and a 30-minute first step.

15 min read · 7 August 2026

Prompt Injection Is Not Your Biggest LLM Data Risk

Prompt injection dominates LLM security checklists, but poisoned training data and label drift often cause more damage. A pipeline-stage model for triage.

17 min read · 3 August 2026

Why APM Misses GPU-Bound LLM Incidents

Application monitoring traces Python calls, not CUDA kernels or batch queues. Here is how to instrument the layers where LLM latency actually hides.

12 min read · 1 August 2026

When a vector DB misconfiguration breaks RAG silently

Vector DB misconfigurations cause RAG outages that look like app bugs. Learn to detect index errors, add circuit breakers, and instrument query latency.

11 min read · 29 July 2026

Track AI agent costs in fractal workflows

Standard FinOps dashboards miss sub-agent, idle, and retry costs. A dependency-graph cost model shows where agent spend actually lands.

12 min read · 27 July 2026

Purpose-built beats general AI tools

When a general AI platform beats a purpose-built stack, how to choose, and how to measure latency, cost and fallback behaviour yourself.

16 min read · 19 July 2026

Observability for Long-Running LLM Agent Workflows

Design an observability stack for agentic LLM workloads: millisecond event timestamps, queue-depth metrics, and bounded buffering that survives outages.

14 min read · 17 July 2026

Stabilize multi-model agent routers: six production rules

Six rules for multi-model agent routers: bounded concurrency, idempotency keys, timeouts, observability, and how to measure each one yourself.

15 min read · 15 July 2026

Stop guessing tech stacks: validate tools in 2 hours

A repeatable two-hour process for validating an unfamiliar library or service before it reaches production, with concrete checks and exit criteria.

13 min read · 12 July 2026

Designing Alert Triage That Suppresses Noise Safely

Classify alerts by shape and downstream impact, then suppress with evidence. Includes PromQL, SQL, breaker logic, and a 30-minute first step.

12 min read · 11 July 2026

Observability for LLM Features: Metrics That Drive Decisions

Log one structured event per LLM interaction instead of tracing every token, then alert only on user-facing outcomes that justify keeping a feature.

14 min read · 8 July 2026

7 ways to measure prompt platform ROI in 2026

How to attribute token cost, latency and developer time to prompt and agent platforms, with instrumentation recipes, failure modes and a decision checklist.

14 min read · 8 July 2026

AI in local currency: costs that surprise…

How to choose where to run AI inference: decouple variable compute cost from fixed per-region compliance cost, and measure the trade-off.

15 min read · 7 July 2026

Build agent workflows that survive restarts

How to build agent task pipelines that resume after crashes, schema drift and model changes, using Redis streams, idempotent steps and replay.

12 min read · 6 July 2026

LLM drift: the silent quality drop metrics miss

Latency and error rate stay flat while LLM output quality quietly degrades. Here is how to log prompts, measure embedding drift, and catch it early.

11 min read · 4 July 2026

High-Volume Log Pipelines: Shipping 50k Events Per Second

Compare a columnar-SQL log pipeline against a search-cluster pipeline at 50k EPS: architecture, failure modes, cost drivers, and how to measure both.

11 min read · 3 July 2026

AI scanners overlook real vulns

AI scanners catch pattern-level flaws but miss logic bugs. Learn a layered approach using deterministic rules, fuzzing, and runtime monitoring.

12 min read · 2 July 2026

5 ways to find hidden latency before production

Five practical ways to surface hidden latency before production: traffic replay, synthetic probes, profiling diffs, query plan diffs, and dependency latency budgets.

12 min read · 30 June 2026

Postmortems for AI agents: drift, cost, replay

How to run postmortems on AI agents: capture prompts and model versions, measure drift and cost per session, and choose between log-based and instrumented pipelines.

12 min read · 29 June 2026

AWS costs for Nairobi SaaS: 2026 bill split

How to split an AWS bill by service for a Nairobi-facing SaaS: cost drivers, measurement commands, and a decision checklist for region choice and DR.

13 min read · 28 June 2026

AI observability: the metrics that matter

AI observability needs drift, confidence, and attribution metrics. Covers instrumentation, cardinality limits, failure modes, and when not to build it.

13 min read · 28 June 2026

Debug AI agent failures differently

Structured logs catch infrastructure faults; behavior traces catch hallucinations and prompt drift. How to instrument both, measure detection time, and pick a stack.

13 min read · 27 June 2026

AI fixes legacy code nobody owns

How to combine static analysis, runtime traces, and LLM contextualization to make unowned legacy systems legible, with code and failure modes.

19 min read · 27 June 2026

LLM output metrics that actually move the needle

A layered approach to measuring LLM output quality in production: structured validation, p95 latency, cost per request, and retention signals.

10 min read · 26 June 2026

AI observability isn’t monitoring — here’s why

Traditional monitoring proves a service is up; AI observability proves its outputs are still correct. Layers, code, drift math, and failure modes.

14 min read · 25 June 2026

Pick one: Stripe, Lemon Squeezy, Paddle in 2026

A constraint-based framework for choosing between Stripe, Paddle and Lemon Squeezy, with latency tests, cost math and a 30-day pilot checklist.

14 min read · 25 June 2026

AI agent postmortems: the three surprises

How to run postmortems for AI agents that fail semantically, not just technically: structured semantic logging, live shadowing, and a decision checklist.

14 min read · 24 June 2026

Istio and Cilium: what really breaks in production

Service meshes amplify latent platform failures. Here is what breaks in Istio and Cilium, how to measure it, and when to defer adoption.

13 min read · 24 June 2026

API abuse patterns rising in 2026

A layered architecture for stopping credential stuffing and scraping: client integrity tokens, cohort-based rate limiting, and behavioral scoring at the CDN edge.

12 min read · 24 June 2026

Map legacy code with AI before it breaks

A practical workflow for using AI, static analysis, and OpenTelemetry to map legacy systems, find silent failures, and validate fixes.

13 min read · 23 June 2026

AI-First vs Human-First Incident Response

Compare AI-first autonomous incident response with human-first ops plus AI triage: architecture, failure modes, cost modeling, and a decision checklist.

12 min read · 23 June 2026

Unlock Python 3.13's free-threaded GIL for web apps

Free-threaded Python removes the GIL, but C extensions, shared state and pools break. A practical checklist for migrating a web service safely.

14 min read · 22 June 2026

LLM evals: skip the hype, track these 4 metrics

A practical guide to production LLM evaluation: four metric classes, code for deterministic validators, and how to detect prompt drift and scorer decay.

11 min read · 22 June 2026

AI glue code: 3 ways to avoid duct-taping models

Low-code AI platforms hide prompt drift, adapter decay and cold starts. Detect each with logged queries, a golden set and pre-warming, wired into CI.

14 min read · 20 June 2026

2026 alert triage: when Opsgenie slept for you

Build a scoring, deduplication and LLM-summary layer over PagerDuty or Opsgenie, plus how to measure whether it actually reduces wake-ups.

16 min read · 20 June 2026

Why monoliths die slowly in 2026

Monolith decomposition usually fails on data locality, not routing. A step-by-step order for extracting data first, routing traffic later, with checks.

13 min read · 19 June 2026

Compare Kafka, Redpanda, NATS for event-driven systems

Compare Kafka, Redpanda and NATS on durability, tail latency and cost, with config defaults, failure modes and a bake-off checklist you can run yourself.

14 min read · 14 June 2026

AI agent postmortems: trace vs logs vs code

Traces, evaluation logs, and code debugging reveal different AI agent failures. Compare overhead, cost, and coverage, then pick the right tool.

13 min read · 14 June 2026

Ship 1MB pages on 4G: Starlink Africa case study

How to cut page weight with Brotli, fallback encoding, CDN caching and QUIC, plus how to measure transfer, TTFB and Core Web Vitals honestly.

13 min read · 12 June 2026

Cutting cloud egress costs: architecture that works

Egress costs come from small, frequent cross-boundary calls. Learn the architecture patterns that reduce data transfer and how to verify savings.

13 min read · 11 June 2026

Build a search system in 2026: semantic, keyword

How to build a hybrid search service with PostgreSQL, pgvector, and Redis: schema, keyword and semantic queries, reranking, caching, and how to measure recall.

13 min read · 10 June 2026

Observability 2026: ditch the three pillars

Why the logs-metrics-traces split breaks under load, and how to design a single event store with schemas, queries and migration steps.

14 min read · 10 June 2026