Kubai Kevin

50 articles tagged Ci/Cd

Canary rollbacks: automate the decision, not the deploy

Design canary analysis that compares against a live baseline, guards on sample size, and avoids the false positives that erode trust in automation.

13 min read · 11 November 2026

Technical debt: defend priorities to stakeholders

A framework that turns technical debt into cost, risk, and timing arguments stakeholders can act on, with scoring, cost-of-delay arithmetic, and capacity rules.

13 min read · 4 November 2026

Dependency pinning: the lockfile lie

A lockfile records a resolution, not a guarantee. Learn where determinism breaks, how to enforce lockfiles in CI, and how pinning interacts with SCA.

16 min read · 31 October 2026

Node.js memory leak: find it in 10 min

Classify rising memory before profiling it: compare heapUsed and rss, take two heap snapshots, read retainer paths, and bound every in-memory structure.

14 min read · 29 October 2026

Docs that survive: the bus factor fix

How to write runbooks and design docs that a new engineer can follow without asking the author: explicit audience, exact commands, expected output, and rollback.

15 min read · 21 October 2026

Staging won't catch prod bugs. Make…

Why full parity between staging and production misses state-dependent bugs, and how to build cheap fault injection that catches them before deploy.

13 min read · 19 October 2026

PCI DSS architecture: 12 requirements that matter

Map PCI DSS requirements to concrete architecture decisions: CDE boundary, envelope encryption and key separation, immutable logging, CI/CD controls, and scope reduction.

14 min read · 19 October 2026

Prompts are code: how to measure their value

Prompts and agent graphs need outcome metrics, not token counts. A practical guide to boundary instrumentation, versioning, and avoiding vanity metrics.

19 min read · 16 October 2026

Flaky tests: reliability bug, not test noise

Flaky tests usually point to real non-determinism in production code. Learn to classify the four families, triage by blast radius, and fix the source.

16 min read · 14 October 2026

7 IDP shifts for AI feature delivery

Seven platform shifts that let teams version prompts, pin models, run offline evals, attribute token cost, and roll out AI changes gradually.

15 min read · 10 October 2026

Deployment platforms: senior vs new hire

How to build a deployment platform that serves both senior engineers and new hires, with escape hatches, guardrails, and a decision checklist.

17 min read · 7 October 2026

Who signs off on AI code?

AI coding assistants produce diffs fast, but accountability still needs a named human. Here is a review workflow with hooks, policy checks, and failure modes.

12 min read · 2 October 2026

Double-entry ledger: concurrent write errors

Why concurrent writes corrupt double-entry ledgers, and how to enforce the debit-equals-credit invariant with isolation, locks, and triggers.

15 min read · 1 October 2026

World models won't replace your backend

World models add a high-rate prediction stream to your system. Treat it as a sensor: schema, retention, backpressure, and rollback, not a new stack.

15 min read · 24 September 2026

Feature flags became AI rollout platforms

How flag platforms handle structured model configs, eval-metric gating, and offline fallback — plus a checklist for choosing one for AI rollouts.

18 min read · 18 September 2026

Agent API versioning: why clients break on upgrade

Why agent version upgrades break downstream clients, the contract drift behind the errors, and how to pin, check and roll out versions safely.

12 min read · 14 September 2026

Why ML Models Drift After Deployment

Offline validation rarely matches production traffic. Learn the evaluation gap, how to measure drift, and concrete checks to add before your next deploy.

16 min read · 11 September 2026

Validating AI-Generated Code Before It Reaches Production

AI-written code often fails quietly under load. Here is how to validate it at build time, guard it at runtime, and keep on-call triage fast.

10 min read · 20 August 2026

AI-era promotions hinge on this one thing

AI features fail through cache eviction, schema drift and missing rollback paths. A practical guide to validation gates, fallback caches and metrics.

11 min read · 15 August 2026

AWS CloudWatch Synthetics for AI pipelines

Build CloudWatch Synthetics canaries that exercise a full AI pipeline journey, not just an endpoint ping, and alarm on whole-flow failure.

13 min read · 10 August 2026

Red-teaming agents without killing velocity

How to embed lightweight adversarial testing for LLM agents into pull requests: fuzzed payloads, resilience assertions, shadow canaries.

11 min read · 17 July 2026

AI model poisoning: 3 real attacks on datasets

How data, model, and dependency poisoning enter AI pipelines, plus concrete controls for provenance, review, sandboxing, and rollback.

15 min read · 4 July 2026

AI made self-service platforms trivial in 2026

AI can draft Terraform and Kubernetes manifests, but correctness still needs policy checks, schema validation, and runtime verification before merge.

10 min read · 3 July 2026

AI Code Generation: Failure Modes and Guardrails

AI code assistants change interfaces, concurrency and state in ways that pass review. Here are the failure modes and the checks that catch them.

11 min read · 3 July 2026

CI/CD for Django in 20ms Africa networks

Design a Django CI/CD pipeline for 1-2 vCPU runners, slow links and metered egress: caching, image size, migrations and health checks.

13 min read · 3 July 2026

When flags become experiments

How to run feature flags and A/B tests on a $5 VPS with no data warehouse, using offline-tolerant self-hosted tooling and SMS replies as conversions.

15 min read · 28 June 2026

AWS Local Zones vs LocalStack: AI microservices in

Compare AWS Local Zones and LocalStack for AI microservices: latency, cost, compliance, and a decision framework, plus how to measure p99 at the edge.

14 min read · 28 June 2026

CI/CD pipelines stall with GPU pods: 2026 benchmarks

GPU pods stuck Pending are usually a scheduling or quota problem, not a broken container. Learn how to diagnose node selectors, device plugins and quotas.

13 min read · 28 June 2026

Spot AI’s hidden cost spike in your stack

AI refactors often trade cache hits for database reads, exhausting pools and inflating bills. Here is how to detect, verify and gate the change.

14 min read · 28 June 2026

Migrations? Stop running them live

Why live schema migrations cause outages, and how versioning the schema as a deployable artifact with expand/contract steps reduces that risk.

11 min read · 28 June 2026

AI agents vs Argo, Tekton: which pipeline heals itself?

How Argo Workflows and Tekton Pipelines differ for automated failure detection and recovery, with working code and failure-mode analysis.

13 min read · 27 June 2026

Platform teams in 2026: Docs vs. production reality

Platform docs describe happy paths; production exposes the gaps. Covers cluster defaults, IDP design by company size, Helm guardrails, and policy enforcement.

18 min read · 27 June 2026

LLM output metrics that actually move the needle

A layered approach to measuring LLM output quality in production: structured validation, p95 latency, cost per request, and retention signals.

10 min read · 26 June 2026

Self-healing pipelines: Argo vs Crossplane in 2026

Compare Argo CD and Crossplane for self-healing infrastructure: reconciliation models, health signals, failure modes, and how to measure detection and recovery yourself.

13 min read · 25 June 2026

Claude Code after a year: daily wins

Design an agentic Dependabot patcher that bumps a dependency, runs tests in a container, and opens a PR only when the suite passes.

13 min read · 25 June 2026

Istio and Cilium: what really breaks in production

Service meshes amplify latent platform failures. Here is what breaks in Istio and Cilium, how to measure it, and when to defer adoption.

13 min read · 24 June 2026

CI/CD for AI code: test, secure, rollback fast

A CI/CD pattern for AI-assisted code changes: deterministic locks, prompt-injection checks, smoke tests, metrics gates, and blue-green rollback.

13 min read · 23 June 2026

Choosing AI Coding Assistants for Cloud-Heavy Stacks

A practical framework for evaluating AI coding assistants on cloud-heavy stacks: what to measure, how to test, and which failure modes matter.

11 min read · 23 June 2026

AI-First vs Human-First Incident Response

Compare AI-first autonomous incident response with human-first ops plus AI triage: architecture, failure modes, cost modeling, and a decision checklist.

12 min read · 23 June 2026

Unlock Python 3.13's free-threaded GIL for web apps

Free-threaded Python removes the GIL, but C extensions, shared state and pools break. A practical checklist for migrating a web service safely.

14 min read · 22 June 2026

LLM evals: skip the hype, track these 4 metrics

A practical guide to production LLM evaluation: four metric classes, code for deterministic validators, and how to detect prompt drift and scorer decay.

11 min read · 22 June 2026

AI glue code: 3 ways to avoid duct-taping models

Low-code AI platforms hide prompt drift, adapter decay and cold starts. Detect each with logged queries, a golden set and pre-warming, wired into CI.

14 min read · 20 June 2026

Use feature flags to deploy AI models safely

Use feature flags to canary, shadow and roll back AI model changes without redeploys, with code, metrics and failure-mode handling.

15 min read · 20 June 2026

Why monoliths die slowly in 2026

Monolith decomposition usually fails on data locality, not routing. A step-by-step order for extracting data first, routing traffic later, with checks.

13 min read · 19 June 2026

Runbooks vs AI Runbooks: A Decision Framework

Compare static and AI-generated runbooks on detection, MTTR, cost, and auditability, with a 30-day experiment to test which fits your team.

12 min read · 18 June 2026

A Solo-Project Deploy Pipeline That Stays Under $20/Month

Build a test-and-deploy pipeline with GitHub Actions and a container host, then measure cost, cold start, and rollback time yourself.

16 min read · 15 June 2026

AI codebases: two onboarding paths that actually work

Compare two ways to onboard developers into codebases containing AI-generated code, with failure modes, cost math, and a decision checklist.

13 min read · 15 June 2026

AI rollouts live or die by flags

How to gate model versions, prompts, and safety filters behind feature flags, with a Lambda evaluator, SDK, timeout handling, canaries, and tests.

14 min read · 12 June 2026

Self-healing agents vs rule-based deployment rollback

Comparing AI-agent and rule-based deployment rollback: failure modes, what to instrument, and a decision checklist for choosing between them.

12 min read · 11 June 2026

Stay relevant when tools change faster than you can

A three-tier filter for deciding which new runtimes, frameworks and patterns deserve your time, with measurement steps and rollback plans.

16 min read · 9 June 2026