The Curious Engineer

All blogs / Distributed systems / cloud

Christopher Meiklejohn

christophermeiklejohn.com

Distributed systems researcher and technical lead. Staff Engineer at DoorDash, adjunct faculty at CMU. From award-winning research to Azure, AWS, and production Erlang systems.

Distributed systems / cloudAI / MLTesting / QA

’15’20’25
Posts
134 from 3 Jun 2013 to 2 Oct 2026
Posting
weekly (45 in the last year)
Kind
personal blog
Feed
https://christophermeiklejohn.com/feed.xml

2026 45

  1. The Things I Wanted to Make
  2. Becoming, Pending Review
  3. Fiction:
  4. The Shortcut Became the System · A pipe disappeared. The rectangle around the door remained.
  5. My Front of House of Cards · The show ended at 10:12. The fix merged at 10:17.
  6. Software for One
  7. The Quickest Path to a Diff · The app died at Red Rocks while its health checks stayed green. Every line of code was fine. The sum was nobody's job.
  8. Every Card Will Show · The earlier implementation, tests, audits, and accepted proofs said the algorithm was correct. None of them stated the whole problem.
  9. The Safeguards Have to Run · Two abandoned notebooks showed me the difference between a research rule and a working control.
  10. The Machine in the Lab:
  11. The Whole Night
  12. Science at LLM Speed · Research-like output is cheap. A valid claim still needs evidence that can show it is wrong.
  13. One Writer · Our tools assume one writer, and assume that writer is a human. Nothing computes what a change reads and writes at runtime, so the only known fix is brute force priced for organizations.
  14. The Hard Call Is the Whole Job · A cancelled flight, three confident machines, and the woman on the Hilton Diamond line who refused to stop.
  15. The App That Lives Between Shows
  16. The Test Suite Was the Incident · A night of brittle fixtures, 49 failed CI runs, and an expensive lesson about what happens when AI agents write the test suite and nobody owns the test data.
  17. Rift · For thirty years I programmed with Phish on, every day. In 2026, the music is out of phase with the work.
  18. Babysitting the Agent · Two weeks in, even with all the hooks I've built, working with the agent has become a chore. Every shipped feature ends with me clicking through it to find out what didn't actually work.
  19. Getting Up to Speed on Multi-Agent Systems, Part 8: Open Questions
  20. Getting Up to Speed on Multi-Agent Systems, Part 7: Benchmarks and What They Miss
  21. Getting Up to Speed on Multi-Agent Systems, Part 6: Verification Patterns
  22. Getting Up to Speed on Multi-Agent Systems, Part 5: Debate, State, and Coordination
  23. Getting Up to Speed on Multi-Agent Systems, Part 4: Wave 2 (Why It Breaks)
  24. Spring Tour Recap: A Month of Shipping on Zabriskie
  25. Getting Up to Speed on Multi-Agent Systems, Part 3: Wave 1 (Can Agents Coordinate At All?)
  26. Getting Up to Speed on Multi-Agent Systems, Part 2: The Vocabulary
  27. Getting Up to Speed on Multi-Agent Systems, Part 1: The Landscape
  28. Getting Up to Speed on Multi-Agent Systems
  29. The Tribe Has to Outlive the Model · Five model swaps in three weeks taught me the project's continuity lives in the humans and conventions around the agents, not the agents themselves.
  30. The Tax on the Happy Path · What I learned by removing the elaborate CI gate I'd built to slow my AI agents down — after realizing it had never actually caught anything.
  31. Opt-In Isn't a Guardrail · A CI gate I built to slow AI agents down passed every paperwork check, missed every real bug, and blocked the rollback during a production outage.
  32. The Structural Engineer's Other Job · AI-generated code increasingly passes review and tests but ships half-working features. What humans have to check for instead.
  33. Caucus V1: Cursor Background Agents and a Multi-Agent Workflow That Actually Loops
  34. The Feature That Has Never Worked · A broken auto-live poller, and what perceived urgency does to Claude Code
  35. Software Engineering Is Becoming Civil Engineering
  36. Multi-Agent Systems Have a Distributed Systems Problem
  37. The Show Is Happening Right Now and Nothing Works
  38. Memory Isn't Learning
  39. Finding Safe Food on the Road
  40. Teaching Claude to QA a Mobile App
  41. What Building With Claude Actually Looks Like
  42. Why I'm Building Zabriskie
  43. Claude Tested Everything Except the One Thing That Mattered
  44. Ten Years of Lasp
  45. I Built a Social App in a Week with Claude Code

2023 19

  1. Filibuster 2.0: Byzantine Fault Injection with Hardcoded Fault Values
  2. Filibuster 2.0: Coaching UI
  3. Filibuster 2.0: Byzantine Fault Injection with Arbitrary Faults
  4. Filibuster 2.0: Computing API coverage of a Microservice Application
  5. Filibuster 2.0: Microservice Linter, Multiple Invocations to the Same RPC Method
  6. Filibuster 2.0: Microservice Linter, Requests become part of a Response
  7. Filibuster 2.0: Microservice Linter, Redundant RPCs
  8. Filibuster 2.0: Improved UI
  9. Filibuster 2.0: Healthcheck your Functional Test Suite with API Coverage
  10. Filibuster 2.0: Redis Fault Injection
  11. Filibuster 2.0: Introducing the new version of Filibuster
  12. Filibuster 2.0: Developer Tooling as a Childhood Dream
  13. Reflecting on 10 years of Blogging, Research, and a (very) long Ph.D.
  14. Testing Applications Resilience in the Presence of Service Tiers
  15. Nested RPCs with Filibuster's UX
  16. Testing Applications for Resilience Bugs vs Testing Applications for Resilience
  17. Revisiting: Chaos Engineering or Software Testing?
  18. Building Tech-Transferrable Research Software
  19. Chaos Engineering or Software Testing?

2022 5

  1. The production handoff →
  2. Extending Filibuster to Test Redis
  3. Zabriskie
  4. Understanding why Resilience Faults in Microservice Applications Occur
  5. What is Chaos Engineering?

2021 6

  1. Dynamic Reduction: Optimizing Service-level Fault Injection Testing With Service Encapsulation
  2. Finding Resilience Issues with Filibuster: A Tutorial
  3. What is Service-level Fault Injection Testing?
  4. Researching Microservice Resilience: Lack of Corpora
  5. Food: A Bittersweet Affair
  6. Erlang's Process Identifiers and My Weekend Bug Hunting

2020 2

  1. COVID-19 Teaching Experience: 17-313 Foundations of Software Engineering
  2. COVID-19 Teaching Experience: Agile

2019 7

  1. Mixed-Cluster Versioning: A Story of Capabilities, Erlang and Riak
  2. Implementing Coordination Free Convergence of Data for Edge Computing
  3. Motivating CRDTs for the Edge
  4. Stateful Serverless: Critical Sections with Azure Durable Functions
  5. Introduction to Stateful Serverless: key-value database with Azure Durable Entities
  6. Using Fault-Injection to Evolve a Reliable Broadcast Protocol
  7. Applied Monotonicity: A Brief History of CRDTs in Riak

2018 6

  1. High performance, exactly-once, failure oblivious distributed programming with AMBROSIA
  2. Guesstimate: A Programming Model for Collaborative Distributed Systems
  3. A CAPable Distributed Programming Model
  4. Inconsistent, Performance-bound, Approximate (IPA)
  5. Anatomy of a Glutening
  6. Celiac Disease: Debugging Your Own Research Life With Blameful Postmortems

2016 8

  1. Argus
  2. Programming Language in the Sky (PLITS)
  3. Remote Procedure Call
  4. Emerald
  5. Hermes
  6. Lasp and the Google Summer of Code 2016
  7. Promises
  8. 2015: A Year in Review

2015 11

  1. Thesis Proposal, Draft
  2. A Certain Tendency Of The Database Community
  3. Programming Models, Part 6: Leaderboard
  4. Meiklejohn's Razor
  5. Moving On, Part 2: Ph.D.
  6. Moving On
  7. Orleans, Distributed Virtual Actors for Programming and Scalability Comparison
  8. Seminars on Lasp in March, 2015
  9. Highly Distributed Computations Without Synchronization
  10. Programming Models, Part 5: Ad Counter, Part 2
  11. 2014: A Year in Review

2014 6

  1. Programming Models, Part 4: One Week in Louvain-la-Neuve
  2. Programming Models, Part 3: Ad Counter, Part 1
  3. Programming Models, Part 2: QuickChecking Derflow
  4. Programming Models, Part 1: Try Derflow!
  5. Readings in conflict-free replicated data types
  6. Readings in linear logic

2013 19

  1. Verified Vector Clocks: An Experience Report, Part 3
  2. Verified Vector Clocks: An Experience Report, Part 2
  3. Announcing Proceedings: Distributed Reading Groups
  4. Verified Vector Clocks: An Experience Report, Part 1
  5. Bloom at Wicked Good Ruby 2013
  6. Erlang/OTP R16B02 on SmartOS
  7. From Ruby To Erlang: An Experience Report
  8. Readings in distributed systems
  9. LambdaJam 2013: Webmachine Tutorial
  10. Guest on the Food Fight show podcast
  11. Riak PG: Pruning terminated processes
  12. Modified Flume HDFS sink for one to one files
  13. Introducing Riak PG: Distributed process groups for Erlang
  14. Distributed data structures with Coq: PN-Counters
  15. Flume-ng configuration with an HDFS sink
  16. Functional reactive programming in JavaScript
  17. Distributed data structures with Coq
  18. Erlang gproc Failure Semantics
  19. Erlang pg2 Failure Semantics