The Curious Engineer

All blogs / DevOps / SRE / infrastructure

Surfing Complexity

surfingcomplexity.blog

Lorin Hochstein's ramblings about software, complex systems, and incidents.

DevOps / SRE / infrastructureDistributed systems / cloudSoftware / engineering practice

’15’20’25
Posts
405 from 12 Dec 2011 to 19 Oct 2026
Posting
weekly (57 in the last year)
Kind
personal blog
Feed
https://surfingcomplexity.blog/feed/

2026 45

  1. Anthropic and the changing nature of work
  2. Quick thoughts on GitHub’s Sept 23 incident
  3. There was a light that never went out
  4. Saturation at GitHub: the saga continues
  5. How the incident happened: a play in one act
  6. Thoughts on the July 8th Telstra outage report
  7. Thoughts on the Buildkite Aug 25 incident
  8. Quick thoughts on GitHub Actions Aug 26 incident
  9. Omnipresent availability risks in cloud software
  10. Wild AI-related reliability incidents are coming
  11. GitHub, autoscaling, and the component substitution fallacy
  12. Tough days at GitHub, a continuing series
  13. Quick thoughts on Azure Regional Outage from July 23, ’26
  14. GitHub has another tough day
  15. Traditional versus resilience engineering views
  16. My talk from the Software Should Work conference
  17. Synthesis is harder than analysis
  18. I am dreading our LLM-written incident report future
  19. Dear researchers column
  20. I can’t bear to read AI-generated prose
  21. The demon of the gaps
  22. Form may follow function, but use doesn’t follow design
  23. The coming coordination calamity
  24. Reliability as a game of improving the odds
  25. Flipping the bozo bit on flips the learning off
  26. How incidents can teach us about what’s already working well
  27. My SREcon talk
  28. Life comes at you fast
  29. The normal work of creating reliability
  30. Thoughts on the Bluesky public incident write-up
  31. References from my SREcon talk on stories
  32. Quick thoughts on GitHub CTO’s post on availability
  33. Grow fast and overload things
  34. Saturation
  35. Quick takes on Feb 20 Cloudflare outage
  36. Poor Deming never stood a chance
  37. Lots of AI SRE, no AI incident management
  38. Nobody knows how the whole system works
  39. On variability
  40. Ashby taught us we have to fight fire with fire
  41. Because coordination is expensive
  42. Amdahl, Gustafson, coding agents, and you
  43. From Rasmussen to Moylan
  44. Telling the wrong story
  45. Verizon outage report predictions

2025 50

  1. On intuition and anxiety
  2. The dangers of SSL certificates
  3. Saturation: Waymo edition
  4. Another way to rate incidents
  5. Quick takes on the Triple Zero Outage at Optus – the Schott Review
  6. Why I don’t like “Correction of Error”
  7. AWS re:Invent talk on their Oct ’25 incident
  8. Quick takes on the Dec 5 Cloudflare outage
  9. Incidents: the exceptional as routine
  10. Fun with incident data and statistical process control
  11. Brief thoughts on the recent Cloudflare outage
  12. You’ll never see attrition referenced in an RCA
  13. Quick thoughts on the recent AWS outage
  14. This is fine!
  15. Caveat promptor
  16. The illegible nature of software development talent
  17. Two thought experiments
  18. A statistic is as a statistic does
  19. Fixation: the ever-present risk during incident handling
  20. The hidden trade-offs of fine-grained progressive rollouts
  21. Nothing fails like a history of success
  22. My favorite developer productivity research method that nobody uses
  23. The problems that accountability can’t fix
  24. Easy will always trump simple
  25. The trap of tech that’s great in the small but not in the large
  26. Formal specs as sets of behaviors
  27. Cloudflare and the infinite sadness of migrations
  28. Re-reading Technopoly
  29. Component defects: RCA vs RE
  30. “What went well” is more than just a pat on the back
  31. Quick takes on the GCP public incident write-up
  32. AI at Amazon: a case study of brittleness
  33. Pattern machines that we don’t understand
  34. Dijkstra never took a biology course
  35. The same incident never happens twice, but the patterns recur over and over
  36. LLMs are weird, man
  37. Not causal chains, but interactions and adaptations
  38. Labeling a root cause is predicting the future, poorly
  39. On work processes and outcomes
  40. When a bad analysis is worse than none at all
  41. You can’t prevent your last outage, no matter how hard you try
  42. Good models protect us from bad models
  43. Tradeoff costs in communication
  44. Model error
  45. Models, models every where, so let’s have a think
  46. Paxos made visual in FizzBee
  47. Locks, leases, fencing tokens, FizzBee!
  48. Resilience: some key ingredients
  49. You’re missing your near misses
  50. The danger of overreaction

2024 46

  1. Whither dashboard design?
  2. The Canva outage: another tale of saturation and resilience
  3. Quick takes on the recent OpenAI public incident write-up
  4. Your lying virtual eyes
  5. MTTR: When sample means and power laws combine, trouble follows
  6. Quick takes on the latest Cloudflare public incident write-up
  7. The Tortoise and the Hare in Alloy
  8. TTR: the out-of-control metric
  9. Reading the Generalized Isolation Level Definitions paper with Alloy
  10. Extending MVCC to be serializable, in TLA+
  11. Multi-version concurrency control in TLA+
  12. The carefulness knob
  13. Specifying serializability in TLA+
  14. If you don’t examine what worked, how will you know what works?
  15. A liveness example in TLA+
  16. Futexes in TLA+
  17. Why I don’t like discussing action items during incident reviews
  18. Linearizability! Refinement! Prophecy!
  19. Safety first!
  20. You can specify even when you can’t implement
  21. The “CrowdStrike” approach to reliability work
  22. CrowdStrike: how did we get here?
  23. Modeling a CLH lock in TLA+
  24. Reproducing a Java 21 virtual threads deadlock scenario with TLA+
  25. Incidents as keys to a lot of value
  26. Second-class interactions are a first-class risk
  27. Expect it most when you expect it least
  28. A still, small sound
  29. Book review: How Life Works
  30. Quick takes on Rogers Network outage executive summary
  31. Dirty writes
  32. Efficiency and bad outcomes
  33. Modeling B-trees in TLA+
  34. Action item template
  35. You can’t judge risk in hindsight
  36. The problem with a root cause is that it explains too much
  37. The error term isn’t Pareto distributed
  38. Green is the color of complacency
  39. The perils of outcome-based analysis
  40. The problem with invariants is that they change over time
  41. When there’s no gemba to go to
  42. What if everybody did everything right?
  43. Book Review: Trust in Numbers
  44. Negotiability is non-negotiable
  45. You should’ve known how to build a non-causal system
  46. Consistency

2023 32

  1. The inherent weirdness of system behavior
  2. Tell me about a time…
  3. On chains and complex systems
  4. The courage to imagine other failures
  5. Any change can break us, but we can’t treat every change the same
  6. “Human error” means they don’t understand how the system worked
  7. For want of a dollar
  8. Accidents manage you
  9. The surprising power of a technical document written by experts
  10. On productivity metrics and management consultants
  11. Operating effectively in high surprise mode
  12. Normal incidents
  13. Why LFI is a tough sell
  14. Oddly influenced podcast
  15. Active knowledge
  16. Resilience requires helping each other out
  17. If you can’t tell a story about it, it isn’t real
  18. When there’s no plan for this scenario, you’ve got to improvise
  19. Treating uncertainty as a first-class concern
  20. My SREcon 23 talk is up
  21. Missing the forest for the trees: the component substitution fallacy
  22. Some thoughts on my presentation style
  23. Making peace with the imperfect nature of mental models
  24. You can’t see what you don’t have a frame of reference for
  25. My talk from the LFI conference
  26. Good category, bad category (or: tag, don’t bucket)
  27. The value of social science research
  28. The high cost of low ambiguity
  29. Why you can’t buy an observability solution off the shelf
  30. Does any of this sound familiar?
  31. A small mistake does not a complex systems failure make
  32. The Allspaw-Collins effect

2022 32

  1. Southwest airlines: a case study in brittleness
  2. Knowledge and Power: A Sociotechnical Systems Discussion on the Future of SRE
  3. Incident categories I’d like to see
  4. CasesConf experience report
  5. Cache invalidation really is one of the hardest problems in computer science
  6. Writing docs well: why should a software engineer care?
  7. Time keeps on slippin’: A review of “Four thousand weeks”
  8. The down side of writing things down
  9. There is no “Three Mile Island” event coming for software
  10. Up and down the abstraction hierarchy
  11. Production pressure
  12. Uvalde: a reasonable officer
  13. Uvalde: would you have taken the shot?
  14. Common ground breakdown in Uvalde
  15. The fog of war in Uvalde
  16. What’s allowed to count as a cause: ALERRT edition
  17. Uvalde
  18. Incident response is a team sport
  19. You’re just going to sit there???
  20. Imagine there’s no human error…
  21. She Blinded Me With Science: A review of Galileo’s Middle Finger
  22. Bad Religion: A review of Work Pray Code
  23. We value possession of experience, but not its acquisition
  24. Software engineering in-the-large: the coordination challenge
  25. OR, DevOps, and LFI
  26. Prussia meets Versailles: a review of Moral Mazes
  27. Making operational work more visible
  28. Code rewrites and joint cognitive systems
  29. Taking the hit
  30. Bitrot
  31. Aristotle’s revenge
  32. Henry Yin on what the cyberneticists got wrong

2021 46

  1. The ambiguity of real work
  2. The Howie Guide: How to get started with incident investigations
  3. I have no idea what I’m doing
  4. How much did that outage cost?
  5. “What could we have done differently?”
  6. OOPS writeups
  7. The danger of hidden functional roles
  8. Plus c’est la même chose, plus ça change
  9. Adapting to a crunch: the Mask Match story
  10. The strange beauty of strange loop failure modes
  11. Live-drawing my slides during a talk
  12. Useful knowledge and improvisation
  13. Inconceivable
  14. Software Misadventures Podcast
  15. Contempt for the glue people
  16. Remembering the important bits when you need them
  17. The power of framing a problem
  18. The local nature of culture
  19. What do you work on, anyway?
  20. Modernists trapped in a post-modern universe
  21. Root cause of failure, root cause of success
  22. Burned by ‘let it burn’
  23. StaffEng podcast
  24. What’s allowed to count as a cause?
  25. The greedy exec trap
  26. Trouble during startup
  27. Controlling a process we don’t understand
  28. Designing like a joint cognitive system
  29. You have to go there
  30. Incident writeup as sociological storytelling
  31. Grappling with contingency
  32. The disaster meeting
  33. Transgressing the boundaries: Rasmussen and Woods
  34. Dealing with new kinds of trouble
  35. Objectives and constraints
  36. Why do config changes keep coming up in major incidents?
  37. Subverting the process
  38. Naming names in incident writeups
  39. Incident analysis as guerrilla case study research
  40. The British General’s problem
  41. Dog smartness
  42. Putting on a different hat
  43. Some quick thoughts on Citi’s $900M OOPSie
  44. Slack’s Jan 2021 outage: a tale of saturation
  45. Error budgets and the legacy of Herbert Heinrich
  46. Making sense of what happened is hard

2020 41

  1. Conscription devices, boundary objects, and GDocs
  2. Coding as a tool of thought
  3. Uber’s adventures in the adaptive universe
  4. Software as a limited medium
  5. Top-down code reviews
  6. Why you should write up your own incident
  7. Taming complexity: from contract to compact
  8. Even the U.S. military
  9. Battleshorts, exaptations, and the limits of STAMP
  10. Programming means never getting to say “it depends”
  11. Abstractions and implicit preconditions
  12. A nudge in the right direction
  13. Thoughts on STAMP
  14. Soak time
  15. In praise of the Wild West engineer
  16. Safe by design?
  17. All we can do is find problems
  18. The power of functionalism
  19. “How could they be so stupid?”
  20. A reasonable system
  21. Who’s afraid of serializability?
  22. SRE, CSE, and the safety boundary
  23. This mess we’re in
  24. Why you can’t just ask “why”
  25. Asking the right “why” questions
  26. Making peace with “root cause” during anomaly response
  27. The hard parts about making it look easy
  28. On the writing styles of some resilience engineering researchers
  29. Chasing down the blipperdoodles
  30. The inevitable double bind
  31. Embracing the beautiful mess
  32. How did software get so reliable without proof?
  33. Rebrand: Surfing Complexity
  34. Rehabilitating “you can’t manage what you can’t measure”
  35. An old lesson about a fish
  36. Have you seen this before?
  37. There is no escape from the adaptive universe
  38. There is no escape from Ashby’s Law
  39. The downsides of expertise
  40. Getting into people’s heads: how and why to fake it
  41. Conveying confusion without confusing the reader

2019 20

  1. Learning From Incidents in Software site
  2. Climbing the mountain
  3. Taking a risk versus running a risk
  4. Stories as a vehicle for learning from the experience of others
  5. The Gamma Knife model of incidents
  6. In service of the narrative
  7. The problem with counterfactuals
  8. Experts aren’t good at building shared understanding
  9. Tuning to the future
  10. Contributors, mitigators & risks: Cloudflare 2019-07-02 outage
  11. Contributors, mitigators & risks: Stripe 2019-07-10 outage
  12. What Deming got wrong
  13. Why incidents can’t be monocausal
  14. Elegance: UI vs implementation
  15. Postmodern engineering
  16. The seductiveness of single-metric decisions
  17. Root cause: line in Shakespearean play
  18. The danger of “insufficient virtue”
  19. Notes on David Woods’s Resilience Engineering short course
  20. Our brittle serverless future

2018 4

  1. Inductive invariants
  2. TLA+ is hard to learn
  3. Why we will forever suffer from missing timeouts, TTLs, and queue size bounds
  4. The Equifax breach report

2017 10

  1. The Tortoise and the Hare in TLA+
  2. Death of a pledge as systems failure
  3. Antics, Drift and Chaos
  4. Assumption of rationality
  5. A conjecture on why reliable systems fail
  6. Engineering research reveals wrongdoing
  7. Sketching on the back end
  8. I’d rather read code than tests
  9. How open source beat agile
  10. Changing a complex system is hard

2016 3

  1. Personal productivity tools
  2. When software takes a human life
  3. Software development as engineering

2015 5

  1. Operations engineering
  2. Review: Software Developers’ Perceptions of Productivity
  3. Over-engineering as a safety technique
  4. Ansible: Up and Running is out!
  5. New paper review

2014 18

  1. Ansible: Up and Running
  2. Paper review: Simple Testing Can Prevent Most Critical Failures
  3. Cloud software, fragility and Air France 447
  4. Estimating confidence intervals, part 6
  5. Good to great
  6. Estimating confidence intervals, part 5
  7. Estimating confidence intervals, part 4
  8. Not apprenticeship!
  9. Estimating confidence intervals, part 3
  10. Crossing the river with TLA+
  11. Estimating confidence intervals, part 2
  12. Presentation as text
  13. “Who owns the fish” in Alloy
  14. Results from estimating confidence intervals
  15. Happier developers are more productive
  16. Reading academic papers on a Kindle Paperwhite
  17. Tackling Hofstadter’s Law with confidence intervals
  18. People don’t understand computer science

2013 14

  1. Adopt an op
  2. Up and running on Rackspace
  3. Automated DevStack install inside of VirtualBox with Vagrant
  4. developerWorks article
  5. Head banging odds ratio
  6. The challenge of the network
  7. Books I wish existed
  8. Software Analysis
  9. No Country for IT
  10. Operator fault tolerance
  11. Payroll systems, not yet a solved problem
  12. Open-source documentation in the wild
  13. ESEM 2013 Industry Track CFP
  14. This really should be easier

2012 29

  1. Tech blog
  2. The good ones tell you when they’re wrong
  3. Line by line
  4. Publications trump ideology
  5. Training is a dirty word
  6. Ansible, Vagrant and OpenStack
  7. Relative confidence in scientific theories
  8. You never really know where you’ll end up
  9. “Don’t go into science”
  10. Black-hat SEO comes to the academy
  11. A point of view is not a bad thing
  12. When something you believe is false
  13. The software engineering grand challenge problem researchers ignore
  14. Apparently, I introduced a phenomenon
  15. Statistician, measure thyself!
  16. Empirical Software Engineering Stack Exchange Proposal
  17. There is no toggle lock signal
  18. How do you capture that?
  19. Unit testing: because all of the cool kids are doing it
  20. Predicting individual performance is hard
  21. Lacking a frame of reference
  22. Sitting has become synonymous with sloth
  23. I predict controversy
  24. Redacting results in reviews
  25. So, what happens if the programmer makes an error?
  26. Well-deserved
  27. Data proposal needs some love
  28. Tumbling
  29. Transition

2011 10

  1. IEEE and the newspaper industry
  2. Bifurcation
  3. Weird and gross
  4. A preliminary empirical study to compare MPI and OpenMP
  5. Newer interfaces aren’t always better
  6. Rankings are hard
  7. Apple and published research
  8. Constraints and complex software deployment
  9. Code is for reading, docs are for writing
  10. Swing and a miss