The Curious Engineer

All blogs / Distributed systems / cloud

Benjamin Cane

bencane.com

Practical engineering notes from Benjamin Cane, usually shared on LinkedIn first and kept readable on the open web.

Distributed systems / cloudPerformance / optimizationSoftware / engineering practice

’25’26
Posts
64 from 8 Aug 2025 to 28 Oct 2026
Posting
weekly (52 in the last year)
Kind
personal blog
Feed
https://bencane.com/feed.xml

2026 44

  1. Production bugs happen. Fix the gap, not just the bug.
  2. A timeout doesn’t mean an operation failed. It means you don’t know the outcome.
  3. Sometimes the biggest contributor to latency isn’t processing or network time. It’s waiting for work to start.
  4. System Design pro-tip: force yourself to create a viable second design.
  5. One of my favorite aspects of gRPC isn’t the performance. It’s the well-defined contracts Protobuf provides.
  6. Message queues are great for handing off work, but how do you know the work actually happens?
  7. Some architecture principles should be rules. Others should be guidelines.
  8. Mishandling concurrency is one of the most common root causes of software bugs
  9. At what point does better performance stop being worth it?
  10. Sometimes good engineering looks like over-engineering
  11. AI makes code cheap to create, not cheap to own
  12. “We can’t run locally” is usually a design smell
  13. To make a service more stable, eliminate dependencies
  14. Caching isn’t hard. Some data is hard to cache
  15. The closer to the edge, the more stable a platform must be
  16. Sometimes the most resilient thing a system can do isn’t retry
  17. Should retries and timeouts live in your application or your service mesh?
  18. Need to migrate from one database to another without downtime?
  19. Glue Services: Part Two — Data Synchronization
  20. When modernizing legacy systems, don’t be afraid to build glue services
  21. Coding agents can’t see your architecture diagrams—fix that
  22. Most teams put low-level architecture in the wrong place
  23. Your coding agent is missing one thing: architectural context
  24. Health-check the listener your gRPC traffic actually uses
  25. Weighted load balancing has saved me more times than I can count
  26. YOLO Is a Terrible Strategy for Validating Production Changes
  27. April 30, 2026 Deterministic routing is one of the most effective ways distributed systems reduce consistency problems at scale performance
  28. April 23, 2026 When you think of microservices, you probably think of centralized shared services. But there's another valid pattern that is rarely discussed architecture
  29. April 16, 2026 Are you using traffic mirroring in production? If not, try it out. reliability
  30. April 9, 2026 Agent Skills Are Becoming the Best Way to Capture Institutional Knowledge agents
  31. April 2, 2026 Saved Prompts Are Dead. Agent Skills Are the Future. agents
  32. March 26, 2026 Generating Code Faster Is Only Valuable If You Can Validate Every Change With Confidence agents
  33. March 19, 2026 When You Go to Production with gRPC, Make Sure You’ve Solved Load Distribution First reliability
  34. March 12, 2026 You may be building for availability, but are you building for resiliency? reliability
  35. March 5, 2026 When your coding agent doesn’t understand your project, you’ll get junk agents
  36. February 26, 2026 You can have 100% Code Coverage and still have ticking time bombs in your code. 💣 engineering
  37. February 19, 2026 Getting More Out of Agentic Coding Tools agents
  38. February 12, 2026 Why is Infrastructure-as-Code so important? Hint: It's correctness architecture
  39. February 5, 2026 Optimizing the team’s workflow can be more impactful than building business features engineering
  40. January 29, 2026 I follow an architecture principle I call The Law of Collective Amnesia architecture
  41. January 22, 2026 Performance testing without a target is like running a race with no finish line performance
  42. January 15, 2026 Many teams think performance testing means throwing traffic at a system until it breaks. That approach is fine, but it misses how systems are actually stressed in the real world. performance
  43. January 8, 2026 Pre-populating caches is a “bolt-on” cache-optimization I've used successfully in many systems. It works, but it adds complexity performance
  44. January 1, 2026 Don't be afraid to build a tool. Just don't become too attached to it. engineering

2025 20

  1. December 26, 2025 One of the toughest engineering skills to develop is accepting a decision you disagree with. 😖 engineering
  2. December 19, 2025 Canary deployments are an operational superpower, but the complexity they bring isn’t for everyone. reliability
  3. December 12, 2025 Everyone has bias, yes, even you. 🫵 engineering
  4. December 5, 2025 Do you use Architecture Decision Records? I’m a big fan, and I think they’re a best practice every engineering org should adopt. architecture
  5. November 28, 2025 Does resource usage within your application or database suddenly spike periodically? Does it cause system slowdown? reliability
  6. November 21, 2025 When you shut down an application instance, don't stop the listener immediately — that's how you end up with failed requests during every application rollout. 😢 reliability
  7. November 14, 2025 A common issue I see when teams first adopt gRPC is managing persistent connections, especially during failovers. reliability
  8. November 7, 2025 A dangerous mindset I’ve seen—and been guilty of—is assuming code doesn't change. engineering
  9. October 31, 2025 ⚡️Does saving 1 millisecond really matter? Answer: more than you’d think. performance
  10. October 27, 2025 Have you heard of Store and Forward? It’s a resiliency design prevalent in card & bank payments, telecommunications, and other industries. reliability
  11. October 24, 2025 When Building Low-Latency, High-Scale Systems, Push as Much Processing as Possible to Later performance
  12. October 10, 2025 Coding is a small part of software engineering. engineering
  13. October 3, 2025 Should I be an individual contributor or a people leader? engineering
  14. September 26, 2025 Improve performance and reduce chances of request failures with this one simple trick! Avoid cross-region calls. performance
  15. September 19, 2025 Did you know Kube-proxy doesn’t perform load-balancing itself? It’s iptables (by default). performance
  16. September 12, 2025 You’ve heard of feature flags, but what about operational flags? ⏯️ engineering
  17. September 5, 2025 A core capability for building low-latency platforms is quickly detecting and reacting to issues. reliability
  18. August 22, 2025 Sometimes when I tell people that logging can impact a microservices response time, I get strange looks. 🤨 performance
  19. August 15, 2025 How many times have you seen analytics on an operational database create issues? I’ve seen it far too often. performance
  20. August 8, 2025 I can't count how often I've seen issues made worse by minor oversights—like not setting a timeout value. ⏱️ reliability