The Curious Engineer

All blogs / Distributed systems / cloud

Jack Vanlightly

jack-vanlightly.com

{ dist sys, formal verification, event streaming }

Distributed systems / cloudDatabasesPerformance / optimization

’16’17’18’19’20’21’22’23’24’25’26
Posts
208 from 4 Jan 2016 to 28 Aug 2026
Posting
monthly (21 in the last year)
Kind
personal blog
Feed
https://jack-vanlightly.com/blog?format=rss

2026 14

  1. The AtomicLog + LogDrive: Strong vs weak sealing
  2. The LogDrive: Flexible Composition Through Abstraction in Shared Logs
  3. Apache Kafka performance #1 - linger.ms
  4. 1BRC on a Threadripper 9980X
  5. Benchmarking Hardwood 1.0 on a Threadripper 9980X
  6. Kafka Share Groups - Pathological fetch waits with record_limit
  7. Can We Agree on a Storage/Workload Architecture Taxonomy?
  8. Raise the ambition threshold
  9. Kafka Share Groups and Parallelizing Consumption - Part 3: Client-local parallelism
  10. Broker-Visible vs Client-Local Parallelism
  11. Kafka Share Groups and Parallelizing Consumption - Part 2: Producer Batches and share.acquire.mode
  12. Kafka Share Groups and Parallelizing Consumption — Part 1: Tuning max.poll.records
  13. Benchmarking Apache Kafka Consumer Groups vs Share Groups (overhead test)
  14. Introducing Dimster, a performance benchmarking tool for Apache Kafka

2025 28

  1. The Three Durable Function Forms
  2. The Durable Function Tree - Part 2
  3. The Durable Function Tree - Part 1
  4. Coordinated Progress
  5. Demystifying Determinism in Durable Execution
  6. Have your Iceberg Cubed, Not Sorted: Meet Qbeast, the OTree Spatial Index
  7. How Would You Like Your Iceberg Sir? Stream or Batch Ordered?
  8. A Fork in the Road: Deciding Kafka’s Diskless Future — Jack Vanlightly
  9. Why I’m not a fan of zero-copy Apache Kafka-Apache Iceberg — Jack Vanlightly
  10. Beyond Indexes: How Open Table Formats Optimize Query Performance — Jack Vanlightly
  11. Understanding Apache Fluss — Jack Vanlightly
  12. table formats internals blog posts
  13. A Conceptual Model for Storage Unification — Jack Vanlightly
  14. Remediation: What happens after AI goes wrong? — Jack Vanlightly
  15. The Cost of Being Wrong — Jack Vanlightly
  16. Responsibility Boundaries in the Coordinated Progress model — Jack Vanlightly
  17. Coordinated Progress – Part 3 – Coupling, Synchrony and Complexity — Jack Vanlightly
  18. Coordinated Progress – Part 4 – A Loose Decision Framework — Jack Vanlightly
  19. Coordinated Progress – Part 2 – Making Progress Reliable
  20. Log Replication Disaggregation Survey - Apache Pulsar and BookKeeper
  21. Log Replication Disaggregation Survey - Kafka Replication Protocol
  22. Log Replication Disaggregation Survey - Neon and MultiPaxos
  23. Towards composable data platforms
  24. How to disaggregate a log replication protocol
  25. Steady on! Separating Failure-Free Ordering from Fault-Tolerant Consensus
  26. An Introduction to Virtual Consensus in Delos
  27. Why Snowflake wants streaming
  28. AI Agents in 2025

2024 48

  1. Verifying Kafka transactions - Diary entry 5 - AddPartitionsToTxn in TLA+
  2. On 1 million page views
  3. To be atomic or non-atomic, that is the question (Fizzbee)
  4. Verifying Kafka transactions - Diary entry 4 - Writing an initial Fizzbee spec
  5. An introduction to symmetry in TLA+
  6. Verifying Kafka transactions - Diary entry 3 - Getting confidence in the TLA+ spec
  7. Verifying Kafka transactions - Diary entry 2 - Writing an initial TLA+ spec
  8. Verifying Kafka transactions - Diary entry 1 - A first step
  9. Dismantling ELT: The Case for Graphs, Not Silos
  10. The Law of Large Numbers: A Foundation for Statistical Modeling in Distributed Systems
  11. Obtaining statistical properties through modeling and simulation
  12. Incremental Jobs and Data Quality Are On a Collision Course - Part 1 - The Problem
  13. Incremental Jobs and Data Quality Are On a Collision Course - Part 2 - The Way Forward
  14. On writing and getting from zero to done
  15. Learning first that you can learn
  16. Forget the table format war; it’s open vs closed that counts
  17. The teacher's nemesis
  18. The curse of Conway and the data space
  19. Change query support in Apache Paimon (0.8)
  20. Change query support in Apache Hudi (0.15)
  21. Table format interoperability, future or fantasy?
  22. Change query support in Delta Lake (3.2.0)
  23. Change query support in Apache Iceberg v2
  24. Table format comparisons - Change queries and CDC
  25. BYOC, not “the future of cloud services” but a pillar of an everywhere platform
  26. Constraints breed innovation and so does tenacity
  27. Table format comparisons - Streaming ingest of row-level operations
  28. Table format comparisons - Append-only tables and incremental reads
  29. Table format comparisons - How do the table formats represent the canonical set of files?
  30. Understanding Apache Iceberg’s Consistency Model Part 3
  31. Understanding Apache Iceberg’s Consistency Model Part 2
  32. Understanding Apache Iceberg's Consistency Model Part 1
  33. Understanding Apache Paimon's Consistency Model Part 1
  34. Understanding Apache Paimon's Consistency Model Part 2
  35. Understanding Apache Paimon's Consistency Model Part 3
  36. A Cost Analysis of Replication vs S3 Express One Zone in Transactional Data Systems
  37. Learning and reviewing system internals: tactics and psychology
  38. Hybrid Transactional/Analytical Storage
  39. Understanding Delta Lake's consistency model
  40. Understanding Apache Hudi's Consistency Model Part 1
  41. Understanding Apache Hudi's Consistency Model Part 2
  42. Understanding Apache Hudi's Consistency Model Part 3
  43. The Sisyphean struggle and the new era of data infrastructure
  44. Tableflow: the stream/table, Kafka/Iceberg duality
  45. The beauty of writing
  46. Scaling models and multi-tenant data systems - ASDS Chapter 6
  47. Serverless ClickHouse Cloud - ASDS Chapter 5 (part 1)
  48. Serverless ClickHouse Cloud - ASDS Chapter 5 (part 2)

2023 26

  1. S3 Express One Zone, not quite what I hoped for
  2. Serverless CockroachDB - ASDS Chapter 4 (part 1)
  3. Serverless CockroachDB - ASDS Chapter 4 (part 2)
  4. Serverless CockroachDB - ASDS Chapter 4 (part 3)
  5. Neon - Serverless PostgreSQL - ASDS Chapter 3
  6. Amazon DynamoDB - ASDS Chapter 1
  7. Kora - Serverless Kafka - ASDS Chapter 2
  8. The Architecture of Serverless Data Systems
  9. The importance of liveness properties (with TLA+ Part 1)
  10. The importance of liveness properties (with TLA+ Part 2)
  11. A primer on formal verification and TLA+
  12. The advantages of queues on logs
  13. On the future of cloud services and BYOC
  14. Kafka KIP-966 - Fixing the Last Replica Standing issue
  15. Kafka vs Redpanda Performance - Part 1 - 4 vs 50 producers
  16. Kafka vs Redpanda Performance - Part 2 - Long running tests
  17. Kafka vs Redpanda Performance - Part 3 - Hitting the retention limit
  18. Kafka vs Redpanda Performance - Part 4 - Impact of record keys
  19. Kafka vs Redpanda Performance - Part 5 - Reaching the limits of the NVMe drive
  20. Kafka vs Redpanda Performance - Part 6 - Draining backlogs
  21. Kafka vs Redpanda Performance - Do the claims add up?
  22. Is sequential IO dead in the era of the NVMe drive?
  23. Why Apache Kafka doesn't need fsync to be safe
  24. Paper: VR Revisited - Checkpoint-Based Replica Recovery (part 6)
  25. Paper: VR Revisited - Log-Based Replica Recovery (part 5)
  26. Paper: VR Revisited - Application state and commit-number monotonicity (part 4)

2022 7

  1. Paper: VR Revisited - State Transfer (part 3)
  2. Windows shortcut to invert colours that works
  3. Paper: VR Revisited - View changes - Answers (part 2)
  4. Paper: VR Revisited - An analysis with TLA+
  5. Paper: VR Revisited - View changes - Questions (part 1)
  6. Applying Flexible Paxos to Raft
  7. Write for others but mostly for yourself

2021 5

  1. Tweaking the BookKeeper protocol - Unbounded Ledgers
  2. Tweaking the BookKeeper protocol - Guaranteeing write quorum
  3. Kafka and RabbitMQ blog posts I wrote elsewhere in 2019
  4. Posts I wrote on the RabbitMQ blog in 2020
  5. Learn about TLA+ and the formal verification of Apache BookKeeper

2020 2

  1. With Great Observation Comes Great Insight
  2. Why I'm Not Writing Much On My Blog These Days

2019 6

  1. A Look at Multi-Topic Subscriptions with Apache Pulsar
  2. Building A "Simple" Distributed System - It's the Logs Stupid
  3. Building A "Simple" Distributed System - The Implementation
  4. Building A "Simple" Distributed System - Formal Verification
  5. Building A "Simple" Distributed System - The Protocol
  6. Building a "Simple" Distributed System - The What

2018 25

  1. Quorum Queues - Making RabbitMQ More Competitive in Reliable Messaging
  2. Why I Am Not a Fan of the RabbitMQ Sharding Plugin
  3. Testing Producer Deduplication in Apache Kafka and Apache Pulsar
  4. How to (not) Lose Messages on an Apache Pulsar Cluster
  5. Understanding How Apache Pulsar Works
  6. How to Lose Messages on a Kafka Cluster - Part 2
  7. How to Lose Messages on a Kafka Cluster - Part 1
  8. How to Lose Messages on a RabbitMQ Cluster
  9. RabbitMQ vs Kafka Part 6 - Fault Tolerance and High Availability with Kafka
  10. RabbitMQ vs Kafka Part 5 - Fault Tolerance and High Availability with RabbitMQ Clustering
  11. AWS Security - Securing Your Use of the AWS CLI and Automation Tools
  12. RabbitMQ Work Queues: Avoiding Data Inconsistency with Rebalanser
  13. Creating Consumer Groups in RabbitMQ with Rebalanser - Part 1
  14. Docker, .NET Core and Redshift Drivers
  15. Event-Driven Architectures - Queue vs Log - A Case Study
  16. Event-Driven Architectures - The Queue vs The Log
  17. SQL Server CDC to Redshift Pipeline
  18. Processing Pipelines Series - Reactive Extensions (Rx.NET)
  19. Processing Pipelines Series - TPL Dataflow - Alternate Scenario
  20. Processing Pipelines Series - TPL Dataflow
  21. Processing Pipelines Series - Concepts
  22. AgileTM vs Real Agility - The Constant Rush
  23. AgileTM vs Real Agility - Truly Self-Organizing Teams
  24. AgileTM vs Real Agility - The Fight Is On
  25. Don't Believe The Hype

2017 20

  1. RabbitMQ vs Kafka Part 4 - Message Delivery Semantics and Guarantees
  2. RabbitMQ vs Kafka Series Introduction
  3. RabbitMQ vs Kafka Part 1 - Two Different Takes on Messaging
  4. RabbitMQ vs Kafka Part 2 - RabbitMQ Messaging Patterns
  5. RabbitMQ vs Kafka Part 3 - Kafka Messaging Patterns
  6. You don't need Scrum, you need Agility
  7. Kathryn McConkey - The Best Osteopath in Barcelona
  8. How to Make Your Messaging System Reliable and Keep Your Support Engineers Happy
  9. DSL Parser - Sample Code
  10. RabbitMq Delayed Retry Approaches (That Work)
  11. Reliability - Custom Retries - NServiceBus with RabbitMq Part 6
  12. Reliability - Default Retries - NServiceBus with RabbitMq Part 5
  13. Custom Topology - NServiceBus with RabbitMq Part 4
  14. Custom Direct Topology - NServiceBus with RabbitMq Part 3
  15. Default Topologies - NServiceBus with RabbitMq Part 1
  16. Default Topologies - NServiceBus with RabbitMq Part 2
  17. How to Deal with Unroutable Messages - RabbitMq Publishing Part 3
  18. Types of Publishing Failures - RabbitMq Publishing Part 1
  19. Sending Messages in Bulk and Tracking Delivery Status - RabbitMq Publishing Part 2
  20. Have you got chronic eye pain like me?

2016 27

  1. new SqlConnection - The requested Performance Counter is not a custom counter
  2. Building Synkronizr - A SQL Server Data Synchronizer Tool - Part 1
  3. Generating SQL from a Data Structure
  4. How to Kill a Keep Alive with a Weak Reference (C#)
  5. How Row Locking Makes Taskling Concurrency Controls Possible
  6. Announcing Taskling.NET, a C# Batch Job API
  7. Too Busy to Create Your Own Visualizations? Just Leverage the Neo4J Console
  8. Why You Should Understand Databases
  9. Exploring the use of Hash Trees for Data Synchronization - Part 1
  10. 5 Tricks For People Who Are Hyper Sensitive to Computer Screens
  11. Optimizing Regex performance with RegexOptions.RightToLeft
  12. A More Efficient Regex Tokenizer
  13. Implementing a DSL Parser
  14. How to Create a Query Language DSL with C#
  15. Creating a Simple Tokenizer (Lexer) in C#
  16. Understanding Grammars
  17. Detective Optimizers
  18. The Freedom Quadrant
  19. Logging Query Language (LQL) – a DSL for Log Analytics
  20. Stop time wasters and protect your knowledge sharing culture
  21. Being Open, Sharing Your Knowledge
  22. Code Analysis Rules and the Click Through Culture
  23. Code analysis rules versus training and coaching
  24. Make mini technical leads
  25. Breaking your own code and designs
  26. "We"
  27. Bitching will erode your team