The Curious Engineer

All blogs / Python

Luminousmen Blog

luminousmen.com

helping robots conquer the earth and trying not to increase entropy using Python, Data Engineering, Machine Learning

PythonDatabasesAI / ML

’17’18’19’20’21’22’23’24’25’26
Posts
159 from 18 Apr 2017 to 26 Oct 2026
Posting
weekly (32 in the last year)
Kind
personal blog
Feed
https://luminousmen.com/feeds/

2026 29

  1. CAP Theorem and PACELC Explained in Plain English
  2. I Miss Writing Code
  3. Python Interview Questions for Junior Developers (50 Q&A)
  4. Python Interview Questions for Mid-Level Developers (50 Q&A)
  5. Python Interview Questions for Experienced (Senior) Developers: 58 Q&A
  6. Benchmaxxing
  7. Apache Spark Core Concepts: RDDs, DAGs, and How Spark Actually Works
  8. Big Data File Formats: CSV, JSON, Avro, Parquet Compared
  9. Spark Partitioning: Repartition vs Coalesce, Shuffle & AQE
  10. Write Skew: The Anomaly the SQL Standard Forgot
  11. ACID vs BASE: Comparison of two Design Philosophies
  12. Spark Tips. Partition Tuning
  13. Fundamentals of Trust in Engineering
  14. Most of Your Backfills Didn't Have to Happen
  15. My Empathy Is a Sticker Detector
  16. Design Patterns Suck
  17. Stop Building AI That Agrees With You
  18. Why is Apache Spark RDD Immutable?
  19. If Your Data Engineer Can Be Replaced by an Agent, You Already Hired Badly
  20. The Apache Spark Optimization Checklist
  21. Spark Memory Management: The Complete 2026 Guide
  22. Drunk Post: Things I’ve Learned as a Senior Engineer
  23. Mastering Theta Sketches for Cardinality Estimation
  24. The Power of Data Sketches: A Comprehensive Guide
  25. Stop Feeding Me AI Slop
  26. Be the Idiot
  27. When to Say No to “Platform Work”
  28. Why Your 5-Second BigQuery Query Isn’t Cheap
  29. Spark Caching Explained: What Really Happens Under the Hood

2025 26

  1. BigQuery Explained: What Really Happens When You Hit “Run”
  2. What Do You Think?
  3. Data Warehouse, Data Lake, Data Lakehouse, Data Mesh: What They Are and How They Differ
  4. The Golden Age of dbt
  5. How Not to Partition Data in S3 (And What to Do Instead)
  6. AI Fluency Expectations
  7. Compaction in Lakehouse
  8. How the Community Turned Into a SaaS Commercial
  9. Anatomy of Apache Spark Application
  10. Choosing the Right Compression Codec
  11. Cluster Managers for Apache Spark: from YARN to Kubernetes
  12. Words Are the New Bytecode
  13. Only the Squeaky Wheel Gets the Oil
  14. Data Partitioning: Partition. Regret. Repeat
  15. Why Parquet Is the Go-To Format for Data Engineers
  16. Pip Constraints Files
  17. Data Partitioning: Slice Smart, Sleep Better
  18. Understanding AWS Regions and Availability Zones: A Guide for Beginners
  19. Change Data Capture (CDC)
  20. Data Engineering: Now with 30% More Bullshit
  21. Mastering Project Clarity: The Power of the RACI Matrix
  22. Management is not a promotion
  23. How to Drop a BigQuery Sharded Table
  24. From Chaos to Clarity: The Power of a Definition of Done
  25. Data Lake vs Data Warehouse
  26. Nobody Actually Cares

2024 81

  1. From ETL and ELT to Reverse ETL
  2. Mastering Database Design: Denormalization
  3. INSERT vs UPDATE and Their Impact on Data Ingestion
  4. Multiple levels of parallel hardware
  5. Delete Sensitive Data From Git
  6. Kubernetes 101
  7. Hive GC Overhead Limit Exceeded
  8. Python Resource Limitation
  9. Why Use `pip install --user`?
  10. Table Selection in Software Engineering
  11. Spark History Server Deep Dive
  12. Mastering Database Design: Normalization Explained
  13. Data Science. Exploratory Data Analysis
  14. Data Science. The Central Limit Theorem and Sampling
  15. Data Science. Correlation
  16. Asynchronous Programming. Python3.5+
  17. Asynchronous Programming. Cooperative Multitasking
  18. Data Science. Probability Distributions
  19. Data Science. Probability
  20. Comparing Dgraph and Neo4j Graph Databases: Key Differences and Use Cases
  21. How to Speed Up Spark Jobs on Small Test Datasets
  22. Exploring the Power of Graph Databases
  23. Data Science. Measures
  24. Asynchronous Programming. Threads and Processes
  25. What is the definition of a good software engineer?
  26. Ode to Unit Tests
  27. Resolve cython and numpy dependencies on setup step
  28. How to estimate time for a project/task accurately
  29. What Are The Best Software Engineering Principles?
  30. Hidden Pitfalls of LLM in Education
  31. Who’s Really Responsible for Team Failures?
  32. Locking Mechanisms in High-Load Systems
  33. How to Build High-Performance Engineering Teams
  34. The Importance of Clear Software Requirements
  35. Senior Engineer Fatigue
  36. Frameworks: A Developer's Dilemma
  37. From RAGs to Riches: Hands-on Retrieval-Augmented Generation
  38. From RAGs to Riches: An In-Depth Look at Retrieval-Augmented Generation
  39. Choosing the Right AWS Storage Service: A Comprehensive Guide to S3, S3N, and S3A
  40. Building a Bloom filter
  41. Spark Tips. Optimizing JDBC data source reads
  42. Cunningham's Law: The sun is flat, isn’t it?
  43. Quality is the responsibility of the team
  44. EMR Serverless a 400-level guide
  45. A very quick introduction to the pain of AWS CloudFormation
  46. First rule of the leader - grow people
  47. Interning in CPython
  48. Spark Tips. Caching
  49. Things to consider while running Google Cloud Dataproc
  50. What is Serverless Architecture and what are its benefits?
  51. HDFS vs Cloud-based Object storage(S3)
  52. Machine Learning types
  53. Architecturally Significant Requirements
  54. Data Challenges in Big Data
  55. 5 lessons learned from writing a tech book
  56. Modern Big Data Architectures - Lambda & Kappa
  57. Clean up your digital hygiene
  58. M-Motivation
  59. Big Data — Big Privacy Hole
  60. Thoughts on (micro)services
  61. Data Science vs Machine Learning
  62. Data Science. Descriptive and Inferential Statistics
  63. Data Science. Data Types
  64. How to start your blog for 20 cents
  65. 7 tips to make an effective Python Style Guide
  66. My unpopular opinion about black code formatter
  67. The ultimate Python style guidelines
  68. Introduction to Pyspark join types
  69. The 5-minute guide to using bucketing in Pyspark
  70. Schema-on-Read vs Schema-on-Write
  71. Data Science. Demystifying Hypothesis Testing
  72. Data Science. Bayes Theorem
  73. 11 steps of Scrum
  74. Concurrency and Parallelism are two different things
  75. Optional arguments MUST use keywords (Python3)
  76. Continuous Integration & Delivery main ideas
  77. Practical Guide to Navigating Tech Job Market: Interview Prep
  78. Remote AWS Certification Exam
  79. Practical Guide to Navigating Tech Job Market: Job Search and Networking
  80. Navigating Workplace Conflicts
  81. How I Became a Writer

2023 4

  1. Python Static Analysis Tools
  2. Two Archetypes of Data Engineers
  3. GitHub Pull Request Templates: Streamlining Code Reviews with Style and Efficiency
  4. Taking on the Big Guns: adapting to the fast-paced environment of Big Tech

2022 3

  1. Concurrency vs Parallelism
  2. Making CI workflow faster with Github Actions
  3. Rumbling about Test Driven Development

2021 7

  1. AWS Lambda abuse
  2. Get Hive count in seconds
  3. Management Challenges in Big Data
  4. Operational Challenges in Big Data
  5. Analytical Challenges in Big Data
  6. Soft Skills guide for Software Engineer
  7. Guidelines for business meetings

2020 7

  1. Airflow dag dependencies
  2. Data Engineering skills
  3. Spark Tips. Don't collect data on driver
  4. How to not leap in time using Python
  5. Spark Tips. Use DataFrame API
  6. Asynchronous Programming. Blocking I/O and non-blocking I/O
  7. __context__ vs __cause__ attributes in exception handling

2019 1

  1. Asynchronous Programming. Await the Future

2017 1

  1. I started (again) my blog