The Curious Engineer

All blogs / Databases

Tech Blog

tech.marksblogg.com

Benchmarks & Tips for Big Data, Hadoop, AWS, Google Cloud, PostgreSQL, Spark, Python & More...

DatabasesSciencePython

’15’20’25
Posts
213 from 28 Oct 2014 to 16 Sep 2026
Posting
weekly (26 in the last year)
Kind
personal blog
Feed
https://tech.marksblogg.com/feeds/all.atom.xml

2026 21

  1. Saving Jet Fuel
  2. Planet Labs' Open Satellite Feed
  3. ICEYE's Open Satellite Feed
  4. Tasking RADARSAT
  5. Hydropower Stations in China in 2026
  6. Flight Planning with Little Navmap
  7. Vantor's Open Satellite Feed
  8. Optimal Seating on the Airbus A380
  9. Open Source Aviation Maps
  10. 1.75B Airport Weather Observations
  11. 12K+ JPEGs from NASA's Artemis II Mission
  12. e-Bike Fleet Monitoring
  13. 10K+ Satellites in Space
  14. Reverse Geocoding with Overture Maps
  15. 3.4M Solar Panels
  16. French Mobile Network Datasets
  17. Canadian Wind Farms
  18. Google Street View in 2026
  19. Data Crew's Route Optimiser Framework
  20. 72M Points of Interest
  21. Microsoft's 2026 Global ML Building Footprints

2025 25

  1. Millions of Locations for Thousands of Brands
  2. Better Boundaries
  3. American Data Centers
  4. American Wind Farms
  5. 2.7 Billion Buildings
  6. Australian Coastline Imagery
  7. American Solar Farms
  8. A Global Mining Dataset
  9. Canada's 14M Buildings
  10. Analysis-Ready OpenStreetMap
  11. Canada's 13M Buildings
  12. The World's 2.75B Buildings
  13. Alberta's Pipelines
  14. Planet Labs' Hyperspectral Imagery
  15. Overture Maps in ArcGIS Pro
  16. The iPhone 15 Pro's Depth Maps
  17. Apple's DepthPro on Maxar's Imagery
  18. Better Building Footprints
  19. Esri's ArcGIS Pro 3.5
  20. Satellites Spotting Depth
  21. Turning Videos into 3D Worlds
  22. GeoDeep's AI Detection on Maxar's Satellite Imagery
  23. Canada's 15.8M Addresses
  24. Wyvern's Open Satellite Feed
  25. Satellogic's Open Satellite Feed

2024 26

  1. Overture Maps' Refreshed Global Geospatial Datasets
  2. Smaller Satellite Images
  3. Language Translation with Python
  4. Foursquare's 104M Points of Interest
  5. OpenStreetMap's New Vector Tiles
  6. 131M American Buildings
  7. Land Cover from Satellite Imagery
  8. Microsoft's 1.4 Billion Global ML Building Footprints
  9. ICEYE's Satellite Imagery
  10. Japan's Building Footprints
  11. Satellites Spotting Aircraft
  12. Baltic Maritime Traffic Feed
  13. AI on Street View
  14. Capella's Open Satellite Feed
  15. Revisiting Overture's Global Geospatial Datasets
  16. AI-Extracted Asian Building Footprints
  17. Satellites Spotting Ships
  18. Maxar's HD Global Imagery Basemap
  19. HeavyIQ: Understanding 220M Flights with AI
  20. Minimalist Guide to AV1 Video Encoding
  21. Umbra's Open Satellite Feed
  22. Global EV Charging Points with Open Charge Map
  23. Aircraft Route Analysis
  24. 1.1 Billion Taxi Rides using ClickHouse on Intel's Core i9-14900K
  25. 1.1 Billion Taxi Rides using DuckDB
  26. Tokyo Walking Tour Guide

2023 20

  1. Extracting OSM Features
  2. Global Flight Tracking
  3. Mapping Estonia with LiDAR
  4. Natural Earth's Global Geospatial Datasets
  5. Maxar's Open Satellite Feed
  6. Overture's Global Geospatial Datasets
  7. A Review of Esri's Imagery in Action MOOC
  8. Versatile Video Coding
  9. Segmenting Satellite Images
  10. A Review of Esri's Spatial Data Science MOOC
  11. Enhancing ClickHouse's Geospatial Support
  12. Asking a Large Language Model How YouTube Works
  13. Geospatial Clustering with Uber's H3 in DuckDB & QGIS
  14. Popular Airline Passenger Routes Refresh
  15. Streaming Video
  16. IPinfo's Free IP Address Location Database
  17. DuckDB's Spatial Extension
  18. Geospatial DuckDB
  19. European Route Planning
  20. Faster PostgreSQL To BigQuery Transfers

2022 15

  1. 1.1 Billion Taxi Rides in ClickHouse on DoubleCloud
  2. Awesome Isochrones
  3. ECharts for Python
  4. Python Data Visualisation
  5. Hardening SSH
  6. Pretty Maps in Python
  7. Making Heatmaps
  8. Minimalist Guide to Poem
  9. Minimalist Guide to Axum
  10. File Sharing with Caddy & MinIO
  11. Deploying 5G Around Trees
  12. The Streets of Monaco
  13. Install ClickHouse Faster
  14. Faster Geospatial Enrichment
  15. Where is every IP Address?

2021 14

  1. Faster Top Level Domain Name Extraction with Go
  2. The Fastest FizzBuzz Implementation
  3. ROAPI: An API Server for Static Datasets
  4. Actix: A Web Framework for Rust
  5. Rocket: A Web Framework for Rust
  6. Building PostgreSQL Extensions with Rust
  7. Faster Top Level Domain Name Extraction with Rust
  8. Track changes in Excel, Word, PowerPoint, PDFs, Images & Videos with Git
  9. Faster Compression with Snappy's S2 Extension
  10. MeiliSearch: A Minimalist Full-Text Search Engine
  11. MinIO: A Bare Metal Drop-In for AWS S3
  12. Monitor ClickHouse with Prometheus & Grafana
  13. Data Fluent for PostgreSQL
  14. 1.1 Billion Taxi Rides using Hydrolix on AWS

2020 3

  1. 1.1 Billion Taxi Rides using OmniSciDB and a MacBook Pro
  2. Python Web Scraping with Virtual Private Networks
  3. Fast IPv4 to Host Lookups

2019 13

  1. Benchmarks
  2. Faster ZIP Decompression
  3. Faster ClickHouse Imports
  4. YouTube's Database "Procella"
  5. Is Hadoop Dead?
  6. LZ77 algorithm
  7. Faster File Distribution with HDFS and S3
  8. A Minimalist Guide to Flume
  9. A Minimalist Guide to FoundationDB
  10. "Architecting Modern Data Platforms" Book Review
  11. 1.1 Billion Taxi Rides: 108-core ClickHouse Cluster
  12. Convert CSVs to ORC Faster
  13. 1.1 Billion Taxi Rides: Spark 2.4.0 versus Presto 0.214

2018 10

  1. Working with the Hadoop Distributed File System
  2. Systems Monitoring: top vs Htop vs Glances
  3. Working with Data Feeds
  4. A Minimalist Guide to Microsoft SQL Server 2017 on Ubuntu Linux
  5. 1.1 Billion Taxi Rides with SQLite, Parquet & HDFS
  6. Customising Airflow: Beyond Boilerplate Settings
  7. Using SQL to query Kafka, MongoDB, MySQL, PostgreSQL and Redis with Presto
  8. Python & Big Data: Airflow & Jupyter Notebook with Hadoop 3, Spark & Presto
  9. 1.1 Billion Taxi Rides: EC2 versus EMR
  10. Hadoop 3 Single-Node Install Guide

2017 15

  1. 1.1 Billion Taxi Rides with BrytlytDB 2.1 & a 5-node IBM Minsky Cluster
  2. 1.1 Billion Taxi Rides with BrytlytDB 2.0 & 2 GPU-Powered p2.16xlarge EC2 Instances
  3. A Minimalist Guide to SQLite
  4. 1.1 Billion Taxi Rides with Spark 2.2 & 3 Raspberry Pi 3 Model Bs
  5. 1.1 Billion Taxi Rides with BrytlytDB & 2 GPU-Powered p2.16xlarge EC2 Instances
  6. Compiling MapD's Source Code
  7. 1.1 Billion Taxi Rides with MapD 3.0 & 2 GPU-Powered p2.8xlarge EC2 Instances
  8. Detecting Bots in Apache & Nginx Logs
  9. Doom Bots in TensorFlow
  10. Analysing Petabytes of Websites
  11. A Review of "Designing Data-Intensive Applications"
  12. 1.1 Billion Taxi Rides on ClickHouse & an Intel Core i5
  13. 1.1 Billion Taxi Rides on Vertica & an Intel Core i5
  14. 1.1 Billion Taxi Rides on AWS EMR 5.3.0 & Spark 2.1.0
  15. 1.1 Billion Taxi Rides on kdb+/q & 4 Xeon Phi CPUs

2016 32

  1. 1.1 Billion Taxi Rides on Amazon Athena
  2. Alenka: A GPU-Driven, Open Source Database
  3. 1.1 Billion Taxi Rides with MapD & 8 Nvidia Pascal Titan Xs
  4. TensorFlow on a GTX 1080
  5. Building a Data Pipeline with Airflow
  6. 1.1 Billion Taxi Rides with MapD & AWS EC2
  7. 1.1 Billion Taxi Rides with MapD & 4 Nvidia Titan Xs
  8. 1.1 Billion Taxi Rides with MapD & 8 Nvidia Tesla K80s
  9. 1.2 Billion Taxi Rides on AWS RDS running PostgreSQL
  10. 1.1 Billion Taxi Rides on a Large Redshift Cluster
  11. All 1.1 Billion Taxi Rides on Redshift
  12. All 1.1 Billion Taxi Rides in Elasticsearch
  13. 50-node Presto Cluster on Google Cloud's Dataproc
  14. Performance Impact of File Sizes on Presto Query Times
  15. Faster IPv4 WHOIS Crawling
  16. 33x Faster Queries on Google Cloud's Dataproc
  17. Mass IP Address WHOIS Collection with Django & Kafka
  18. A Billion Taxi Rides: AWS S3 versus HDFS
  19. A Billion Taxi Rides on Google's Dataproc running Presto
  20. 50-node Presto Cluster on Amazon EMR
  21. A Billion Taxi Rides on Google's BigQuery
  22. Bulk IP Address WHOIS Collection with Python and Hadoop
  23. A Billion Taxi Rides in PostgreSQL
  24. A Billion Taxi Rides in Elasticsearch
  25. A Billion Taxi Rides on Amazon EMR running Spark
  26. A Billion Taxi Rides on Amazon EMR running Presto
  27. Kafka Producer Latency with Large Topic Counts
  28. A Billion Taxi Rides in Hive & Presto
  29. A Billion Taxi Rides in Redshift
  30. Presto, Parquet & Airpal
  31. A Million Songs on AWS Redshift
  32. Hadoop Up and Running

2015 6

  1. Faster Testing with RAM Drives
  2. Popular Airline Passenger Routes
  3. Recommendation Engine built using Spark and Python
  4. Tightening Django Admin Logins
  5. Linting UK Postcodes
  6. Passwords in Django

2014 13

  1. Faster Python
  2. Crushing, caching and CDN deployment in Django
  3. Better Python Package Management
  4. Load balancing Django
  5. Faster Django Testing
  6. Django exception archaeology
  7. Python's killer apps for blogging: Pelican and S3cmd
  8. Collecting all IPv4 WHOIS records in Python
  9. Former PHP developer
  10. File uploads to Amazon S3 in Django
  11. IP Address lookups using Python
  12. Django speaking JSON
  13. Querying Elasticsearch from Google App Engine