The Curious Engineer

All blogs / AI / ML

Giles' blog

gilesthomas.com

Currently mostly AI because in 2026 of course it is. Grinding through the nuts and bolts of LLMs and trying to work out why this stuff works...

AI / MLPython

’10’15’20’25
Posts
372 from 1 Nov 2006 to 23 Oct 2026
Posting
weekly (64 in the last year)
Kind
personal blog
Feed
https://www.gilesthomas.com/feed/rss.xml

2026 58

  1. RMSNorm
  2. Fun with low-rank vocab matrices (and a bonus test loss reduction?)
  3. Why do OpenAI's GPT-2 weights beat mine? Part five: data quality
  4. October 2026
  5. Extending Raschka's GPT-2: an MoE trained from scratch on an RTX 3090
  6. Putting my JAX-trained models on the Hugging Face Hub
  7. September 2026
  8. Why do OpenAI's GPT-2 weights beat mine? Part four: digging into dropout
  9. Adding diagrams to my static site generator with D2
  10. Use the built-in GELU, don't roll your own!
  11. A quick(ish) Chinchilla check
  12. August 2026
  13. How I use AI on this blog
  14. July 2026
  15. Writing an LLM from scratch, part 34b -- from bigrams to GPT-2, one component at a time (in JAX)
  16. 39 million of them
  17. calculated
  18. overtrained a couple of GPT-2 style models
  19. not as good at an instruction-fine-tuning test
  20. [ Read more ]
  21. [ Read more ]
  22. June 2026 (7)
  23. [ Read more ]
  24. [ Read more ]
  25. [ Read more ]
  26. [ Read more ]
  27. [ Read more ]
  28. [ Read more ]
  29. [ Read more ]
  30. May 2026 (2)
  31. On first looking into jax
  32. [ Read more ]
  33. April 2026 (11)
  34. more details here
  35. an equivalent PyTorch model
  36. [ Read more ]
  37. in the cloud
  38. [ Read more ]
  39. [ Read more ]
  40. [ Read more ]
  41. [ Read more ]
  42. [ Read more ]
  43. [ Read more ]
  44. Automating starting Lambda Labs instances
  45. March 2026 (3)
  46. cost in quality
  47. [ Read more ]
  48. [ Read more ]
  49. February 2026 (4)
  50. tiny improvement in loss terms
  51. interventions
  52. adding gradient clipping
  53. removing dropout
  54. January 2026 (4)
  55. existing PyTorch script
  56. Llm from scratch 30 digging into llm as a judge
  57. a quick-and-dirty bit of curve-fitting
  58. [ Read more ]

2025 64

  1. December 2025 (1)
  2. four times longer
  3. Why smart instruction-following makes prompt injection easier
  4. November 2025 (3)
  5. set myself a number of follow-on goals
  6. [ Read more ]
  7. October 2025 (9)
  8. rebuilding Andrej Karpathy's 2015-vintage RNNs
  9. Llm from scratch 24 the transcript hack
  10. [ Read more ]
  11. [ Read more ]
  12. A classifier using Qwen3
  13. [ Read more ]
  14. [ Read more ]
  15. post on cross entropy
  16. [ Read more ]
  17. September 2025 (3)
  18. How do LLMs work?
  19. The maths you need to start understanding LLMs
  20. An addendum to 'the maths you need to start understanding LLMs'
  21. August 2025 (5)
  22. the feed-forward network
  23. Writing an LLM from scratch, part 19 -- wrapping up Chapter 4
  24. What AI chatbots are actually doing under the hood
  25. [ Read more ]
  26. [ Read more ]
  27. July 2025 (1)
  28. LayerNorm
  29. June 2025 (2)
  30. sabbatical
  31. [ Read more ]
  32. May 2025 (3)
  33. Back in May
  34. [ Read more ]
  35. [ Read more ]
  36. April 2025 (2)
  37. not all that hard to add that on
  38. [ Read more ]
  39. March 2025 (7)
  40. dropout
  41. part 8 of this series
  42. Dropout and mandatory vacation
  43. causal attention
  44. [ Read more ]
  45. [ Read more ]
  46. [ Read more ]
  47. February 2025 (10)
  48. learn in public
  49. an article I wrote back in February
  50. last update
  51. [ Read more ]
  52. LaTeX working on this site
  53. [ Read more ]
  54. blogged
  55. [ Read more ]
  56. [ Read more ]
  57. Blog design update
  58. January 2025 (6)
  59. taken from us before his time
  60. part 6 in this series
  61. Writing an LLM from scratch, part 6b -- a correction
  62. Do reasoning LLMs need their own Philosophical Language?
  63. [ Read more ]
  64. Happy New Year!

2024 25

  1. December 2024 (7)
  2. the start of this series
  3. token and position embedding mappings
  4. some of my lab notes posts
  5. my last post in this series
  6. [ Read more ]
  7. [ Read more ]
  8. [ Read more ]
  9. September 2024 (1)
  10. [ Read more ]
  11. August 2024 (2)
  12. [ Read more ]
  13. [ Read more ]
  14. July 2024 (2)
  15. Done some initial experiments into memory usage for a 0.5B model
  16. last post
  17. May 2024 (2)
  18. Run some multi-GPU training
  19. successful fine-tune of the 8B model
  20. April 2024 (2)
  21. the start of these experiments
  22. Done the same, but in the cloud
  23. February 2024 (2)
  24. [ Read more ]
  25. Giving up on the AI chatbot tutorial (for now)

2023 5

  1. April 2023 (1)
  2. [ Read more ]
  3. March 2023 (2)
  4. I first started looking into LLMs
  5. Ai llm bot beginners tutorial 00

2022 4

  1. September 2022 (1)
  2. sold the business
  3. February 2022 (1)
  4. A somewhat indirect way of reporting stolen cards to the bank

2021 7

  1. November 2021 (1)
  2. [ Read more ]
  3. March 2021 (1)
  4. [ Read more ]
  5. February 2021 (2)
  6. my new static site generator
  7. Comments are back!

2019 2

  1. August 2019 (1)
  2. Installing the unifi controller on Arch

2018 2

  1. November 2018 (1)
  2. [ Read more ]

2017 2

  1. May 2017 (1)
  2. Creating a time series from existing data in pandas

2016 4

  1. December 2016 (1)
  2. Parsing website SSL certificates in Python
  3. April 2016 (1)
  4. process namespace

2015 2

  1. August 2015 (1)
  2. [ Read more ]

2014 6

  1. December 2014 (1)
  2. Does #EUVAT make accepting bitcoins impossible for EU-based digital services businesses?
  3. August 2014 (1)
  4. An HTTP request's journey through a platform-as-a-service
  5. March 2014 (1)
  6. A fun bug

2013 20

  1. December 2013 (1)
  2. ...just resting...
  3. October 2013 (3)
  4. [ Read more ]
  5. ...and another sidetrack -- a new theme!
  6. A brief sidetrack: Varnish
  7. September 2013 (4)
  8. the second part
  9. the third part
  10. There's more about the bug here
  11. Writing a reverse proxy/loadbalancer from the ground up in C, pause to regroup: fixed it!
  12. August 2013 (2)
  13. more background here
  14. Writing a reverse proxyloadbalancer from the ground up in c part 1
  15. July 2013 (1)
  16. [ Read more ]
  17. June 2013 (1)
  18. How many Python programmers are there in the world?
  19. February 2013 (1)
  20. A super-simple chat app with AngularJS, SockJS and node.js

2012 10

  1. October 2012 (1)
  2. Reverse proxying HTTP and WebSockets with virtual hosts using nginx and tcp_proxy_module
  3. June 2012 (1)
  4. [ Read more ]
  5. May 2012 (1)
  6. Running Django unit tests on PythonAnywhere
  7. April 2012 (1)
  8. New business idea
  9. February 2012 (1)
  10. PythonAnywhereAnywhere

2011 14

  1. October 2011 (1)
  2. Teaching programming
  3. June 2011 (1)
  4. If programming languages were literary genres...
  5. May 2011 (1)
  6. Resolver is hiring
  7. April 2011 (1)
  8. time to pivot again
  9. March 2011 (1)
  10. How to bet on the bubble?
  11. February 2011 (1)
  12. Some old JavaScript
  13. January 2011 (1)
  14. London Financial User Group Meeting: 17 January

2010 32

  1. December 2010 (3)
  2. And the same to you too, Google!
  3. [ Read more ]
  4. Fun with the Audio Data API
  5. November 2010 (1)
  6. A Dirigible screencast
  7. October 2010 (1)
  8. Dirigible
  9. September 2010 (1)
  10. [ Read more ]
  11. August 2010 (1)
  12. London Financial User Group Meeting: September 15
  13. July 2010 (1)
  14. Bare Git repositories
  15. May 2010 (3)
  16. [ Read more ]
  17. [ Read more ]
  18. [ Read more ]
  19. April 2010 (1)
  20. Regular expressions and Resolver One column-level formulae
  21. London Financial Python Users' Group
  22. March 2010 (2)
  23. An aside: SEO for restaurants
  24. [ Read more ]
  25. Next London Financial Python Users Group meeting
  26. February 2010 (3)
  27. Playing with NLTK
  28. [ Read more ]
  29. January 2010 (4)
  30. New laptop!
  31. Joining TheyWorkForYou to Twitter
  32. [ Read more ]

2009 43

  1. December 2009 (2)
  2. New gadget: Nokia N900
  3. A website for LFPUG
  4. November 2009 (5)
  5. [ Read more ]
  6. Shiny new blog theme
  7. New York Financial Users Group
  8. London Financial Python Users Group
  9. New splogging technique?
  10. October 2009 (2)
  11. at the end of October
  12. Webgl
  13. September 2009 (2)
  14. [ Read more ]
  15. [ Read more ]
  16. August 2009 (3)
  17. [ Read more ]
  18. hExcel -- A Hexagonal Spreadsheet
  19. Clicking the tabs from left to right
  20. July 2009 (1)
  21. A Resolver One model on the FT politics blog
  22. May 2009 (1)
  23. Talk at London Geek Night
  24. April 2009 (1)
  25. [ Read more ]
  26. March 2009 (5)
  27. Ada Lovelace day
  28. Ssds
  29. One-day discount
  30. Click-through ratios
  31. R in Resolver One (and perhaps IronPython generally)
  32. February 2009 (5)
  33. [ Read more ]
  34. Usability testers needed
  35. Praying that this isn't a hoax...
  36. Xmlrpc
  37. Good music composition software
  38. January 2009 (5)
  39. How much we decided to charge for our software
  40. how much we should charge for our software
  41. Spam subject line of the day
  42. The Resolver One Spreadsheet Challenge: a winner for round one!
  43. Resolver Systems competition closing soon

2008 35

  1. December 2008 (3)
  2. Money for spreadsheets
  3. [ Read more ]
  4. [ Read more ]
  5. November 2008 (7)
  6. to start going bust
  7. [ Read more ]
  8. Ironclad 0.7 released
  9. Resolver One plug
  10. Affective robots
  11. Best. Video. Ever.
  12. Book meme
  13. October 2008 (4)
  14. Evolution in action
  15. Recovered!
  16. I came for the article, I stayed for the comments
  17. Why use IronPython?
  18. September 2008 (2)
  19. A bit of fun
  20. Off to BoS
  21. August 2008 (1)
  22. Resolver One as a Python Success Story
  23. July 2008 (1)
  24. Workaround for Vista stupidity
  25. June 2008 (1)
  26. A Thinking Ape's Critique of Trans-Simianism
  27. May 2008 (1)
  28. Making a fool of yourself in public
  29. April 2008 (1)
  30. Building a collection of classical music
  31. January 2008 (4)
  32. Off to visit the Beast of Redmond ;-)
  33. Why should the government fund space exploration?
  34. Dear lazyweb: what is is about Linux and WPA?
  35. New gadget!

2007 13

  1. December 2007 (3)
  2. Eee, day 2
  3. Eee pc
  4. Screencast
  5. March 2007 (3)
  6. Patronising messages
  7. View from a launched rocket
  8. HTML tattoo
  9. February 2007 (1)
  10. Feelix Growing
  11. January 2007 (2)
  12. 'Dancing mule'
  13. Building the coat hanger walker

2006 24

  1. December 2006 (4)
  2. Another robot
  3. Make:06 Trimet... hmmm.
  4. Installing Asterisk
  5. New project: VoIP
  6. MSBuild WTF: 'The error was:'
  7. November 2006 (18)
  8. Christmas has come early
  9. Back again
  10. [ Read more ]
  11. Part 1
  12. Part 2
  13. Part 3
  14. Part 4
  15. Part 5
  16. Part 6
  17. replace
  18. getting Ruby up and running
  19. [ Read more ]
  20. [ Read more ]
  21. [ Read more ]
  22. [ Read more ]
  23. Ipv6
  24. Hello, world!