The Curious Engineer

All blogs / Algorithms / data structures

RealTime Data Compression

fastcompression.blogspot.com

Development blog on compression algorithms

Algorithms / data structuresPerformance / optimizationC

’10’15’20’25
Posts
139 from 14 Nov 2010 to 15 Mar 2019
Posting
quiet since 2019
Kind
personal blog
Feed
http://feeds.feedburner.com/FastDataCompression

2019 7

  1. Presenting XXH3
  2. Compiler-checked contracts
  3. Compile-time tests
  4. Compiler Warnings
  5. Opaque types and static allocation
  6. The type system
  7. Writing safer C code

2018 3

  1. xxHash for small keys: the impressive power of modern compilers
  2. When to use Dictionary Compression
  3. Zstandard Overview

2017 2

  1. Dealing with library version mismatch
  2. The art of designing advanced API

2016 6

  1. Zstandard v1.0 is out
  2. Specification of Zstandard compression format
  3. Zstandard reaches Candidate Compression Format
  4. Finalizing a compression format
  5. Working with streaming
  6. Compressing small data

2015 10

  1. Huffman revisited part 5 : combining multi-streams with multi-symbols
  2. Huffman revisited, Part 4 : Multi-bytes decoding
  3. Fuzz testing Zstandard
  4. Accessing unaligned memory
  5. Huffman revisited - Part 3 - Depth limited tree
  6. Huffman revisited - Part 2 : the Decoder
  7. Huffman revisited - part 1
  8. Changing course
  9. Sampling, or a faster LZ4
  10. Zstandard - A stronger compression algorithm

2014 18

  1. Portability woes : Endianess and Alignment (Part 2)
  2. Portability woes : Endianess and Alignment (Part 1)
  3. Counting bytes fast - little trick from FSE
  4. xxHash wider : assessing quality of a 64-bits hash function
  5. Streaming API for LZ4
  6. Taking advantage of unequalities to provide better compression
  7. Ultra-fast normalization
  8. Perfect Normalization
  9. Better normalization, for better compression
  10. FSE encoding : Mapping sub-ranges in a memory efficient way
  11. FSE encoding, Part 2
  12. FSE encoding : how it works
  13. FSE decoding : wrap up
  14. FSE : distributing symbol values
  15. FSE : Defining optimal subranges
  16. A comparison of Arithmetic Encoding with FSE
  17. Huffman, a comparison with FSE
  18. FSE decoding : how it works

2013 11

  1. Zhuff v0.9, or a first FSE application
  2. Finite State Entropy - A new breed of entropy coder
  3. Compression stream (part 3)
  4. Stream Decoding (part 2)
  5. Towards a Streaming Interface for LZ4
  6. Inter-block compression
  7. Part 3 : Inlining big functions
  8. Fighting Code Bloat - Part 2 - Inline functions
  9. Fighting code bloat (C template-style)
  10. LZ4 Frame format : Final specifications
  11. A Streaming format for LZ4

2012 12

  1. xxHash : new version
  2. Log file compression
  3. Compressed data transmission
  4. Members properties
  5. Useful compressed streaming properties
  6. Dealing with blocks side-effects
  7. Memory buffer management
  8. Selecting a Checksum algorithm
  9. Detecting errors
  10. A file container format for LZ4
  11. Probability Update
  12. Binary Arithmetic Coder

2011 46

  1. LZ4 into Hadoop-MapReduce
  2. Advanced Parsing Strategies
  3. Fast sequence comparison
  4. Zhuff get upgraded (v0.8)
  5. A format for compressed streams, part 2
  6. A format for compressed streams
  7. Progressive Hash Series : a new method to find repetitions
  8. LZ4 in commercial application
  9. Cost of accessing variables
  10. LZ4 reaches v1.0
  11. LZ4-HC : High Compression LZ4 version is now Open Sourced
  12. LZ4 - Improved performance
  13. LZ4 for Internet communications
  14. LZ4 explained
  15. ZhuffMax parsing improvements
  16. Long Range Failure
  17. ZhuffMax
  18. LZ4 source code now Opened
  19. Zhuff + MMC = Zhuff HC
  20. A look into zip
  21. Comparing compressors : new ranking system
  22. Zhuff crowned as best Speed-Oriented compression program
  23. Ross William's LZRW
  24. Best of both worlds : mixed entropy
  25. Zhuff : new version with multi-threading (v0.7)
  26. Advanced Multithreading algorithm (Huff0 v0.9)
  27. Multi-threaded entropy experiment : Huff0 v0.8
  28. Reaching system speed limits
  29. LZ4 : confirmed as world's fastest compressor
  30. Multi-threading compression available : LZ4 v0.9
  31. LZ4 on steroids : version 0.8
  32. Multi-threaded compression experiment : LZ4 version 0.7
  33. Huff0 : Early detection offers better speed (v0.7)
  34. Zhuff : new version v0.6
  35. Improved sampling for better compression
  36. A Homepage for MMC
  37. ARM Programming (part 2)
  38. LZ4 : World's fastest compressor
  39. ARM Programming (part 1)
  40. Multi-threaded compression
  41. Improving I/O : Zhuff v0.5
  42. An old friend returns : new version of Zhuff (v0.4)
  43. Entropy evaluation : small changes in benchmark corpus
  44. Range0 : new version (v0.7)
  45. Huff0 : New version (v0.6)
  46. Quick look back at Huff0 : an entropy coder analysis

2010 24

  1. MMC available as Source Code
  2. A new look at collisions
  3. Parsing Level 1 : Lazy matching
  4. MMC : Secondary Promotion : Experimental results
  5. Rzm
  6. LZ4 : New version (0.9HC)
  7. BST : First experimental results
  8. Cache-friendly search algorithms
  9. MMC last boost : Secondary sorting
  10. BST : Binary Search Tree
  11. LZ4 : New version
  12. Multiple level promotions : Experimental Results
  13. Early end optimization
  14. Multiple level promotions
  15. Fusion searches - segments & sequences fully integrated
  16. The cost of History
  17. CPU cache
  18. Cuckoo's Hash
  19. LZ4 : New version
  20. First results of fusion search
  21. Compressing the web
  22. Unramping benefits
  23. Fast note forwarding
  24. LZ4 - New version