Compression has always been a cumbersome task in VB. Microsoft provides very weak support in their standard libraries, and while there are good 3rd-party solutions for VB developers, they are either expensive (http://www.dart.com/zip-activex-library-api.aspx) or very difficult to use correctly (http://www.7-zip.org/sdk.html).
So for many years, VB6 developers have fallen back on the classic zLib compression library (http://zlib.net/). zLib is an open-source compression library with a very permissive license, and it is "good enough" for most tasks: decent compression ratios, but with relatively slow compression and decompression speeds.
But in recent years, even zLib has become problematic for VB6 users. The stdcall variant of zLib hasn't been updated in over a decade, and it contains serious known security bugs. You can always compile your own version of zLib from the latest source code, but the core library definitions are bugged, so this requires a fairly deep knowledge of C and a lot of patience. (Also, zLib's source code hasn't been updated in over three years, and there are a huge number of bug fixes that have yet to be incorporated.)
And even if you do manage to survive all this and successfully build a recent version of zLib, you're still left with compression technology that is 20+ years old. A ton of compression research has been done since 1995 (when zLib first released), and we now have libraries that are both much faster, and with even better compression ratios.
So here's what this small project does: it provides a small "Compression" module that wraps four different open-source compression libraries: zLib, zstd, lz4, and lz4_hc. The compression/decompression functions are unified so you simply call a function like "Compress", and pass a "compression library enum" that specifies which compression engine you want to use.
To simplify this demo, precompiled DLLs are provided for each library. Because these are all based off open-source projects (links to code below), I believe these still meet the vbforums requirements for precompiled binaries. You are of course free to compile these yourself, from the latest source code, but you will need a modern copy of Visual Studio, some knowledge of compiling C code, and you must manually modify the project files to build stdcall variants. (They all default to cdecl, as-is.)
These are all bare C libraries, so they do not need to be registered on target PCs. Simply ship them in a subfolder of your project - for example, this demo project uses a "\Plugins\" subfolder, and the DLLs are all loaded at run-time via LoadLibrary.
Here is a brief overview of the provided compression libraries, all of which are 100% open-source and free to use in personal or commercial projects (with attribution - see the included license files for details).
- zLib is the classic library you know and love. I've freshly compiled the newest build (v1.2.8) for this demo. Despite its age, zLib remains a solid general-purpose compression library, with good compression ratios across a wide variety of data, but with slow compression speeds compared to the competition. zLib supports a "compression level" parameter that allows you to choose a trade-off between faster but worse compression, or slower but better compression. Generally speaking, there is no longer much reason to use zLib, unless you specifically need the DEFLATE algorithm it provides (e.g. to work with .gz files).
- zstd (or "zstandard") is a modern replacement for zLib. It was originally developed by Yann Collet, and its ongoing development is now sponsored by Facebook. It is 100% open-source and BSD licensed. zstd is significantly faster than zLib at both compression and decompression, and it also achieves better compression ratios. It provides a "compression level" parameter just like zLib, but with a much wider range, including extremely slow speeds but extremely good compression ratios if you need that sort of thing. For most users, zstd could replace zLib in their existing projects, and they'd immediately get a "free" performance boost from it.
- lz4 is a real-time compression engine that emphasizes performance above all else. It was also developed by Yann Collet, and it is also 100% open-source and BSD licensed. lz4 is so fast that it is now used for OS-level compression (Linux), file system compression (OpenZFS, SquashFS), database compression (MySQL), RAM caching (Emscripten, ZRam), and a whole bunch of video games (Battlefield 4, Black Ops 3, etc). LZ4's speed comes at a trade-off, however - it does not compress as well as zLib or zstd on most data. It also provides an adjustable "compression level" parameter, but instead of providing "slower but better" compression as you increase this value, lz4 provides "faster but worse" compression. It is the best solution when speed is paramount. (For example, lz4 is one of the few algorithms fast enough to provide a performance benefit vs raw uncompressed data when reading/writing to a hard drive.)
- lz4_hc comes "for free" with lz4. It is a "high-compression" variant of lz4, with much better compression ratios but much slower compression speeds. Decompression speed remains the same. It is a good solution if you have all the time in the world for compression, but you still require very fast decompression. (This is the version that video games use, for example.)
The included demo project allows you to compare compression speed, decompression speed, and compression ratio across all libraries. A baseline comparison of "no compression" is also provided, which measures timing against bare RtlMoveMemory calls. I've included a few multilanguage XML files for comparison (because they're small enough to fit inside vbforum size limits), but for best results, you should test some of your own files. Just drag-and-drop a file onto the project window to run an automated test across all libraries.
Note that - by design - the Compression module operates entirely on byte arrays and/or bare pointers (passed using VarPtr()). This makes it trivial to compress source data of any size or type. Specialized functions for Strings or other data types could always be added, but for now, those are left as an exercise to the reader.
Bug reports and feedback welcome, of course.
Download here:
Compression.zip
So for many years, VB6 developers have fallen back on the classic zLib compression library (http://zlib.net/). zLib is an open-source compression library with a very permissive license, and it is "good enough" for most tasks: decent compression ratios, but with relatively slow compression and decompression speeds.
But in recent years, even zLib has become problematic for VB6 users. The stdcall variant of zLib hasn't been updated in over a decade, and it contains serious known security bugs. You can always compile your own version of zLib from the latest source code, but the core library definitions are bugged, so this requires a fairly deep knowledge of C and a lot of patience. (Also, zLib's source code hasn't been updated in over three years, and there are a huge number of bug fixes that have yet to be incorporated.)
And even if you do manage to survive all this and successfully build a recent version of zLib, you're still left with compression technology that is 20+ years old. A ton of compression research has been done since 1995 (when zLib first released), and we now have libraries that are both much faster, and with even better compression ratios.
So here's what this small project does: it provides a small "Compression" module that wraps four different open-source compression libraries: zLib, zstd, lz4, and lz4_hc. The compression/decompression functions are unified so you simply call a function like "Compress", and pass a "compression library enum" that specifies which compression engine you want to use.
To simplify this demo, precompiled DLLs are provided for each library. Because these are all based off open-source projects (links to code below), I believe these still meet the vbforums requirements for precompiled binaries. You are of course free to compile these yourself, from the latest source code, but you will need a modern copy of Visual Studio, some knowledge of compiling C code, and you must manually modify the project files to build stdcall variants. (They all default to cdecl, as-is.)
These are all bare C libraries, so they do not need to be registered on target PCs. Simply ship them in a subfolder of your project - for example, this demo project uses a "\Plugins\" subfolder, and the DLLs are all loaded at run-time via LoadLibrary.
Here is a brief overview of the provided compression libraries, all of which are 100% open-source and free to use in personal or commercial projects (with attribution - see the included license files for details).
- zLib is the classic library you know and love. I've freshly compiled the newest build (v1.2.8) for this demo. Despite its age, zLib remains a solid general-purpose compression library, with good compression ratios across a wide variety of data, but with slow compression speeds compared to the competition. zLib supports a "compression level" parameter that allows you to choose a trade-off between faster but worse compression, or slower but better compression. Generally speaking, there is no longer much reason to use zLib, unless you specifically need the DEFLATE algorithm it provides (e.g. to work with .gz files).
- zstd (or "zstandard") is a modern replacement for zLib. It was originally developed by Yann Collet, and its ongoing development is now sponsored by Facebook. It is 100% open-source and BSD licensed. zstd is significantly faster than zLib at both compression and decompression, and it also achieves better compression ratios. It provides a "compression level" parameter just like zLib, but with a much wider range, including extremely slow speeds but extremely good compression ratios if you need that sort of thing. For most users, zstd could replace zLib in their existing projects, and they'd immediately get a "free" performance boost from it.
- lz4 is a real-time compression engine that emphasizes performance above all else. It was also developed by Yann Collet, and it is also 100% open-source and BSD licensed. lz4 is so fast that it is now used for OS-level compression (Linux), file system compression (OpenZFS, SquashFS), database compression (MySQL), RAM caching (Emscripten, ZRam), and a whole bunch of video games (Battlefield 4, Black Ops 3, etc). LZ4's speed comes at a trade-off, however - it does not compress as well as zLib or zstd on most data. It also provides an adjustable "compression level" parameter, but instead of providing "slower but better" compression as you increase this value, lz4 provides "faster but worse" compression. It is the best solution when speed is paramount. (For example, lz4 is one of the few algorithms fast enough to provide a performance benefit vs raw uncompressed data when reading/writing to a hard drive.)
- lz4_hc comes "for free" with lz4. It is a "high-compression" variant of lz4, with much better compression ratios but much slower compression speeds. Decompression speed remains the same. It is a good solution if you have all the time in the world for compression, but you still require very fast decompression. (This is the version that video games use, for example.)
The included demo project allows you to compare compression speed, decompression speed, and compression ratio across all libraries. A baseline comparison of "no compression" is also provided, which measures timing against bare RtlMoveMemory calls. I've included a few multilanguage XML files for comparison (because they're small enough to fit inside vbforum size limits), but for best results, you should test some of your own files. Just drag-and-drop a file onto the project window to run an automated test across all libraries.
Note that - by design - the Compression module operates entirely on byte arrays and/or bare pointers (passed using VarPtr()). This makes it trivial to compress source data of any size or type. Specialized functions for Strings or other data types could always be added, but for now, those are left as an exercise to the reader.
Bug reports and feedback welcome, of course.
Download here:
Compression.zip