Building a faster LZ4 with Intel® Integrated Performance Primitives

Intel® Integrated Performance Primitives (Intel® IPP) provides optimization for LZ4 - open source lossless data compression library. The data compression domain of the Intel® IPP library contains several functions which could speed up LZ4 data compression method for both data compression and decompression operations. In this article, the process of building LZ4 program with Intel® IPP patched library will be discussed. An simple case is also provided to measure the performance by using Intel® IPP library for LZ4 compression/decompression functions.

Intel® IPP as a component is available as part of the Intel® oneAPI Base Toolkit.

Build LZ4 with Intel® IPP on Linux*/macOS*

Intel® IPP provides LZ4 patch file, archived at the <ipp directory>/components/ directory. The archive contains the patch file (saved in $IPPROOT/components/components/interfaces/ipp_lz4) of drop-in optimization with Intel® IPP functions for the LZ4 source code. Before building the library, you need to download v1.9.2.tar.gz file from LZ4 site and un-archive compressed file to working directory, for instance, mine is $HOME/IPP_DC. 

tar -xvzf v1.9.2.tar.gz

Patching Source Code

First, set up Intel® IPP environment with following commands:

source /opt/intel/compilers_and_libraries_<version.update.build>/linux/ipp/bin/ippvars.sh <ia32|intel64>

Patch the source code using the following commands (it is assumed that lz4-1.9.2 patch file is in the working directory):

cd lz4-1.9.2
patch -p1 < ../lz4-1.9.2.patch.bin
patching file examples/Makefile
patching file lib/dll/example/Makefile
patching file lib/lz4.c
patching file lib/Makefile
...

Building LZ4 Library with Intel® IPP

After LZ4 source code files are patched successfully, you need to set building environment for "make" procedure. Now, you are staying in lz4-1.9.2 folder, before make lz4 libraries, you need to

• Add the WITH_IPP definition to compiler command line;
• Add the $IPPROOT/include directly to the list of header files search for compiler;
• Makefiles for building of library, LZ4 utilities and tests are modified during patch process to include Intel® IPP library files. This is done, because for LZ4 makefiles are not based on LDFLAGS environment variable.

Build the library using standard make utility with the following preliminary actions:

export CFLAGS="-O3 -DWITH_IPP -I$IPPROOT/include"
# Build the library and LZ4 tool
make WITH_IPP=yes ARCH=intel64

If you want to build LZ4 for IA-32 architecture specify "ARCH=ia32" in the "make" command line.

Build LZ4 with Intel® IPP on Windows*

To download the LZ4 source (.zip) for Widows* from LZ4 site. However the patching and building processes on Windows* are different. On Windows, you could use some unzip tool like 7-zip to inflate the zip file. 

Patching Source Code

There are no standard system tools for patching on Windows* platform, you need to use 3-rd party tool. For example, GnuWin32 from Free Software Foundation. Open Command Prompt Window, with the GnuWin32 patch tool, use following command line to process:

> "C:\Program Files (x86)\GnuWin32\bin\patch.exe" -p1 --binary < <path-of-patch-file>/lz4-1.9.2.patch.bin

*Please note: "--binary" option here is important to avoid confusion because of Linux*/Windows* "end-of-line" character difference.

Building LZ4 With Intel® IPP

To build LZ4 library and tools on Windows*, there is a Microsoft* Visual Studio* solution file and project files in lz4-1.9.2\visual\VS2017 directory. 

> ./lz4 -b tests/COPYING
  1#COPYING           :     18092 ->     10588 (1.709), 603.1 MB/s ,3015.3 MB/s