HOWTO: Exploiting Hardware Compression in zlib with IBM z15

This blog posting details how to take advantage of the hardware compression feature in z15 with zlib in Linux distributions.
The following build instructions have been tested on RHEL 7.7 and SLES 12 SP4. However, they can be easily adopted to other Linux distributions or levels as well. Ubuntu 19.10 includes support out of the box.

1. Install build requirements:
   RHEL 7.7:
     sudo yum install git autoconf automake libtool cpp gcc \
       glibc-devel glibc-headers kernel-headers libgomp \
       libmpc m4 mpfr perl-Test-Harness perl-Thread-Queue python
   SLES 12 SP4:
     sudo zypper install git-core autoconf automake libltdl7 \
       libtool python gcc
2. Validate current compression performance by running a typical python compression test:
     time python -c 'import gzip
     gzip.open("test.gz","wb",compresslevel=6). \
       write(b"abcdefghijklmnopqrstuvwxyz" * 50000000)'
3. Clone the IBM zlib fork which already contains the required patch:
     git clone https://github.com/iii-i/zlib.git --branch dfltcc \
       --single-branch
     cd zlib
4. Configure with the proper build flags to enable hardware compression support. You can specify options to determine at which compression levels hardware compression should be used. This can also be overriden at runtime using the DFLTCC_LEVEL_MASK environment variable.
     CFLAGS="-O2 -DDFLTCC -DDFLTCC_LEVEL_MASK=0x7e" ./configure \
       --libdir=/usr/local/lib64
   The DFLTCC_LEVEL_MASK macro at configuration time as well as the environment variable at runtime are expected to contain bitmasks with ones enabling hardware compression for a particular level. A value of 0x7e (0...01111110) enables hardware compression for levels 1-6 out of 0-9. Note that the hardware compression algorithm will not distinguish between compression levels for which it is enabled - it will always produce compression ratios similar to software compression at level 1.
   We build with compiler optimization level -O2 to be comparable to the regular Linux distribution builds. Performance enthusiasts might want to bump this to -O3 here.
   The --libdir setting is required to make the library be installed into the lib64 sub directory instead of just lib after installation.
5. During the build step (using the make command) the additional files need to be pulled in. So build with:
     make OBJA=dfltcc.o PIC_OBJA=dfltcc.lo
6. Install as usual:
     sudo make install
7. (RHEL only): Enable shared libraries to be searched for in /usr/local/lib64
     sudo sh -c 'echo /usr/local/lib64 > /etc/ld.so.conf.d \
       /usrlocal.conf'
     sudo ldconfig
8. Verify that existing binaries linked against zlib have the dynamic linker now resolve to our new version of the zlib library (located in /usr/local/lib64 instead of /usr/lib64):
     ldd $(which git) | grep libz
     libz.so.1 => /usr/local/lib64/libz.so.1 (0x000003ff99900000)
9. Validate the improved compression performance by re-running the test from step 2.
     time python -c 'import gzip; gzip.open("test.gz", "wb", \
       compresslevel=6).write(b"abcdefghijklmnopqrstuvwxyz" * \
       50000000)'
   The new time should be about a fifth of the old value. (We do not reach top-speed here since Python calls into zlib with small buffer sizes)

See here for additional information on how to enable existing support in Linux distributions.