Setting up a Multiarch Environment for s390x

A multiarch environment allows the execution of binary executables for multiple architectures: The host architecture, and any other architecture that support is added for. Multiarch environments can be used for cross-platform development and (limited) testing of user-space programs for a foreign architecture.
This can be very helpful in case access to IBM Z or LinuxONE hardware is limited: By utilizing multiarch environments, developers can conveniently develop code e.g. on their laptops in an environment of their choice.

Note:
Multiarch is supported only by Debian and Ubuntu, and the remainder of this article assumes a Debian or Ubuntu environment.

TL;DR

The following commands set up a multiarch environment for the foreign architecture s390x on any different host architecture (e.g. x86_64/amd64):

    $ sudo dpkg --add-architecture s390x
    $ sudo apt update
    $ sudo apt upgrade
    $ sudo apt install libc6:s390x
    $ sudo apt install qemu-user

Executables for s390x can now be started seamlessly.

Description

A multiarch environment allows the execution of binaries for an architecture different of the host architecture.

Binary executables and shared libraries mostly consist of instructions for a specific CPU architecture. Most installations only consist of applications and libraries for the host architecture, so that these instructions are processed directly by the host CPU. Linux distributions typically package applications and libraries and define dependencies between them. If an application requires a library at runtime, the package management will not only install the application package, but also the package of the required libraries.

If the binary executable (or library) is for an architecture different to the host architecture, some more infrastructure is required. The required libraries of an (foreign architecture) application must be available, and for processing the foreign architecture instructions an emulation must be available, too. If package management should be used to manage packages for a foreign architecture as well, package management should be aware of it. Fortunately, some Linux distributions provide support to set up such an environment.

If a multiarch environment should also be used for cross-development, additionally the cross-compiler and -toolchain for each foreign architecture are also required.

Note:
Currently, a multiarch environment requires a Debian or Ubuntu base installation. Fedora, Red Hat, and SUSE do not support out-of-the-box multiarch at the moment.

Installation

The support for a foreign architecture requires in most cases no manual configuration changes and can be set up completely with the Debian package management, dpkg and apt.

The following commands mainly follow the instructions described in the Debian multiarch howto and hints. They can be used to set up a multiarch-environment to run s390x-binaries on a x86_64/amd64 machine.

By replacing the s390x architecture name with another one (e.g. arm64), the commands can be used to set up other multiarch environments as well. The commands also work on host architectures other than x86_64/amd64.

First of all, the package management of the system needs to be aware of the foreign architecture.

    #
    # configure package management
    # generic command:
    #   dpkg --add-architecture <foreign-arch>
    #
    $ sudo dpkg --add-architecture s390x

To check the previous step, the host and foreign architecture can be queried from the package management.

    #
    # optional: check architectures of package management
    #
    $ dpkg --print-architecture
    $ dpkg --print-foreign-architectures

Note:
Debian systems with the old apt configuration file format might require an additional manual modification of each entry in the sources list. The host and foreign architectures must be listed between the deb or deb-src keyword and the related repository URL. Example (in /etc/apt/sources.list):

deb [arch=amd64,s390x] http://ftp.de.debian.org/debian <release> main

Ubuntu systems do not need any further changes.

After configuring the foreign architecture, the lists of the package manager requires an update.

    #
    # refresh package management
    #
    $ sudo apt update
    $ sudo apt upgrade

To run any dynamically linked binary, the architecture-specific dynamic linker is required. The dynamic linker is packaged with the dynamic libraries of glibc. Therefore, the libc6 package for the foreign architecture must be installed.

    #
    # install base runtime library and dynamic loader for s390x
    # generic command:
    #   apt install <pkg-name>:<foreign-arch>
    #
    $ sudo apt install libc6:s390x

Because the host CPU can not run directly the instructions in the foreign architecture binaries, an emulator is required. The qemu-user package provides an emulator of binary formats (binfmts) for the supported foreign architectures.

    #
    # install binfmt for foreign architectures
    #
    sudo apt install qemu-user

For developing/compiling applications for a foreign architecture on the host, a cross-compiler and -toolchain can be installed.

    #
    # optional: install cross-compiler
    # generic command:
    #   apt install gcc-<foreign-arch>-linux-gnu
    #
    $ sudo apt install gcc-s390x-linux-gnu

If the developed application uses a library, install the development packages of this library for the foreign architecture. E.g. for libstd, run:

    #
    # optional: install build dependencies
    # generic command:
    #   apt install <library>-dev:s390x
    #
    $ sudo apt install libzstd-dev:s390x

To run the developed application, also the runtime package for the used library is required for the foreign architecture. E.g. for libstd, run:

    #
    # optional: install runtime dependencies
    # generic command:
    #   apt install <library>:s390x
    #
    $ sudo apt install libzstd1:s390x

Working with multiarch

After installing all required packages, the multiarch environment is ready to work with. The easiest way to get an application for a foreign architecture is to install the packaged version for the foreign architecture.

Cross-compiled applications can also be started, as shown in the following example:

    $ cat > main.c << EOF
    > #include <stdio.h>
    >
    > int main(void)
    > {
    >     printf("hello foreign architecture!\n");
    >     return 0;
    > }
    > EOF
    $ s390x-linux-gnu-gcc -Wall -Wextra -o hello-arch main.c
    $ file hello-arch
    hello-arch: ELF 64-bit MSB pie executable, IBM S/390, version 1 (SYSV),
                dynamically linked, interpreter /lib/ld64.so.1, BuildID[sha1]=<...>,
                for GNU/Linux 3.2.0, not stripped
    $ ./hello-arch
    hello foreign architecture!

Note:
There might be limitations in the emulation of the foreign architecture. Not only with respect to the performance, but also for the supported hardware capabilities. E.g. the emulation for s390x does not support hardware accelerators like the message security assist (MSA).