Linux on IBM Z and LinuxONE: How to define a persistent HSCI interface in RHEL 8.6 and 9.0 or later

These instructions work for RHEL 8.6 & 9.0 or later.

Remove external/OSA and HiperSockets (HS) devices from the ignore list
```
$ cio_ignore -r 0.0.8000-0.0.8002
$ cio_ignore -r 0.0.b000-0.0.b002
```

Enable Layer2 for HS device and activate both devices

$ chzdev -e 0.0.8000 layer2=1
$ chzdev -e 0.0.b000 vnicc/flooding=1 vnicc/mcast_flooding=1

Rename the NetworkManager (NM) connections

The NM recognizes the enabled devices and adds a connection for each new interface. Renaming the connections is not required but helps later to easily identify the interfaces and their corresponding NM connection.
```
$ nmcli connection modify "Wired connection 1" connection.id enc8000
$ nmcli connection modify "Wired connection 2" connection.id encb000
```
Install the s390utils package

The s390utils package contains the hsci tool, but is not part of the basic RHEL 9.0 setup.
```
$ dnf install s390utils.s390x
```

Create the HSCI interface

Add HSCI for enc8000 (HS dev) and encb000 (OSA dev)

$ which hsci
  /usr/sbin/hsci

$ hsci -v
  hsci utility: version 2.19.0-2.el9
  Copyright IBM Corp. 2020

$ hsci add enc8000 encb000
  Verifying net dev encb000 and HiperSockets dev enc8000
  Adding hsci8000 with a HiperSockets dev enc8000 and an external \
                                                      dev encb000
  Add HSCI interface hsci8000

$ hsci show
  HSCI      PNET_ID           HiperSockets     External
  ------------------------------------------------------------
  hsci8000  NET1              enc8000          encb000

Check the NM status

NM re-creates the connections enc8000 and encb000 and assigns them to the hsci8000-br bridge.

Note
The re-creation process can take some time, but it will happen. It creates duplicate entries in the nmcli connections table, which you can remove via their UUID, because they are not needed anymore.

$ nmcli c
  NAME         UUID                                  TYPE      DEVICE
  encbd00      372a1df0-9862-4451-a949-ef898b465f35  ethernet  encbd00
  hsci8000-br  e8e07f51-4176-4984-ac33-f951f35feaca  bridge    hsci8000-br
  enc8000      52342c73-9b99-4df8-8155-8eab8f043afe  ethernet  enc8000
  encb000      bdd90d17-a5be-4ea0-9d47-deace390322d  ethernet  encb000
  enc8000      65a5b885-0a65-3c79-bdc6-789ac8384695  ethernet  --
  encb000      4050749a-e25e-31fa-a4c1-948a97e0ade9  ethernet  --

$ nmcli d
  DEVICE       TYPE      STATE                   CONNECTION
  encbd00      ethernet  connected               encbd00
  hsci8000-br  bridge    connected (externally)  hsci8000-br
  enc8000      ethernet  connected (externally)  enc8000
  encb000      ethernet  connected (externally)  encb000
  hsci8000     ethernet  unmanaged               --
  hsci8000-bp  ethernet  unmanaged               --
  lo           loopback  unmanaged               --

$ nmcli c del 65a5b885-0a65-3c79-bdc6-789ac8384695
$ nmcli c del 4050749a-e25e-31fa-a4c1-948a97e0ade9

Persist the HiperSockets and OSA device

NM recognizes the HSCI setup and you only have to persist the appropriate NM connections. If you do not persist these interfaces now, existing NM connection definitions are deleted together with all their settings after reboot.
```
nmcli c modify encb000 connection.autoconnect yes
nmcli c m enc8000 connection.autoconnect yes
```
Persist the HSCI bridge device
```
nmcli c m hsci8000-br connection.autoconnect yes
```
To prepare the NM for the new hsci tool (with MultiMAC support), create two NM connections for the veth devices, which are generated by the hsci tool. One of these two veth-devices will act as the main hsci interface (hsci8000). Assign an appropriate IP address to this hsci8000 NM connection so that the NM knows what to do when the hsci tool generates this device.
```
nmcli c add type veth ifname hsci8000-bp con-name hsci8000-bp \
    autoconnect yes save yes peer hsci8000 master hsci8000-br

nmcli c add type veth ifname hsci8000 con-name hsci8000 \
    autoconnect yes ipv4.addresses 10.30.83.24/16 ipv4.method \
    manual peer hsci8000-bp
```

Configure the NM network

Note
NM stores the persistent network configuration in *.nmconnection-files in /etc/NetworkManager/system-connections.

$ ls -1 /etc/NetworkManager/system-connections
  enc8000.nmconnection
  encb000.nmconnection
  encbd00.nmconnection
  hsci8000-bp.nmconnection
  hsci8000-br.nmconnection
  hsci8000.nmconnection

Add the NM dispatcher script

NM dispatcher scripts are called by the NM every time the network interface status changes (this happens during every reboot, too). The NM sends two parameters to these dispatcher scripts:
- The name of the interface (IFACE=$1)
- The name of the interface (EVENT=$2)
Notes

The script 100-hsci8000-mm-setup_rhel9.x.sh (reprinted below!) must be placed in /etc/NetworkManager/dispatcher.d and serves as an example (with enc8000 and encb000), which you must adapt according to your network device names. It explicitly checks if the first parameter (interface name) equals enc8000 or encb000 and second parameter contains UP. If this is the case, it configures the HSCI by calling /usr/sbin/hsci add enc8000 encb000.
The hsci-tool shipped with RHEL8.6 and later (version >=2.19) is idempotent and thus can be run multiple times with the same result.

#!/usr/bin/env bash

IFACE=$1
EVENT=$2

HS_DEV=enc8000
EXT_DEV=encb000
HSCI_DEV=hsci8000

################################################################################
# Check if hsci-device already exists
check_for_existing_hsci () {
	if [[ $(hsci show | grep $HSCI_DEV | wc -l) -eq 1 ]]; then
		return 1
	else
		return 0
	fi
}

################################################################################
# Check,
#   if received device is a HS HSCI port
#   if received device got event UP
#   if the other HSCI (external) port is UP
# Then run "hsci add ..."
if [[ $IFACE == $HS_DEV ]] && [[ $EVENT == up ]] && [[ $(ip link show $EXT_DEV  \
                                             | grep UP | wc -l) -eq 1 ]]; then
	logger "hsci-nmdispatcher-$HSCI_DEV: Configure and start $HSCI_DEV"
	logger "hsci-nmdispatcher-$HSCI_DEV: IFACE matched $HS_DEV, and $EXT_DEV is UP"

	check_for_existing_hsci
	if [[ $? -ne 0 ]]; then
		logger "hsci-nmdispatcher-$HSCI_DEV: $HSCI_DEV already configured and running"
		exit 1
	fi

	logger "hsci-nmdispatcher-$HSCI_DEV: /usr/sbin/hsci add $HS_DEV $EXT_DEV"
	/usr/sbin/hsci add $HS_DEV $EXT_DEV
fi

################################################################################
# Check,
#   if received device is an external HSCI port
#   if received device got event UP
#   if the other HSCI port (HS) is UP
# Then run "hsci add ..."
if [[ $IFACE == $EXT_DEV ]] && [[ $EVENT == up ]] && [[ $(ip link show $HS_DEV | grep UP \
                                                                 | wc -l) -eq 1 ]]; then
	logger "hsci-nmdispatcher-$HSCI_DEV: Configure and start $HSCI_DEV"
	logger "hsci-nmdispatcher-$HSCI_DEV: IFACE matched $EXT_DEV, and $HS_DEV is UP"

	check_for_existing_hsci
	if [[ $? -ne 0 ]]; then
		logger "hsci-nmdispatcher-$HSCI_DEV: $HSCI_DEV already configured and running"
		exit 1
	fi

	logger "hsci-nmdispatcher-$HSCI_DEV: /usr/sbin/hsci add $HS_DEV $EXT_DEV"
	/usr/sbin/hsci add $HS_DEV $EXT_DEV
fi

Linux on IBM Z and LinuxONE

How to define a persistent HSCI interface in RHEL 8.6 and 9.0 or later

No comments:

Post a Comment

Popular Posts

Disclaimer