Containers: How to build a Monitoring Stack on IBM Z

The Monitoring stack of the tools allows to gather node metrics and alerts and shows them in a nice way. The Monitoring stack on IBM Z and LinuxONE consists of the following parts:

• Grafana
• Prometheus
• Node exporter
• Alertmanager
• cAdvisor

This article will show how to build binaries and Docker images for all parts.

It is possible to use any available OS with Docker support on Z. However the steps below are for Ubuntu 18.10.

First, let's install the common dependencies:

apt-get update
apt-get install git curl wget phantomjs docker.io -y

Grafana

  Install the dependencies (Yarn, Node.js, Golang)

curl -o- -L https://yarnpkg.com/install.sh | bash
export PATH="$HOME/.yarn/bin:$HOME/.config/yarn/global/node_modules/.bin:$PATH"
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.34.0/install.sh | bash
export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"
nvm install 10.16.0
wget https://dl.google.com/go/go1.12.6.linux-s390x.tar.gz
tar -xvf go1.12.6.linux-s390x.tar.gz
mv go /usr/local
export GOROOT=/usr/local/go
export PATH=$PATH:/usr/local/go/bin

 

 Build Grafana binary

git clone https://github.com/grafana/grafana
cd grafana
QT_QPA_PLATFORM=offscreen npm install phantomjs
QT_QPA_PLATFORM=offscreen make

Build Grafana Docker image

Update the Dockerfile with the following patch:

diff --git a/Dockerfile b/Dockerfile
index fd305d8af..f0e6da211 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -22,7 +22,8 @@ WORKDIR /usr/src/app/
 COPY package.json yarn.lock ./
 COPY packages packages

-RUN yarn install --pure-lockfile --no-progress
+RUN apt-get update; apt-get install phantomjs -y
+RUN QT_QPA_PLATFORM=offscreen yarn install --pure-lockfile --no-progress

 COPY Gruntfile.js tsconfig.json tslint.json ./
 COPY public public
@@ -71,7 +72,7 @@ RUN mkdir -p "$GF_PATHS_HOME/.aws" && \
     chown -R grafana:grafana "$GF_PATHS_DATA" "$GF_PATHS_HOME/.aws" "$GF_PATHS_LOGS" "$GF_PATHS_PLUGINS" "$GF_PATHS_PROVISIONING" && \
     chmod 777 -R "$GF_PATHS_DATA" "$GF_PATHS_HOME/.aws" "$GF_PATHS_LOGS" "$GF_PATHS_PLUGINS" "$GF_PATHS_PROVISIONING"

-COPY --from=0 /go/src/github.com/grafana/grafana/bin/linux-amd64/grafana-server /go/src/github.com/grafana/grafana/bin/linux-amd64/grafana-cli ./bin/
+COPY --from=0 /go/src/github.com/grafana/grafana/bin/linux-s390x/grafana-server /go/src/github.com/grafana/grafana/bin/linux-s390x/grafana-cli ./bin/
 COPY --from=1 /usr/src/app/public ./public
 COPY --from=1 /usr/src/app/tools ./tools
 COPY tools/phantomjs/render.js ./tools/phantomjs/render.js

Run it:

docker build -t grafana/grafana:dev .

Prometheus, Node exporter and Alertmanager

You can get prebuilt binaries with

prom_version=2.10.0
wget https://github.com/prometheus/prometheus/releases/download/v${prom_version}/prometheus-${promversion}.linux-s390x.tar.gz
alert_version=0.17.0
wget https://github.com/prometheus/alertmanager/releases/download/v${alert_version}/alertmanager-${alert_version}.linux-s390x.tar.gz
node_exp_version=0.18.1
wget https://github.com/prometheus/node_exporter/releases/download/v${node_exp_version}/node_exporter-${node_exp_version}.linux-s390x.tar.gz

Build Prometheus binary

git clone https://github.com/prometheus/prometheus
cd prometheus
make build

The Prometheus binaries will then be available in your build directory.

Build Prometheus Docker image

git clone https://github.com/prometheus/prometheus
cd prometheus
mkdir -p build/linux-s390x/
cp console_libraries consoles prometheus promtool ./documentation/examples/prometheus.yml build/linux-s390x/
sed -i "s|quay\.io\/prometheus\/busybox.*:|s390x\/busybox:|g" Dockerfile
sed -i "s|documentation\/examples\/prometheus.yml|\.build\/linux-s390x\/prometheus.yml|g" Dockerfile

sed -i "s|console_libraries/|\.build\/linux-s390x\/console_libraries|g" Dockerfile
sed -i "s|consoles/|\.build\/linux-s390x\/consoles|g" Dockerfile
sed -i "s|\.build|build|g" Dockerfile
sed -i "s|amd64|s390x|g" Dockerfile
docker build -t "prom/prometheus-linux-s390x:latest" --build-arg ARCH="s390x" --build-arg OS="linux" --no-cache .

Build Node exporter binary

git clone https://github.com/prometheus/node_exporter
cd node_exporter
make build

The node_exporter binary will be available at the build directory.

 

Build Node exporter Docker image

git clone https://github.com/prometheus/node_exporter
cd node_exporter
mkdir -p build/linux-s390x/
cp node_exporter build/linux-s390x/
sed -i "s|quay\.io\/prometheus\/busybox.*:|s390x\/busybox:|g" Dockerfile
sed -i "s|\.build|build|g" Dockerfile
sed -i "s|amd64|s390x|g" Dockerfile
docker build -t "prom/node-exporter-linux-s390x:latest" --build-arg ARCH="s390x" --build-arg OS="linux" --no-cache .

Build Alertmanager binary

git clone https://github.com/prometheus/alertmanager
cd alertmanager
make build

The alertmanager binaries will be available in your building directory.

Build Alertmanager Docker image

git clone https://github.com/prometheus/alertmanager
cd alertmanager
mkdir -p build/linux-s390x/
cp alertmanager amtool build/linux-s390x/
sed -i "s|quay\.io\/prometheus\/busybox.*:|s390x\/busybox:|g" Dockerfile
sed -i "s|\.build|build|g" Dockerfile
sed -i "s|amd64|s390x|g" Dockerfile
docker build -t "prom/alertmanager-linux-s390x:latest" --build-arg ARCH="s390x" --build-arg OS="linux" --no-cache .

cAdvisor

Build cAdvisor binary

go get -d github.com/google/cadvisor
go get github.com/klauspost/crc32
cd $GOPATH/src/github.com/google/cadvisor
make build

The cAdvisor binaries will be placed in $GOPATH/src/github.com/google/cadvisor

Build cAdvisor Docker image

Change Dockerfile by adding

FROM ubuntu:18.10

and remove all apk lines

To build Docker images from source, run

./deploy/build.sh

Finally, we have all Docker images for the monitoring stack as follows:

• grafana/grafana:dev
• prom/prometheus-linux-s390x
• prom/alertmanager-linux-s390x
• prom/node-exporter-linux-s390x
• google/cadvisor:beta