Bert's blog

Containerized Nginx reverse-proxy and Let's Encrypt certificates

Mon, 19 Dec 2022 00:00:00 +0000

When exposing services running on your home server to the Internet, you only allow encrypted connections. I’m exposing Home Assistant, Jellyfin and some other things via a containerized Nginx with HTTP3 and I use Let’s Encrypt to supply certificates.

With this setup, you have an A+ rating at Qualys.

What’s in this guide?

When you follow this guide you will have:

Nginx set up as a reverse proxy
Nginx running in a container
HTTP3 support in Nginx
TLS provided by Let’s Encrypt certificates
Certificates installed by containerized Certbot
Automatic renewal of certificates using a systemd timer

Prerequisites:

A container engine like Podman or Docker. I’m using Podman on an RHEL system (with SELinux enforced)
Podman-compose or docker-compose

Port forwarding and firewall management are out of the scope of this guide. For HTTP3 you will need to open port 80/TCP (only to redirect to port 443) and port 443 (UDP and TCP).

HTTP3 and Nginx

HTTP3 may be officially released as a standard, in Nginx it has not yet made it into the stable release. Nginx does, however, have the code ready for those brave enough to try. And we sure are brave, aren’t we?

F5, the company behind Nginx even provides a ready-to-use Dockerfile so you can build Nginx yourself. This also comes in handy if you want to add your own modules.

The source of the Containerfile is found on the Nginx website: https://www.nginx.com/blog/our-roadmap-quic-http-3-support-nginx/

Copy and paste the code from the blog entry and confirm it builds.

$ podman build -t nginx-http3 .

Directory structure and files

Because I’m going to run an Nginx and Certbot container separately, I have to take care of the directory structure. I also want the Nginx logs to persist, outside of the container, so I can run log-parsing services like fail2ban.

$ home - nginx
           |
           + container-compose.yml
           + Containerfile
           +- conf
           |    |
           |    + nginx.conf
           |    + http3.conf
           |    + letsencrypt-acme-challenge.conf
           |    +- conf.d
           |         |
           |         + sub.domain.tld.conf
           +- data
           |   |
           |   +- letsencrypt
           |   +- www
           +- log
               |
               +- letsencrypt
               +- nginx
                    |
                    + access.log
                    + error.log

(directories have a dash before their name)

The Nginx log files are already created. I noticed that Nginx sometimes has problems with creating the files on startup.

~/nginx/container-compose.yml:

version: "3.9"
services:
  nginx-http3:
    build: ./
    container_name: nginx
    #restart: unless-stopped
    volumes:
      - ~/nginx/conf:/etc/nginx:Z,ro
      - ~/nginx/log:/var/log:z
      - ~/nginx/data/www:/srv:z,ro
      - ~/nginx/data/letsencrypt:/letsencrypt:z,ro
    network_mode: host
    #ports:
    #  - "80:80"
    #  - "443:443/tcp"
    #  - "443:443/udp"
    #networks:
    #  - nginx_backend

#networks:
#  nginx_backend:
#    name: nginx_backend

A few words of explanation here:

The Nginx container uses host networking mode. This is to have the real client’s IP address available for logging purposes. There is a new network driver coming to Podman but it hasn’t made it to the standard RHEL repos yet.
Because we use the host network, you cannot connect to a second Podman network. Backend services will have to expose their port on the post to be reachable. Note that their port should still be firewalled.
Volumes are mapped as we created them in our home directory. They all are read-only except the log directory. Don’t forget the SELinux flags!

~/nginx/Containerfile:

FROM nginx AS build

WORKDIR /src
RUN apt-get update && apt-get install -y git gcc make autoconf libtool perl
RUN git clone -b v3.6.1 https://github.com/libressl-portable/portable.git libressl && \
    cd libressl && \
    ./autogen.sh && \
    ./configure && \
    make check && \
    make install

RUN apt-get install -y mercurial libperl-dev libpcre3-dev zlib1g-dev libxslt1-dev libgd-ocaml-dev libgeoip-dev
RUN hg clone -b quic https://hg.nginx.org/nginx-quic && \
    cd nginx-quic && \
    auto/configure `nginx -V 2>&1 | sed "s/ \-\-/ \\\ \n\t--/g" | grep "\-\-" | grep -ve opt= -e param= -e build=` \
      --with-http_v3_module --with-stream_quic_module \
      --with-debug --build=nginx-quic \
      --with-cc-opt="-I/src/libressl/build/include" --with-ld-opt="-L/src/libressl/build/lib" --with-ld-opt="-static" && \
    make

FROM nginx
COPY --from=build /src/nginx-quic/objs/nginx /usr/sbin
RUN /usr/sbin/nginx
EXPOSE 80 443 443/udp

Mind that HTTP3 uses UDP on port 443. You can leave out the --with-debug option if you’re not interested in debugging. Because it’s still an experimental build, I’m leaving it.

~/nginx/conf/nginx.conf

worker_processes  4;

error_log  /var/log/nginx/error.log  warn;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    log_format quic '$remote_addr - $remote_user [$time_local] '
                      '"$request" $status $body_bytes_sent '
                      '"$http_referer" "$http_user_agent" "$http3"';

    access_log  /var/log/nginx/access.log  quic;

    include /etc/nginx/conf.d/*.conf;
}

~/nginx/conf/http3.conf

ssl_protocols TLSv1.2 TLSv1.3; # TLSv1.3 necessary for http3
quic_retry on;
ssl_early_data on;

#ssl_prefer_server_ciphers on; # Let the client choose, as of 2022-05 these ciphers are all still secure.
ssl_dhparam /etc/nginx/dhparams4096.pem; # generate with 'openssl dhparam -out dhparam.pem 4096'
ssl_ciphers TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
ssl_ecdh_curve X25519:prime256v1:secp384r1;

ssl_session_cache shared:SSL:5m;
ssl_session_timeout 1h;
ssl_session_tickets off;
ssl_buffer_size 4k; # This is for performance rather than security, the optimal value depends on each site.
                                # 16k default, 4k is a good first guess and likely more performant.

ssl_stapling on;      # As of 2022-05 this version of nginx dosen't support ssl-stapling, but it might be in the future.
ssl_stapling_verify on;
resolver 192.168.130.10 8.8.8.8 valid=300s; # Use whichever resolvers you'd like, these are Cloudflare's and is one of the fastest DNS resolvers.
resolver_timeout 5s;

proxy_request_buffering off;

add_header alt-svc 'h3=":443"; ma=86400'; # Absolutely necessary header. This informs the client that HTTP/3 is available.
add_header Strict-Transport-Security max-age=15768000; # Optional but good, client should always try to use HTTPS, even for initial requests.

gzip off; #https://en.wikipedia.org/wiki/BREACH

Mind that certificates will be specified by the per-server configuration file

letsencrypt-acme-challenge.conf

location ^~ /.well-known/acme-challenge/ {
    default_type "text/plain";
    root /srv/letsencrypt;
}
location = /.well-known/acme-challenge/ {
    return 404;
}

~/nginx/conf/conf.d/sub.domain.tld

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

server {
    listen 80;
    #listen [::]:80;
    server_name sub.domain.tld;

    # Uncomment to redirect HTTP to HTTPS
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl http2;
    listen 443 http3
    listen [::]:443 ssl http2;
    listen [::]:443 http3
    server_name sub.domain.tld;

    # use a variable to store the upstream proxy
    # in this example we are using a hostname which is resolved via DNS
    # (if you aren't using DNS remove the resolver line and change the variable to point to an IP address e.g `set $jellyfin 127.0.0.1`)
    set $backend_service backend_service;
    #resolver 127.0.0.1 valid=30;

    include /etc/nginx/http3.conf;

    ssl_certificate /letsencrypt/live/sub.domain.tld/fullchain.pem;
    ssl_certificate_key /letsencrypt/live/sub.domain.tld/privkey.pem;
    ssl_trusted_certificate /letsencrypt/live/sub.domain.tld/chain.pem;

    # Security / XSS Mitigation Headers
    # NOTE: X-Frame-Options may cause issues with the webOS app
    add_header X-Frame-Options "SAMEORIGIN";
    add_header X-XSS-Protection "1; mode=block";
    add_header X-Content-Type-Options "nosniff";

    # Content Security Policy
    # See: https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP
    # Enforces https content and restricts JS/CSS to origin
    # External Javascript (such as cast_sender.js for Chromecast) must be whitelisted.
    # NOTE: The default CSP headers may cause issues with the webOS app
    #add_header Content-Security-Policy "default-src https: data: blob: http://image.tmdb.org; style-src 'self' 'unsafe-inline'; script-src 'self' 'unsafe-inline' https://www.gstatic.com/cv/js/sender/v1/cast_sender.js https://www.gstatic.com/eureka/clank/95/cast_sender.js https://www.gstatic.com/eureka/clank/96/cast_sender.js https://www.gstatic.com/eureka/clank/97/cast_sender.js https://www.youtube.com blob:; worker-src 'self' blob:; connect-src 'self'; object-src 'none'; frame-ancestors 'self'";

    # enable Letsencrypt validation
    include /etc/nginx/letsencrypt-acme-challenge.conf;

    proxy_buffering off;

    location / {
        proxy_pass http://$backend_service:port;
        proxy_set_header Host $host;
        proxy_redirect http:// https://;
        proxy_http_version 1.1;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
    }
}

The per-service configuration files will be different depending on the actual backend service. This example is taken from my Home Assistant setup. Obviously, the domain name has to be changed but also this line might need a change:

set $backend_service backend_service;

It puts the name of the host where the service is reachable into a variable. Since Nginx is running in the host network stack, you might want to point it to localhost or 127.0.0.1.

Please consult the documentation of the service you want to make available.

First run

If you start the Nginx container now, it will fail. This is because the certificates are not yet available and the server will throw an error. To overcome this, you can run Certbot in standalone mode.

Make sure port 80 is not firewalled on your server. Also, check that your router is properly forwarding the port.

Try out with

$ podman run -it --rm -p 80:80 -v ~/nginx/data/letsencrypt:/etc/letsencrypt:z -v ~/nginx/log:/var/log:z certbot/certbot certonly --standalone --staging --dry-run --key-type ecdsa --rsa-key-size 4096 -d sub.domain.tld

And do it for real with

podman run -it --rm -p 80:80 -v ~/nginx/data/letsencrypt:/etc/letsencrypt:z -v ~/nginx/log:/var/log:z certbot/certbot certonly --standalone --key-type ecdsa --rsa-key-size 4096 -d sub.domain.tld

Certbot will ask you a few questions such as your e-mail address and if you could agree with their terms. I assume you do. This info will be stored so you can automate the command when renewing. You can provide multiple domains by adding extra -d domain arguments.

Now that you have the certificates, you can start Nginx:

[nginx] $ podman-compose --no-pod up -d

Automatically start Nginx

Podman provides a simple way to generate systemd service files. So we make use of this functionality to auto-start Nginx when booting the server:

Make sure the Nginx container is still running.

Go to ~/.config/systemd/user

[~/.config/systemd/user] $ podman generate systemd --new --name --files nginx
[~/.config/systemd/user] $ systemctl --user daemon-reload
[~/.config/systemd/user] $ systemctl --user enable container-nginx.service
[~/.config/systemd/user] $ systemctl --user start container-nginx.service

Automatic renewal

Certbot is not integrated into the Nginx container so we must run it separately. This is quite straightforward with Systemd timers.

We’re still in the ~/.config/systemd/user directory and create two files.

renew-certificates.service

[Unit]
Description=Renew Let's Encrypt certificates
After=container-nginx.service

[Service]
Type=oneshot
ExecStart=podman run -it --rm -v %h/nginx/data/letsencrypt:/etc/letsencrypt:z -v %h/nginx/data/www/letsencrypt:/srv/letsencrypt:z -v %h/nginx/log:/var/log:z certbot/certbot renew --webroot -w /srv/letsencrypt
ExecStart=podman exec nginx nginx -s reload

[Install]
WantedBy=default.target

There are two commands in this unit file. The first command checks for renewal. The second command reloads Nginx inside its container so new certificates are picked up. Mind that changes in configuration files are also picked up. If you made changes that contain errors, neither the changes nor new certificates are activated.

renew-certificates.timer

[Unit]
Description=Renew Let's Encrypt certificates daily at 01:30

[Timer]
OnCalendar=*-*-* 01:30:00

[Install]
WantedBy=timers.target

Enable the timer:

$ systemctl --user daemon-reload
$ systemctl --user enable renew-certificates.timer

Done! Now you have a running Nginx reverse proxy with Let’s Encrypt certificates and automatic renewal. At the time of writing, this setup has an A+ rating at Qualys SSL Labs.

Switch Photo by Thomas Jensen on Unsplash

Hardware transcoding for Jellyfin

Sat, 10 Dec 2022 00:00:00 +0000

After relying on MiniDLNA for a long time I switched to Jellyfin as my media server. And I decided to give transcoding on-the-fly a go. Getting it to work wasn’t difficult but not as easy as toggling a switch either. How to get hardware transcoding working in Jellyfin for a J4105 processor?

Getting started

I’m not going to go into detail on how to get Jellyfin itself up and running. Jellyfin has great documentation. IT even has a section about Podman. This itself is already remarkable. For most software “containers” as the same as Docker.

I’m running Podman on Rocky Linux 9. My machine has an Intel J4105 processor.

This is the command to get the container up:

$ podman run \
  --cgroups=no-conmon \
  --sdnotify=conmon \
  -d \
  --rm
  --label io.containers.autoupdate=registry \
  --name jellyfin \
  --net=host \
  --user $(id -u):$(id -g) \
  --userns keep-id \
  --group-add keep-groups \
  -v /home/user/jellyfin/cache:/cache:Z \
  -v /home/user/jellyfin/config:/config:Z \
  -v /mnt/data/media:/media:ro,z \
  --device=/dev/dri/renderD128:/dev/dri/renderD128 \
  --device=/dev/dri/card0:/dev/dri/card0 \
  jellyfin/jellyfin:latest

It’s a long command but you can easily hide this in a container-compose.yml file. Or create a systemd file and run it using systemd. The command is taken from the documentation. Changes are:

use host networking instead of just port mapping. This is to enable DLNA discovery.
use a plain volume mount for media
add the devices to the container.

This last change is important of course. Without the hardware devices to transcode, you can’t use them obviously.

Don’t forget to open the relevant ports in your firewall.

Browse to your Jellyfin instance and add some libraries. Test and play around. I added some 4k films and enabled the DLNA server.

Enable hardware transcoding

As soon as I enabled hardware transcoding, my (old) television refused to play the 4k movies. “This file is not supported”, it said.

I had set the hardware acceleration to “Intel Quicksync (QSV)”. After all, Intel itself says on their website that the processor supports it.

I clearly need some extra configuration to do. There are some references in Jellyfin’s documentation but it’s not a step-by-step guide.

Getting it to work

Enable driver

The first step is to enable the driver in the kernel by setting the kernel options:

$ sudo modinfo i915 | egrep -i "guc|huc|dmc"

firmware:       i915/kbl_huc_4.0.0.bin
firmware:       i915/cml_huc_4.0.0.bin
firmware:       i915/icl_huc_9.0.0.bin
firmware:       i915/ehl_huc_9.0.0.bin
...
firmware:       i915/glk_dmc_ver1_04.bin
firmware:       i915/dg1_dmc_ver2_02.bin
firmware:       i915/adls_dmc_ver2_01.bin
firmware:       i915/adlp_dmc_ver2_14.bin
parm:           enable_guc:Enable GuC load for GuC submission and/or HuC load. Required functionality can be selected using bitmask values. (-1=auto [default], 0=disable, 1=GuC submission, 2=HuC load) (int)
parm:           guc_log_level:GuC firmware logging level. Requires GuC to be loaded. (-1=auto [default], 0=disable, 1..4=enable with verbosity min..max) (int)
parm:           guc_firmware_path:GuC firmware path to use instead of the default one (charp)
parm:           huc_firmware_path:HuC firmware path to use instead of the default one (charp)
parm:           dmc_firmware_path:DMC firmware path to use instead of the default one (charp)

Create a file /etc/modprobe.d/i915.conf:

options i915 enable_guc=2

Then update GRUB (mind that I’m on Rocky Linux, your command might need to be adapted):

$ sudo grub2-mkconfig -o /boot/efi/EFI/rocky/grub.cfg

Rebuild initramfs:

$ sudo dracut --force

Then reboot.

Verify:

$ dmesg | grep drm
[    6.270645] systemd[1]: Starting Load Kernel Module drm...
[    6.333527] ACPI: bus type drm_connector registered
[    7.116873] i915 0000:00:02.0: [drm] couldn't get memory information
[    7.118726] i915 0000:00:02.0: [drm] Applying Increase DDI Disabled quirk
[    7.123425] i915 0000:00:02.0: [drm] Finished loading DMC firmware i915/glk_dmc_ver1_04.bin (v1.4)
[    8.233395] i915 0000:00:02.0: [drm] failed to retrieve link info, disabling eDP
[    8.261328] i915 0000:00:02.0: [drm] GuC firmware i915/glk_guc_69.0.3.bin version 69.0
[    8.261347] i915 0000:00:02.0: [drm] HuC firmware i915/glk_huc_4.0.0.bin version 4.0
[    8.275174] i915 0000:00:02.0: [drm] HuC authenticated
[    8.275189] i915 0000:00:02.0: [drm] GuC submission disabled
[    8.275193] i915 0000:00:02.0: [drm] GuC SLPC disabled
[    8.277572] [drm] Initialized i915 1.6.0 20201103 for 0000:00:02.0 on minor 0
[    8.282514] i915 0000:00:02.0: [drm] Cannot find any crtc or sizes
[    8.285528] i915 0000:00:02.0: [drm] Cannot find any crtc or sizes
[    8.287996] i915 0000:00:02.0: [drm] Cannot find any crtc or sizes

Step one is done!

Permissions

To avoid issues with filesystem permissions inside and outside the container I opted to just give RW access to all on both the /dev/dri/card0 and /dev/dri/renderD128. My user had group access to both paths but the groups inside and outside the container are not the same and filesystem permissions fail. By granting everyone read and write access, this issue is out of the way.

This is what it looks like on my side:

$ ls -al /dev/dri
total 0
drwxr-xr-x.  3 root root        100 Dec 10 09:20 .
drwxr-xr-x. 21 root root       3520 Dec 10 09:20 ..
drwxr-xr-x.  2 root root         80 Dec 10 09:20 by-path
crw-rw-rw-.  1 root video  226,   0 Dec 10 09:20 card0
crw-rw-rw-.  1 root render 226, 128 Dec 10 09:20 renderD128

SELinux

Playback still doesn’t work. A common trick I try is to temporarily disable SELinux. Also this time: magic! Transcoding works!

And now we need to enable SELinux again.

$ sudo setsebool -P container_use_devices=true

and

$ sudo ausearch -c 'ffmpeg' --raw | audit2allow -M ffmpeg-renderD128
******************** IMPORTANT ***********************
To make this policy package active, execute:

semodule -i ffmpeg-renderD128.pp
$ sudo semodule -i ffmpeg-renderD128.pp

Don’t forget to re-enable SELinux.

Done!

You can now enjoy hardware-supported transcoding in Jellyfin on your J4105 machine.

Chip Photo by Brian Kostiuk on Unsplash

RHEL 9 and msmtp

Tue, 23 Aug 2022 00:00:00 +0000

I use msmtp as a simple SMTP relay to forward system messages to my Gmail address. After upgrading Rocky Linux to version 9, I noticed msmtp isn’t (yet?) available in the repositories. So I built it myself.

My first rpm ever

I surely could download the source code and install all the dependencies and tools to build and install. However, I don’t want to clutter my base system with all these one-time tools. So I decided to build it in a container. I’m way out of my comfort zone here.

After some trial and error and browsing for solutions, I ended up with this result.

Preparation

I’m building on the server so all the action below is done over an ssh connection. At the time of writing, the latest version of msmtp is 1.8.22.

$ mkdir -p msmtp/{build,src} && cd msmtp
$ touch Containerfile build.bash build/build_rpm.bash build/msmtp.spec

You now should have the following layout:

$ -- msmtp
       |
       +-- build
       |     |
       |     +-- build_rpm.bash
       |     +-- msmtp.spec
       +-- src (empty dir)
       +-- Containerfile
       +-- build.bash

Let’s see what should be in the files.

Containerfile

FROM rockylinux:9
RUN dnf install -y epel-release dnf-plugins-core
RUN dnf config-manager --set-enabled crb
RUN dnf install -y rpm-build redhat-rpm-config make gcc nano git tar unzip rpmlint autoconf automake libtool && \
    dnf clean all
RUN mkdir -p /rpmbuild/rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS}
COPY build/build_rpm.bash /rpmbuild/build_rpm.bash
COPY build/msmtp.spec /rpmbuild/msmtp.spec
ADD src /rpmbuild/rpmbuild/SOURCES/

Breakdown of the Containerfile: We start from the same base system as we’re going to run the result on. Next, we enable extra repos to be able to find msmtp’s dependencies and the developer tools. (Red Hat calls this CRB or CodeReady Linux Builder repository.) We’re then able to install everything we need to build the rpm.

All that’s left is to prepare the directory structure and copy/add all the files.

build.bash

#!/bin/bash

PACKAGE_TO_BUILD=msmtp

# directory where the rpm will be stored
if [ -d artifacts ] ; then
 rm -rf artifacts
fi
mkdir artifacts
chmod 777 artifacts

# First build an image that contains the sources and necessary packages for rpmbuild
podman build -t ${PACKAGE_TO_BUILD}-build .

# then run the image
if [ -z $WORKSPACE ] ; then
  WORKSPACE=$(pwd)
fi
podman run --rm -e PACKAGE=${PACKAGE_TO_BUILD} -v ${WORKSPACE}/artifacts:/artifacts:Z ${PACKAGE_TO_BUILD}-build /rpmbuild/build_rpm.bash

This is the script that eventually will launch the container to build the rpm. It’s simple. All it does is create an output directory, build the image and run the image. Nothing special here.

chmod +x so the file is executable.

build/build_rpm.bash

#!/bin/bash

# check if spec file is present
cd /rpmbuild
if [ ! -f ./${PACKAGE}.spec ] ; then
 echo Sorry, can not find rpm spec file
 exit 1
fi
cp ${PACKAGE}.spec /rpmbuild/rpmbuild/SPECS
dnf builddep -y ${PACKAGE}.spec

# then execute the rpmbuild command
cd rpmbuild
rpmbuild -ba --define "_topdir `pwd`" ./SPECS/${PACKAGE}.spec
# copy the rpms to the artifact directory.
if [[ -d /artifacts ]] ; then
 cp ./RPMS/x86_64/${PACKAGE}*.rpm /artifacts/
fi

It’s again a quite short script. It performs a check for the .spec file and then installs the dependencies that are specified in that spec file. When that’s done, the rpm is built and copied into the artifacts directory (which is created by the build.bash script above)

Don’t forget to chmod +x the file.

msmtp.spec

Name:           msmtp
Version:        1.8.22
Release:        1%{?dist}
Summary:        SMTP client
License:        GPLv3+
URL:            https://marlam.de/%{name}/
Source0:        https://marlam.de/%{name}/releases/%{name}-%{version}.tar.xz
#Patch1:        msmtp-0001-Fix-linking-w-o-gsasl.patch

BuildRequires: make
%if 0%{?el5}
BuildRequires:  openssl-devel
%else
BuildRequires:  gnutls-devel
%endif

BuildRequires:  autoconf
BuildRequires:  automake
BuildRequires:  gcc
BuildRequires:  gettext-devel
BuildRequires:  libidn-devel
BuildRequires:  libgsasl-devel
BuildRequires:  libsecret-devel

Requires(post):         %{_sbindir}/alternatives
Requires(postun):       %{_sbindir}/alternatives

%description
It forwards messages to an SMTP server which does the delivery.
Features include:
  * Sendmail compatible interface (command line options and exit codes).
  * Authentication methods PLAIN,LOGIN,CRAM-MD5,DIGEST-MD5,GSSAPI,and NTLM
  * TLS/SSL both in SMTP-over-SSL mode and in STARTTLS mode.
  * Fast SMTP implementation using command pipe-lining.
  * Support for Internationalized Domain Names (IDN).
  * DSN (Delivery Status Notification) support.
  * RMQS (Remote Message Queue Starting) support (ETRN keyword).
  * IPv6 support.

%prep
%autosetup -p1

%build
autoreconf -ivf
%configure --disable-rpath --with-libsecret --with-libgsasl %{?el5:--with-ssl=openssl}
make %{?_smp_mflags}

%install
make install DESTDIR=%{buildroot} INSTALL='install -p'
rm -f scripts/Makefile*
%find_lang %{name}
rm -f %{buildroot}%{_infodir}/dir

# setup dummy files for alternatives
touch %{buildroot}%{_bindir}/msmtp

%post
%{_sbindir}/update-alternatives --install %{_sbindir}/sendmail mta %{_bindir}/msmtp 40 \
  --slave %{_prefix}/lib/sendmail mta-sendmail %{_bindir}/msmtp \
  --slave %{_mandir}/man8/sendmail.8.gz mta-sendmailman %{_mandir}/man1/msmtp.1.gz \
  --slave %{_bindir}/mailq mta-mailq %{_bindir}/msmtp \
  --slave %{_mandir}/man1/mailq.1.gz mta-mailqman %{_mandir}/man1/msmtp.1.gz

%postun
if [ $1 -eq 0 ] ; then
        %{_sbindir}/update-alternatives --remove mta %{_bindir}/msmtp
fi

%files -f %{name}.lang
%license COPYING
%doc AUTHORS NEWS README THANKS scripts
%doc doc/msmtprc-system.example doc/msmtprc-user.example
%{_bindir}/%{name}*
%{_infodir}/%{name}.info*
%{_mandir}/man1/%{name}*.1*

%changelog
* Wed Aug 10 2022 My Name <e_mail@domain.com> - 1.8.22-1
- Ver. 1.8.22

I copied this file from Fedora’s source and happened to work on RHEL9 which isn’t a surprise of course.

Get the source

One last thing before building it is getting the actual source code. So download the .tar.xz file here and put it in the src directory. You don’t need to untar.

Build!

Now, just run the script:

$ ./build.bash

It builds an image called “msmtp-build”. Then creates a container that builds the rpm (and destroys the container afterwards).

Install msmtp by a simple:

$ sudo dnf install artifacts/msmtp-1.8.22-1.el9.x86_64.rpm

Done! Configure msmtp the same way as on other Linux systems as I did here.

Instead of copy-pasting, you can download the whole set of files from my Github: build-msmtp.

Pihole and OpenWRT

Tue, 01 Mar 2022 00:00:00 +0000

Pi-hole is a DNS server you run on your own network which filters out any unwanted lookups and therefore reduces your bandwidth usage. I installed it in a Podman container and configured my OpenWRT box.

Preparation

My server runs on Rocky Linux and Podman is supported out of the box. Check Podman’s documentation to install it if needed.

Download the Pi-hole image. Probably it’ll be fetched from the Docker registry but it is compatible.

$ podman pull pihole/pihole

Create directories to have your settings persisted. My Podman containers run under my user, data is in my home directory.

$ mkdir /home/<username>/pihole
$ mkdir /home/<username>/pihole/dnsmasq
$ mkdir /home/<username>/pihole/pihole

Next, give the directories proper permissions. Because Podman does not run as admin, it cannot write to the newly created directories. Inside the container, Pi-hole runs as user pihole. To the outside world, this user is mapped to a fictive user with id 100998:100998. This is how it is mapped on my system.

Adjust the permissions accordingly:

$ sudo chown -R 100998:100998 ~/pihole/*

Open relevant ports on your server

Via the cockpit GUI, go to “Networking”. In the section “Firewall”, click “Edit rules and zones”. From there add port 80 (only TCP) for Pi-hole’s web GUI and port 53 (TCP and UDP) for the DNS server.

Start Pi-hole

Starting is done by one command:

$ podman run --init -d --restart=always --net=host -e TZ=Europe/Warsaw -e WEBPASSWORD=<yourpassword> -e SERVERIP=<yourserverip> -v ~/pihole/pihole:/etc/pihole:Z -v ~/pihole/dnsmasq:/etc/dnsmasq.d:Z --name=pihole pihole/pihole

Mind there is no sudo here. Most of the options explain themselves. The password and server’s IP address need to be set to your wish.

You can already browse the Pi-hole web interface. Be sure to test this.

If you see the webpage showing up, create a service so Pi-hole starts automatically whenever your server reboots. (Remember, you have automatic security updates enabled. This can cause restarts!)

$ cd ~/.config/systemd/user
$ podman generate systemd --new --name --files pihole
$ systemctl --user daemon-reload
$ systemctl --user enable container-pihole.service

Adjust the network settings

My router runs on OpenWRT. All we need to do here is add a DHCP option so the router tells all clients to use Pi-hole as the DNS server.

Go to Network –> Interfaces. Click “Edit” on the LAN interface. In the “DHCP Server” tab, choose the “Advanced Settings” tab.

Here the screen already tells you how to do it: Add “6," as an option and hit "Save".

Rogue devices

Some devices have hardcoded DNS servers, Google devices among them. You can force them to use Pi-hole using the firewall on OpenWRT.

Go to Network –> Firewall and select the “Custom Rules” tab. Add the following rules:

iptables -t nat -A POSTROUTING -j MASQUERADE
iptables -t nat -I PREROUTING -i br-lan -p tcp --dport 53 -j DNAT --to <yourserverip>
iptables -t nat -I PREROUTING -i br-lan -p udp --dport 53 -j DNAT --to <yourserverip>
iptables -t nat -I PREROUTING -i br-lan -p tcp -s <yourserverip> --dport 53 -j ACCEPT
iptables -t nat -I PREROUTING -i br-lan -p udp -s <yourserverip> --dport 53 -j ACCEPT

It will forward traffic destined to outside DNS servers back to your Pi-hole instance.

What about DHCP?

In my setup, DHCP is still managed by OpenWRT. That’s because I have multiple VLANs with different subnets. Pi-hole lacks the functionality to handle them and I like to keep things together.

Pi-hole configuration

A few things still need to be configured on the Pi-hole side.

Log in to Pi-hole (http:///admin/) and go to "Local DNS" --> "DNS Records". Any hostnames you had in your network can be defined here.

In “Settings” –> “DNS” you need to define the upstream DNS servers. I use the servers given by my ISP. You can use Google’s or OpenDNS’s.

One last thing is “Conditional forwarding”. Since we’re not using Pi-hole as our DHCP server, it doesn’t know about client names, only about IP addresses. This can be tiresome to analyze the traffic as “Wife’s phone” is easier to understand than “192.168.1.125”.

Add your network configuration and the IP address of your DHCP server (which is my router’s).

If you restart your computer and are still able to browse on the Internet, you’re done!

Finishing Rocky setup

Wed, 01 Dec 2021 00:00:00 +0000

After the basic installation of Rocky Linux, it’s time to finish the setup. We’re preparing the system to actually deploy useful stuff.

Check

when you finished the basic setup, you can log in via the terminal or the web GUI (Cockpit). Check basic settings like hostname, time and timezone and so on.

Cockpit shows some statistics about your system. If you want to persist them you’ll have to install pcp. this can be done through the GUI. Alternatively, install cockpit-pcp using the command line:

$ sudo yum install cockpit-pcp

On the “Software updates” tab in the GUI, I also enabled automatic software updates (for security-related updates) and kernel patches. As the screen tells you, this means the server will reboot from time to time. Keep this in mind when you manually start programs you rely on. On the same page, you can do a manual update.

Add disks

Before adding my data disk (in the OS, that is), I added the relevant Cockpit addon and smartmontools. The disk supports SMART.

$ sudo yum smartmontools cockpit-storaged

Create a mount point:

$ sudo mkdir /mnt/data

Now add the disk. First, find its UUID.

$ sudo blkid
/dev/sdb1: UUID="1fdd1ceb-70e8-476b-819d-d7082c7b4749" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="4f330730-6bd1-4325-9343-37242b985408"
/dev/sda1: UUID="D41A-D5C0" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="EFI System Partition" PARTUUID="fca2473b-8a02-4600-a776-d9e58b572b32"
/dev/sda2: UUID="106247eb-24a7-4d88-857b-937a09100aba" BLOCK_SIZE="512" TYPE="xfs" PARTUUID="429fe31a-6a8b-4682-8103-a646248bc767"
/dev/sda3: UUID="9RG0BQ-1yKg-Nem8-O1vF-f20Q-YI45-WWJ7t7" TYPE="LVM2_member" PARTUUID="99e3b950-819c-428e-a391-6f962dbbfc1f"
/dev/mapper/rl_server-root: UUID="89b352ee-e051-4a94-bb3e-87d6348dc8d9" BLOCK_SIZE="512" TYPE="xfs"
/dev/mapper/rl_server-swap: UUID="91074d79-f7f3-4ec2-a824-2c530aec8ad7" TYPE="swap"
/dev/mapper/rl_server-home: UUID="43034b2e-0af3-49be-be5a-c4b7ed46b0aa" BLOCK_SIZE="512" TYPE="xfs"

The top entry is the right one. It’s the only one with ext4. I know the drive is formatted like that. If you’re unsure, manually mount and check. Add it to fstab.

$ sudo nano /etc/fstab

Add a line like this:

UUID=1fdd1ceb-70e8-476b-819d-d7082c7b4749 /mnt/data auto nofail,noatime,noexec,errors=remount-ro 0 0

Every item is separated by a space. Some explanation:

UUID: the long code found by blkid
mount point: where in the filesystem will the disk be attached
filesystem type: etx4 or leave auto
options
- nofail: ignore errors at boot when the disk is not there
- noatime: don’t save access time (files and dirs)
- noexec: don’t allow executables on this drive
- errors=remount-ro: when errors are encountered, remount the drive read-only to avoid further damage
0: dump, leave disabled
0: fsck, set to zero to skip checks at boot

If you want it, you can also use the disk’s SMART capabilities:

$ sudo systemctl start smartd
$ sudo systemctl enable smartd

Podman

On my Debian installation, I ran Podman and a few virtual machines. It’s my goal to move everything to Podman. Podman is Redhat’s Docker. It runs daemonless and is capable of running rootless. These are advantages but might bring some difficulties, especially with SELinux. We’ll see where it goes.

At least installation is simple:

$ sudo yum install podman cockpit-podman

Adjust the port range Podman is allowed to use:

$ sudo nano /etc/sysctl.conf

Add

net.ipv4.ip_unprivileged_port_start=80

And reload

$ sudo sysctl --system

On last thing

While we’re at it, we might want to enable the “extra packages for enterprise linux”. Up to now we didn’t need any extra packages but we might run into this in the future and who knows, cause some head scratching. It doesn’t hurt to enable it:

$ sudo yum install epel-release

Photo Hercule’s Rock by Dzidek Lasek, https://pixabay.com/photos/paternal-national-park-autumn-2742115/

Moved to Rocky Linux

Thu, 18 Nov 2021 00:00:00 +0000

I’ve always been a big fan of Debian’s ultra-stability. But stability came with Debian’s number one downside: old packages. I took a leap of faith and switched to Rocky Linux.

Step 1: Backup

Although my data is on a separate drive, I backed up everything. Not only data (documents, music, pictures…) but also all my settings (Containerfiles and volumes most notably).

Step 2: Installation

Rocky Linux has plenty of documentation and this obviously included the installation procedure. Mind that for installation I use a monitor, keyboard and mouse.

In short:

Download the appropriate image. I chose the minimal version
Flash to a USB drive (I use BalenaEtcher for this)
Reboot the server and adjust the boot settings in BIOS to boot from the USB drive
Reboot again and follow the on-screen instructions.

Things I did during installation:

Enable network. My server gets its IP via DHCP but with a static lease it always gets the same address.
Adjust the timezone, and enable network time. Mind that you first have to enable network before enabling network time.
Enable “Security Policy”. This enables SELinux.
Software Selection: set it to “Minimal Install”. I’m freaky about this. If you’re more relaxed you could opt for “Server”.
Keep the root account disabled
Create a non-root account but make it an administrator
Select the “Installation Destination”. On My system, I’ve got two disks: one for the OS and related and one for data. Obviously, I select the OS drive and opt for automatic storage configuration. I don’t want to preserve anything so I “reclaim” all space on the drive. This deletes everything that is currently on the drive.

Click “Begin Installation” and done.

Step 3: Finish setup

After rebooting, your trusted but boring login screen will be served. But this is not Debian anymore. We can now have a built-in management interface.

Install by

sudo yum install cockpit

And enable by

sudo enable cockpit.socket
sudo systemctl start cockpit.socket

You can now browse to http://your.server.ip:9090. Cockpit will add a firewall rule to allow traffic to port 9090.

Et voilà, you’re ready to go.

Attention!

Rocky Linux comes with SELinux. It can be really annoying but that’s a good thing. If you would like to enable TLS on your cockpit GUI, you might experience the joys of dealing with SELinux. Imagine you upload your certificate to your home directory and move it into place from there. Then your file doesn’t have the right label and Cockpit won’t be able to read it. Cockpit will only give you an error saying it failed to start. Whether this is SELinux-related or not, you can easily check by running

$ sudo setenforce 0

Probably cockpit will happily serve you the webpage over TLS. But we want to enable SELinux again so we have to restore the file labels:

$ sudo restorecon -rv F /etc/cockpit
restorecon: lstat(/etc/cockpit/F) failed: No such file or directory
Relabeled /etc/cockpit/ws-certs.d/your-cert.cert from unconfined_u:object_r:user_home_t:s0 to unconfirmed_u:object_r:etc_t:s0
Relabeled /etc/cockpit/ws-certs.d/your-cert.key from unconfined_u:object_r:user_home_t:s0 to unconfirmed_u:object_r:etc_t:s0

You can now enable SELinux again using Cockpit or on the command line (sudo setenforce 1).

Photo Morskie Oko by Greg Trowman, https://unsplash.com/photos/D9QmRGgxRkk

Self signed certificates in Windows 11

Tue, 09 Nov 2021 00:00:00 +0000

If you generate your own, self-signed root certificate Windows will complain about it because Windows doesn’t trust you. It is a good thing but it also is annoying. I’ll show you how you can add your self-signed certificate to the certificate store so Windows 11 will trust you.

After following this guide you have your own, self-signed certificate. It’s the ~/keys/ca/root-ca.crt file. There are a few steps to follow but none of them involves a command line.

I created this guide on Windows 11, Windows 10 should also be OK.

Open the Microsoft Management Console

Open the “start menu” and type mmc. Windows will search for a program that matches “mmc”. Click the icon with the red toolbox. This will open the “Microsoft Management Console” which looks like this:

Add certificate manager

In the MMC, go to “File” » “Add/Remove Snap-in”. A new window will open.

From the list of snap-ins, chose “Certificates” and click “Add” and follow the wizard:

Select “Computer account”

And next chose “Local computer” (no other choices are available.) and click “Finish”.

The snap-in is added, you can now click “OK

List certificates

You can now browse through the already available certificates. In the middle pane, double click “Certificates (local Computer)” and then “Trusted Root Certificate Authorities” and finally “Certificates”. you’ll see a list of all root CAs the computer trusts. We created our own root certificate authority so we will have to add ours here.

Add our Root CA

With the list of certificates in the middle pane, right-click on “Certificates” in the left pane, go the “All Tasks” and “Import”. Alternatively, you can select “Action” from the top menu bar, go to “All Tasks” and finally “Import”. Follow the wizard.

Click “Next”

Browse to your root certificate (keys/ca/root-ca.crt) and click “Next”.

Click “Next” again. The wizard already selected the right option for us to store the new certificate in the right store.

Check and done!

Your certificate is now added to Windows. Your computer will now trust all the certificates that have been signed by this root certificate (and derivates). You might need to restart your browser for it to work with the newly added certificate.

Notary sign image by Peter H from Pixabay

Self signed certificates in Firefox Ubuntu

Wed, 21 Apr 2021 00:00:00 +0000

Even if you have imported your self-signed root certificate to your Ubuntu system, Firefox won’t use it. This has to be solved and I found a way.

You could just add an exception for your website to Firefox, but it still shows the little warning icon in the address bar to show something is fishy.

Firefox uses its own certificate store and doesn’t care about the certificates the system trusts.

Luckily, this answer on Ask Ubuntu has the path to the answer. At the time of writing, the answer is not the accepted one, nor has it a high rating. But in my opinion, it is the best one.

Even Mozilla’s website mentions it.

Install the replacement lib

The trick is changing Firefox’s behaviour by using another certificate management library.

The replacement lib is to be found on their website or on their Github page. The lib is also available on Ubuntu’s repositories.

$ sudo apt install p11-kit

On my system, it was already installed. Yeah!

Instruct Firefox to use the p11-kit

Unlike the Ask Ubuntu answer (which is already a few years old), Firefox has the option to import security modules. So we’ll include the P11-kit module to import the certificates from our system into Firefox.

Open the settings page in Firefox

Click “Privacy & Security” and scroll to “Security”

View certificates

When you click “View Certificates” you’ll get a list with all the trusted root certificates. Yours will not be in the list.

Close this screen.

Add the custom module

Back in the settings screen, click “Security Devices”. you’ll get a screen with all the modules that take care of certificate management.

Click “Load”.

Give the added module a sensible name and add the path to the P11-kit module:

Module name: P11 kit
Module filename: /usr/lib/x86_64-linux-gnu/pkcs11/p11-kit-trust.so

Click “OK” The Device manager now shows the included module with its certificate source.

Check certificates

when you click “View Certificates” in the settings screen, you can find your self-signed root certificate in the certificates list.

Unlike the answer on Ask Ubuntu, the added module is a setting that survives updates.

Notary sign image by Peter H from Pixabay

Certificate management

Wed, 21 Apr 2021 00:00:00 +0000

OpenSSL and certificate management are a black box to me. So I thought I’d better write down how I managed to create my certificates. I followed this tutorial: https://pki-tutorial.readthedocs.io/en/latest/index.html.

There are a lot of OpenSSL tutorials online to create your self-signed certificates. And in fact, when you have a public-facing service you’d better stick to trusted platforms like Let’s Encrypt. For our internal services, I’m trustworthy enough to have my own certificate authority. I want to create certificates for my OpenVPN server, my Mosquitto broker and the Cockpit control panel.

DISCLAIMER: The following steps are from https://pki-tutorial.readthedocs.io/en/latest/index.html. They are more elaborate than most other guides but the steps are clear and understandable and you learn how the chain of trust works.

ATTENTION: Private keys are to be kept strictly private so you don’t want the key files to be exposed to the Internet. It is best to set up the key infrastructure on a computer that is not connected to the Internet to prevent any unauthorized access. Alternatively, instead of generating the PKI in your home directory, you can put all the files on removable media like a USB stick and store it somewhere safe.

Preparation

You probably already have everything installed on your system. I’m creating the certificates on my Lubuntu laptop and afterwards I’ll transfer them to the server. It works equally fine in WSL or WSL2 on Windows.

In case you haven’t installed anything yet:

$ sudo apt update && sudo apt install openssl

Now, prepare the config files and directory layout:

$ mkdir ~/keys
$ cd ~/keys
$ mkdir -p ca/root-ca/private ca/root-ca/db crl certs etc
$ chmod 700 ca/root-ca/private
$ touch etc/root-ca.conf etc/tls-ca.conf etc/tls-server.conf etc/tls-client.conf

File etc/root-ca.conf

# ACME Root CA

[ default ]
ca                      = root-ca                   # CA name
dir                     = .                         # Top dir
base_url                = http://www.acme.com/ca    # CA base URL
aia_url                 = $base_url/$ca.cer         # CA certificate URL
crl_url                 = $base_url/$ca.crl         # CRL distribution point
name_opt                = multiline,-esc_msb,utf8   # Display UTF-8 characters

# CA certificate request

[ req ]
default_bits            = 4096                  # RSA key size
encrypt_key             = yes                   # Protect private key
default_md              = sha256                # MD to use
utf8                    = yes                   # Input is UTF-8
string_mask             = utf8only              # Emit UTF-8 strings
prompt                  = no                    # Don't prompt for DN
distinguished_name      = ca_dn                 # DN section
req_extensions          = ca_reqext             # Desired extensions

[ ca_dn ]
countryName             = "US"
organizationName        = "Acme"
organizationalUnitName  = "Acme Certificate Authority"
commonName              = "Acme Root CA"

[ ca_reqext ]
keyUsage                = critical,keyCertSign,cRLSign
basicConstraints        = critical,CA:true
subjectKeyIdentifier    = hash

# CA operational settings

[ ca ]
default_ca              = root_ca               # The default CA section

[ root_ca ]
certificate             = $dir/ca/$ca.crt       # The CA cert
private_key             = $dir/ca/$ca/private/$ca.key # CA private key
new_certs_dir           = $dir/ca/$ca           # Certificate archive
serial                  = $dir/ca/$ca/db/$ca.crt.srl # Serial number file
crlnumber               = $dir/ca/$ca/db/$ca.crl.srl # CRL number file
database                = $dir/ca/$ca/db/$ca.db # Index file
unique_subject          = no                    # Require unique subject
default_days            = 3652                  # How long to certify for
default_md              = sha256                # MD to use
policy                  = match_pol             # Default naming policy
email_in_dn             = no                    # Add email to cert DN
preserve                = no                    # Keep passed DN ordering
name_opt                = $name_opt             # Subject DN display options
cert_opt                = ca_default            # Certificate display options
copy_extensions         = none                  # Copy extensions from CSR
x509_extensions         = signing_ca_ext        # Default cert extensions
default_crl_days        = 365                   # How long before next CRL
crl_extensions          = crl_ext               # CRL extensions

[ match_pol ]
countryName             = match                 # Must match 'US'
stateOrProvinceName     = optional              # Included if present
localityName            = optional              # Included if present
organizationName        = match                 # Must match 'Acme'
organizationalUnitName  = supplied              # Must be present
commonName              = supplied              # Must be present

[ any_pol ]
domainComponent         = optional
countryName             = optional
stateOrProvinceName     = optional
localityName            = optional
organizationName        = optional
organizationalUnitName  = optional
commonName              = optional
emailAddress            = optional

# Extensions

[ root_ca_ext ]
keyUsage                = critical,keyCertSign,cRLSign
basicConstraints        = critical,CA:true
subjectKeyIdentifier    = hash
authorityKeyIdentifier  = keyid:always

[ signing_ca_ext ]
keyUsage                = critical,keyCertSign,cRLSign
basicConstraints        = critical,CA:true,pathlen:0
subjectKeyIdentifier    = hash
authorityKeyIdentifier  = keyid:always
authorityInfoAccess     = @issuer_info
crlDistributionPoints   = @crl_info

[ crl_ext ]
authorityKeyIdentifier  = keyid:always
authorityInfoAccess     = @issuer_info

[ issuer_info ]
caIssuers;URI.0         = $aia_url

[ crl_info ]
URI.0                   = $crl_url

In file etc/tls-ca.conf

# Acme TLS CA

[ default ]
ca                      = tls-ca                # CA name
dir                     = .                     # Top dir
base_url                = http://www.acme.com/ca    # CA base URL
aia_url                 = $base_url/$ca.cer     # CA certificate URL
crl_url                 = $base_url/$ca.crl     # CRL distribution point
name_opt                = multiline,-esc_msb,utf8 # Display UTF-8 characters

# CA certificate request

[ req ]
default_bits            = 4096                  # RSA key size
encrypt_key             = yes                   # Protect private key
default_md              = sha256                # MD to use
utf8                    = yes                   # Input is UTF-8
string_mask             = utf8only              # Emit UTF-8 strings
prompt                  = no                    # Don't prompt for DN
distinguished_name      = ca_dn                 # DN section
req_extensions          = ca_reqext             # Desired extensions

[ ca_dn ]
countryName             = "US"
organizationName        = "Acme"
organizationalUnitName  = "Acme Certificate Authority"
commonName              = "Acme TLS CA"

[ ca_reqext ]
keyUsage                = critical,keyCertSign,cRLSign
basicConstraints        = critical,CA:true,pathlen:0
subjectKeyIdentifier    = hash

# CA operational settings

[ ca ]
default_ca              = tls_ca                # The default CA section

[ tls_ca ]
certificate             = $dir/ca/$ca.crt       # The CA cert
private_key             = $dir/ca/$ca/private/$ca.key # CA private key
new_certs_dir           = $dir/ca/$ca           # Certificate archive
serial                  = $dir/ca/$ca/db/$ca.crt.srl # Serial number file
crlnumber               = $dir/ca/$ca/db/$ca.crl.srl # CRL number file
database                = $dir/ca/$ca/db/$ca.db # Index file
unique_subject          = no                    # Require unique subject
default_days            = 730                   # How long to certify for
default_md              = sha256                # MD to use
policy                  = match_pol             # Default naming policy
email_in_dn             = no                    # Add email to cert DN
preserve                = no                    # Keep passed DN ordering
name_opt                = $name_opt             # Subject DN display options
cert_opt                = ca_default            # Certificate display options
copy_extensions         = copy                  # Copy extensions from CSR
x509_extensions         = server_ext            # Default cert extensions
default_crl_days        = 1                     # How long before next CRL
crl_extensions          = crl_ext               # CRL extensions

[ match_pol ]
countryName             = match                 # Must match 'US'
stateOrProvinceName     = optional              # Included if present
localityName            = optional              # Included if present
organizationName        = match                 # Must match 'Acme'
organizationalUnitName  = supplied              # Must be present
commonName              = supplied              # Must be present

[ extern_pol ]
countryName             = supplied              # Must be present
stateOrProvinceName     = optional              # Included if present
localityName            = optional              # Included if present
organizationName        = supplied              # Must be present
organizationalUnitName  = supplied              # Must be present
commonName              = supplied              # Must be present

[ any_pol ]
domainComponent         = optional
countryName             = optional
stateOrProvinceName     = optional
localityName            = optional
organizationName        = optional
organizationalUnitName  = optional
commonName              = optional
emailAddress            = optional

# Extensions

[ server_ext ]
keyUsage                = critical,digitalSignature,keyEncipherment
basicConstraints        = CA:false
extendedKeyUsage        = serverAuth,clientAuth
subjectKeyIdentifier    = hash
authorityKeyIdentifier  = keyid:always
authorityInfoAccess     = @issuer_info
crlDistributionPoints   = @crl_info

[ client_ext ]
keyUsage                = critical,digitalSignature
basicConstraints        = CA:false
extendedKeyUsage        = clientAuth
subjectKeyIdentifier    = hash
authorityKeyIdentifier  = keyid:always
authorityInfoAccess     = @issuer_info
crlDistributionPoints   = @crl_info

[ crl_ext ]
authorityKeyIdentifier  = keyid:always
authorityInfoAccess     = @issuer_info

[ issuer_info ]
caIssuers;URI.0         = $aia_url

[ crl_info ]
URI.0                   = $crl_url

In file etc/tls-server.conf

# TLS server certificate request

[ default ]
SAN                     = DNS:acme.com      # Default value

[ req ]
default_bits            = 4096                  # RSA key size
encrypt_key             = no                    # Protect private key
default_md              = sha256                # MD to use
utf8                    = yes                   # Input is UTF-8
string_mask             = utf8only              # Emit UTF-8 strings
prompt                  = yes                   # Prompt for DN
distinguished_name      = server_dn             # DN template
req_extensions          = server_reqext         # Desired extensions

[ server_dn ]
countryName             = "1. Country Name (2 letters) (eg, US)       "
countryName_max         = 2
stateOrProvinceName     = "2. State or Province Name   (eg, region)   "
localityName            = "3. Locality Name            (eg, city)     "
organizationName        = "4. Organization Name        (eg, company)  "
organizationalUnitName  = "5. Organizational Unit Name (eg, section)  "
commonName              = "6. Common Name              (eg, FQDN)     "
commonName_max          = 64

[ server_reqext ]
keyUsage                = critical,digitalSignature,keyEncipherment
extendedKeyUsage        = serverAuth,clientAuth
subjectKeyIdentifier    = hash
subjectAltName          = $ENV::SAN

In file etc/tls-client.conf

# TLS client certificate request

[ req ]
default_bits            = 4096                  # RSA key size
encrypt_key             = yes                   # Protect private key
default_md              = sha256                # MD to use
utf8                    = yes                   # Input is UTF-8
string_mask             = utf8only              # Emit UTF-8 strings
prompt                  = yes                   # Prompt for DN
distinguished_name      = client_dn             # DN template
req_extensions          = client_reqext         # Desired extensions

[ client_dn ]
countryName             = "1. Country Name (2 letters) (eg, US)       "
countryName_max         = 2
stateOrProvinceName     = "2. State or Province Name   (eg, region)   "
localityName            = "3. Locality Name            (eg, city)     "
organizationName        = "4. Organization Name        (eg, company)  "
organizationalUnitName  = "5. Organizational Unit Name (eg, section)  "
commonName              = "6. Common Name              (eg, full name)"
commonName_max          = 64
emailAddress            = "7. Email Address            (eg, name@fqdn)"
emailAddress_max        = 40

[ client_reqext ]
keyUsage                = critical,digitalSignature
extendedKeyUsage        = clientAuth
subjectKeyIdentifier    = hash
subjectAltName          = email:move

Setup CA

A certificate authority takes care of signing certificates and keeping track which certificates it has signed but also which ones it has revoked. We will set up this infrastructure.

$ cp /dev/null ca/root-ca/db/root-ca.db
$ cp /dev/null ca/root-ca/db/root-ca.db.attr
$ echo 01 > ca/root-ca/db/root-ca.crt.srl
$ echo 01 > ca/root-ca/db/root-ca.crl.srl

Create a CA request

$ openssl req -new \
    -config etc/root-ca.conf \
    -out ca/root-ca.csr \
    -keyout ca/root-ca/private/root-ca.key

When asked for a passphrase, enter your password. The command uses the info from the root-ca.conf file. The result is a private key and a CSR (certificate signing request).

Create a CA certificate

$ openssl ca -selfsign \
    -config etc/root-ca.conf \
    -in ca/root-ca.csr \
    -out ca/root-ca.crt \
    -extensions root_ca_ext

We now have a self-signed root CA. This CA is the origin of trust in your public key infrastructure. To create “usable” certificates, you also need to have signing certificates. These are created in the next part. Notice the -selfsign option.

CRL

The root CA can also revoke certificates. Hence we have to create the certificate revoke list.

$ openssl ca -gencrl \
    -config etc/root-ca.conf \
    -out crl/root-ca.crl

Create TLS server CA

The TLS-server CA is an intermediate certificate that will be used to sign all TLS certificates. If that doesn’t make sense think of this: Would the president sign all the passports of his citizens by himself? Probably not. The president does however grant certain people the right to sign on his behalf. That’s what we’re also doing by creating a TLS server CA.

First, create the directories to store the files.

$ mkdir -p ca/tls-ca/private ca/tls-ca/db
$ chmod 700 ca/tls-ca/private

Create the database:

$ cp /dev/null ca/tls-ca/db/tls-ca.db
$ cp /dev/null ca/tls-ca/db/tls-ca.db.attr
$ echo 01 > ca/tls-ca/db/tls-ca.crt.srl
$ echo 01 > ca/tls-ca/db/tls-ca.crl.srl

Create a CA request

$ openssl req -new \
    -config etc/tls-ca.conf \
    -out ca/tls-ca.csr \
    -keyout ca/tls-ca/private/tls-ca.key

As a regular CA, this creates the private key and a CSR. Also, give the key a password to protect from unauthorized use.

Create CA

$ openssl ca \
    -config etc/root-ca.conf \
    -in ca/tls-ca.csr \
    -out ca/tls-ca.crt \
    -extensions signing_ca_ext

The root CA validates and signs the request, creating the TLS server CA.

It all starts coming together when looking at the parameters for the command: We create a ca (openssl ca -out ca/tls-ca.crt) using the CRS (-in ca/tls-ca.csr). The outcome is a signing CA (-extension signing_ca_ext). Mind that we here use the root CA’s config file. After all, the request was made by the TLS server CA’s config and now the work has to be done by the root.

CRL

Also here, create a certificate revoke list.

openssl ca -gencrl \
    -config etc/tls-ca.conf \
    -out crl/tls-ca.crl

PEM bundle

The PEM bundle is a certificate chain.

$ cat ca/tls-ca.crt ca/root-ca.crt > \
    ca/tls-ca-chain.pem

If you look at the contents of the file, it is nothing more than the root certificate and the tls-server certificate pasted after each other.

Issuing certificates

Our infrastructure is ready to create certificates that will be deployed to our servers.

Let’s create a certificate for our Cockpit control panel. The control panel is reachable on “cpanel.acme.com”¹.

Our infrastructure is completely ready now. When we want to have a certificate for one of our services we create a request and the authority signs it. Think of it like these real-world analogies:

bank notes backed up by a central bank
passports signed by a country’s representative
contracts notarized by a notary

Create a TLS server request

$ SAN=DNS:cpanel.acme.com \
openssl req -new \
    -config etc/tls-server.conf \
    -out certs/cpanel.acme.com.csr \
    -keyout certs/cpanel.acme.com.key

Mind the environment variable we set first. We also don’t set a passphrase.

Create the TLS certificate

$ openssl ca \
    -config etc/tls-ca.conf \
    -in certs/cpanel.acme.com.csr \
    -out certs/cpanel.acme.com.crt \
    -extensions server_ext

If this is your first certificate, a copy will be stored under ca/tls-ca/01.pem.

Which files do you need in your actual service?

tls-ca.crt: the public root certificate as the basis of trust
cpanel.acme.com.crt: the public server certificate which has the FQDN incorporated
cpanel.acme.com.key: the private key to be able to decrypt

Copy to cockpit

Cockpit requires to have the server certificate and the intermediate certificate in the same file. Although not mentioned in the docs, I also had to copy the private key into the same file to get it working. Also, notice the file extension change when moving the certificate chain to Cockpit: .crt becomes .cert.

$ cat certs/cpanel.acme.com.crt ca/tls-ca.crt certs/cpanel.acme.com.key > certs/cpanel.acme.com-chain.crt

Copy certs/cpanel.acme.com-chain.cert to your server and place it in Cockpit’s config directory.

$ sudo mv cpanel.acme.com-chain.crt /etc/cockpit/ws-certs.d/cpanel.acme.com.cert

Give the files the proper ownership and permissions:

$ sudo chown root:cockpit-ws /etc/cockpit/ws-certs.d/cpanel.acme.com.cert
$ sudo chmod 640 /etc/cockpit/ws-certs.d/cpanel.acme.com.cert

Revoking certificates

Should you want to revoke a certificate (the private key has been compromised) before its expiration date, you can add it to the CRL (certificate revoke list). You have to know the id-number of the certificate though. If you didn’t remember, look into the signing-db: ~/keys/ca/signing-ca/db/signing-ca.db.

Revoke a certificate

openssl ca \
    -config etc/signing-ca.conf \
    -revoke ca/signing-ca/01.pem \
    -crl_reason superseded

The compromised certificate is marked ‘revoked’ and will be included in new CRLs the CA issues.

Create a CRL

openssl ca -gencrl \
    -config etc/signing-ca.conf \
    -out crl/signing-ca.crl

Adding your own certificates your system

You’ve set up your own public key infrastructure and certificate authority. However, your browser (and other programs) doesn’t know about this.

To have your computer trust the root certificate, copy it to /usr/local/share/ca-certificates/.

$ sudo cp ~/keys/ca/root-ca.crt /usr/local/share/ca-certificates/
$ sudo update-ca-certificates

If you use Firefox as I do, you will notice that your cockpit login page still shows a warning. Firefox does not take your system’s certificates as a trusted store. To solve this, head over to this post.

You’ll then have this screen:

It still says the certificate is not trusted by Mozilla, but since you trust it Firefox won’t make a fuss about it.

Lock photo by pasja1000 from Pixabay

The control panel is only reachable from the LAN. Its domain name is defined in the router’s dnsmasq service. ↩

Eaton UPS monitoring

Tue, 13 Apr 2021 00:00:00 +0000

Power outages are rare nowadays. But they are not nonexistent. I bought a second-hand UPS and a new battery to keep my server up during an eventual short outage.

So I bought a second-hand Eaton Ellipse ASR 750. It’s a rather old device but it came with a new battery. And it has a USB connection so I plugged it into my server and started with Network UPS tools.

Installation

Network UPS Tools or NUT has been around for ages, so it made its way into the standard Debian repositories.

$ sudo apt install nut

This installs bash-completion libupsclient4 nut-client nut-server. Your machine may yield different results.

There were some error messages. We’ll take care of them later:

Created symlink /etc/systemd/system/multi-user.target.wants/nut-monitor.service → /lib/systemd/system/nut-monitor.service.
Job for nut-monitor.service failed because the service did not take the steps required by its unit configuration.
See "systemctl status nut-monitor.service" and "journalctl -xe" for details.
invoke-rc.d: initscript nut-client, action "start" failed.
● nut-monitor.service - Network UPS Tools - power device monitor and shutdown controller
   Loaded: loaded (/lib/systemd/system/nut-monitor.service; enabled; vendor preset: enabled)
   Active: failed (Result: protocol) since Mon 2021-04-19 20:34:41 CEST; 10ms ago
  Process: 31660 ExecStart=/sbin/upsmon (code=exited, status=0/SUCCESS)
      CPU: 2ms

Apr 19 20:34:41 server systemd[1]: Starting Network UPS Tools - power device monitor and shutdown controller...
Apr 19 20:34:41 server upsmon[31660]: upsmon disabled, please adjust the configuration to your needs
Apr 19 20:34:41 server upsmon[31660]: Then set MODE to a suitable value in /etc/nut/nut.conf to enable it
Apr 19 20:34:41 server systemd[1]: nut-monitor.service: Can't open PID file /run/nut/upsmon.pid (yet?) after start: No such file or directory
Apr 19 20:34:41 server systemd[1]: nut-monitor.service: Failed with result 'protocol'.
Apr 19 20:34:41 server systemd[1]: Failed to start Network UPS Tools - power device monitor and shutdown controller.
Apr 19 20:34:41 server systemd[1]: nut-monitor.service: Consumed 2ms CPU time.

Now check if the UPS is connected and recognized:

$ lsusb | grep UPS
Bus 001 Device 002: ID 0463:ffff MGE UPS Systems UPS

Bingo!

Vendor ID: 0463
Device ID: ffff

Configuration

We will now set up nut to check on the UPS.

In the file /etc/nut/ups.conf we add the following:

[eaton]
driver = usbhid-ups
port = auto
desc = "Eaton Ellipse ASR 750"
synchronous = "yes"

Let’s test

# this doesn't work
$ upsdrvctl start
-bash: upsdrvctl: command not found
# should be somewhere...
$ whereis upsdrvctl
upsdrvctl: /usr/sbin/upsdrvctl /usr/share/man/man8/upsdrvctl.8.gz
$ /usr/sbin/upsdrvctl start
Network UPS Tools - UPS driver controller 2.7.4
Can't open /etc/nut/ups.conf: Can't open /etc/nut/ups.conf: Permission denied

My UPS only has one “consumer”, which is my server. So NUT can perfectly be set to standalone mode.

In the file /etc/nut/nut.conf set:

...
MODE=standalone

With the driver configured, there are two services left: upsd and upsmon.

upsd, or the daemon will serve the UPS data to the clients and is listening on localhost, port 3493.

In the file /etc/nut/uspd.conf set:

...
# comment out this line:
LISTEN 127.0.0.1 3493
...

upsmon interacts with upsd it’s configuration is two-fold: access.

In the file /etc/nut/upsd.users set:

[upsmonitor]
password = ups_password
upsmon master

The daemon is listening and knows who to allow access. So now configure the monitor.

In the file /etc/nut/upsmon.conf add:

MONITOR eaton@localhost 1 upsmonitor ups_password master

Test

You could simply pull the plug. But first, make sure all services are running correctly!

$ sudo systemctl start ups-driver
$ sudo systemctl start nut-server
$ sudo systemctl start ups-monitor

A reboot may work too because the services are marked for auto-start.

You can check the installation with:

$ upsc eaton@localhost
battery.charge: 100
battery.charge.low: 30
battery.runtime: 3000
battery.type: PbAc
device.mfr: EATON
device.model: Ellipse 750
device.serial: BDCL440GL
device.type: ups
driver.name: usbhid-ups
driver.parameter.pollfreq: 30
driver.parameter.pollinterval: 2
driver.parameter.port: auto
driver.parameter.synchronous: yes
driver.version: 2.7.4
driver.version.data: MGE HID 1.39
driver.version.internal: 0.41
input.transfer.high: 264
input.transfer.low: 184
outlet.1.desc: PowerShare Outlet 1
outlet.1.id: 2
outlet.1.status: on
outlet.1.switchable: no
outlet.desc: Main Outlet
outlet.id: 1
outlet.switchable: no
output.frequency.nominal: 50
output.voltage: 230.0
output.voltage.nominal: 230
ups.beeper.status: enabled
ups.delay.shutdown: 20
ups.delay.start: 30
ups.load: 2
ups.mfr: EATON
ups.model: Ellipse 750
ups.power.nominal: 750
ups.productid: ffff
ups.serial: BDCL440GL
ups.status: OL
ups.timer.shutdown: -1
ups.timer.start: -10
ups.vendorid: 0463

Connection refused?

NUT runs as nut user and has no access to USB by default. So we have to create udev-rules to solve this.

Create a file /etc/udev/rules.d/90-nut-ups.rules with the following content. Mind the idVendor and idProduct we discovered in the first step.

# Eaton ellipse ASR 750
ACTION=="add", SUBSYSTEM=="usb", ATTR{idVendor}=="0463", ATTR{idProduct}=="ffff", MODE="0660", GROUP="nut"

The UPS is gone?

Sometimes the UPS driver fails and loses connection. That would be disastrous. After all, knowing how the UPS is doing is crucial for a gracious shutdown should the battery get low.

You can find out with this command:

$ /bin/upsc eaton | grep 'device.type'
Init SSL without certificate database
Error: Data stale

Data stale? Then reload the driver. While we could restart the basic sudo /lib/nut/usbhid-ups -a eaton this doesn’t help when the nut-driver service already failed. So we restart the service and the service will take care of restarting the underlying driver.

$ sudo systemctl restart nut-driver
Network UPS Tools - UPS driver controller (2.7.4)
USB communication driver 0.33
Network UPS Tools - Generic HID driver 0.41 (2.7.4)
Using subdriver: MGE HID 1.39
Duplicate driver instance detected! Terminating other driver!

# Now check again
$ /bin/upsc eaton | grep 'device.type'
Init SSL without certificate database
device.type: ups

Best to put this in a script and run it regularly.

Create a file ~/checkups.sh

#!/bin/bash

if [[ $(/bin/upsc eaton | grep 'device.type') == *ups ]]; then
  exit 0
else
  systemctl restart nut-driver
fi

The frequency to run this script depends on the size of the UPS and the power of your server. The power source of my server is rated 80W. The UPS is 750VA or 450W with a 12V/9Ah battery. The battery can deliver about 108VAh. On maximum load, it should theoretically last about one hour. So a safe interval of checking every 10 minutes should be more than enough. There is one thing we have to take care of though. The default value in upsmon.conf makes upsmon check the UPS every 5 seconds. When a UPS isn’t answering to the polls, it is marked “stale”. When this condition lasts for more than DEADTIME, the UPS is marked “dead”.

I chose a checking frequency of 60 seconds while on AC power and 10 seconds while on battery power.

Since we only check the USB connection every 10 minutes but poll every minute, or 10 seconds depending on the power source, we should set DEADTIME and NOCOMMWARNTIME to 600 seconds. This will avoid spamming the log when the USB connection to the UPS goes down.

In file /etc/nut/upsmon.conf:

POLLFREQ 60
POLLFREQALERT 10
DEADTIME 600
NOCOMMWARNTIME 600

While we could use systemd for this, crontab is more straightforward and equally reliable.

Cron is running out of the box on Debian 10 but it doesn’t hurt to check:

$ sudo systemctl start cron
● cron.service - Regular background program processing daemon
   Loaded: loaded (/lib/systemd/system/cron.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2021-04-12 17:44:11 CEST; 2 days ago
     Docs: man:cron(8)
 Main PID: 412 (cron)
    Tasks: 2 (limit: 4915)
   Memory: 190.7M
      CPU: 1min 58.452s
...

The script has to run as root since we don’t want to be asked for our sudo password when we’re low on power. So use $ sudo crontab -e instead of your user’s $ crontab -e.

$ sudo crontab -e

# add in the following
# the script will run every half hour, 5 past the hour and at 35 past
5,35 * * * * /home/<username>/checkups.sh 2>&1 | /usr/bin/logger -t checkups

I stop worrying about ungraceful shutdowns because of power outages.

Live test

I did a live test by pulling the plug on the UPS and checked the UPS’s parameters a few times.

$ upsc eaton@localhost | grep 'battery'
Init SSL without certificate database
battery.charge: 91
battery.charge.low: 30
battery.runtime: 2730
battery.type: PbAc

“I love it when a plan comes together.” - Colonel John “Hannibal” Smith -

Power meter photo by taner ardalı on Unsplash