I encountered them when I got a bill with a QR code on it that I could scan with my banking app. When I did it, it autopopulated all of the necessary fields (recipient, IBAN, amount, etc). I wondered how it all works and it turns out being much simpler than I thought.

I initially thought it must be some kind of URL that causes an API request to be made to populate the necessary data. But when I scanned the code with a regular QR scanner I noticed that it’s a simple text format:

BCD
002
1
SCT

Alice
DE89370400440532013000
EUR5

The lines mean the following and are quite self-explanatory:

1. Service Tag (always BCD)
2. Version (001 or 002)
3. Character encoding (1=UTF-8)
4. “Identification” (always SCT)
5. BIC of the recipient (optional in version 2 if inside the EEA)
6. Name of recipient
7. IBAN of recipient
8. Payment amount (optional)
9. Purpose (optional)
10. Reference (optional)
11. Reason for transfer (optional)
12. additional (optional) (note to the user of max. 70 characters)

All optional fields can be either left empty or completely omitted if they are not followed by other non-empty fields.

All in all, it’s a pretty simple and robust system. It’s mostly handy for organisations sending out bills. QR codes can be autogenerated and scanning one is simpler and less error-prone than entering the information manually. It can also be used by organisations soliciting donations.

It (currently) isn’t ideal for quickly sending money between friends, because there’s no “user-friendly” way of generating these codes. Theoretically mobile banking apps could generate them pretty easily, since they already have all of the information. The user would just have to optionally enter an amount. Combined with SEPA instant transfer (which all Eurozone payment providers now have to support), it would be a pretty simple, decentralised, and privacy-preserving way of sending money between friends. I’m not aware of any banking application that uses this scheme, but it would be quite easy to implement and a nice alternative to centralised payment providers (e.g. Paypal).

Full disclosure: I tried to use neomutt as an email client multiple times but got frustrated and gave up quite quickly. I was prepared for the eventuality that I would have the same experience with aerc, so I approached it with almost zero expectations. I’m glad to say that not only were my expectations more than met, I can tentatively say that I’ve enjoyed using this client more than any other client I’ve ever used. Whether it is better than any other email client remains to be seen.

This article will contain a quick rundown of the different features of the client and how I interacted with them. It tries to highlight how I personally experienced and used the program, so it will be highly subjective.

Documentation

The program has an inbuilt help feature that can be accessed via the command :help. The tutorial is a good place to start and succinctly describes the most important features and keybinds to get set up quickly. The other help pages describe different parts of the application and are written clearly and concisely. It makes it possible to use the client and all of its features without having to rely on any external source. A handy feature is that all of the help articles are just man pages that are piped to less. This means they are also installed as plain man pages (eg. aerc-tutorial (7)). This is a really nice way of shipping a “portable” help system inside your TUI applications! I hope other applications/projects start using this approach.

Account Configuration

Accounts are relatively easy to add, thanks to a simple and intuitive wizard. This wizard creates a new entry in the accounts config file (~/.config/aerc/accounts.conf). This file uses a simple INI syntax, so it is really easy to read and modify. For example, this is my config section for a private account:

[Sergeant Personal]
source        = imaps://user:password@mail.example.com:993
outgoing      = smtps://user:password@mail.example.com:465
default       = INBOX
from          = "User" <user@example.com>
copy-to       = Sent
cache-headers = true
folders-sort  = INBOX, Drafts, Sent, spambucket, Active, Waiting, Done
archive       = Archives
check-mail    = 1m
folder-map    = /home/user/.config/aerc/folder-maps/foo.map

Most of these options are self-explanatory, but there are a few that I want to draw special attention to. These are the options for folder management (folders-sort and folder-map)

folders-sort

This option is really simple: It allows us to override automatic sorting for certain folders. The specified folders are shown at the top of the folder list in the provided order. The rest are sorted alphabetically. This option should be in every email client. Sadly, I’ve never seen it in a graphical client (Thunderbird/Outlook) before.

folder-map

This is an amazing feature that solves a really annoying problem: different email providers have different folder structures. For example, Gmail puts everything in a folder named [Gmail], some providers put ‘Sent’ and ‘Drafts’ under ‘INBOX’, others don’t, etc. Since this folder structure is provided by the server, there is no way to change it (unless you administer the email server). Aerc has a really nice solution for this problem. It consists of a file with mapping rules, that maps server folders to displayed folder names. This makes for a flexible remapping system that works on single and multiple folders, allows removing prefixes from folders and much more. The examples from the man page show this feature quite well.

Remap a single folder:
Spam = [Gmail]/Spam

Remap the folder and all of its subfolders:
G = [Gmail]*

Remove a prefix for all subfolders:
* = [Gmail]/*

Remap all subfolders and avoid a folder collision:
Archive/existing = Archive*
Archive = OldArchive*

It’s hard to overstate how much I love this feature. It allows me to have a consistent folder structure that I can interact with in exactly the same way for each and every one of my accounts (regardless of the folder structure on the server). Consistent folder structures are especially important when you manage emails via a CLI instead of a GUI.

Email Viewing

My email viewing experience has been generally mediocre. This has nothing to do with aerc itself, but rather with HTML emails. I suspected that this would be non-ideal in a TUI client and that fact has been proven correct. In most multipart emails the plain text version is much less readable than the HTML version. It’s possible to grasp the meaning most of the time but sometimes it is just a garbled mess. Aerc has a few features to make this experience less bad. One option is viewing them through a HTML parser/browser (a combination of w3m and dante. This works well for some emails. The generated output is sometimes better than the plain text version of the email. As a last resort I just use :open to open them in my default web browser. The way aerc does this is actually quite nice: it uses custom filters. These are *nix pipelines that allow the message to be piped through the filter before being displayed. It has default filters for plain text (colorising rules), calendar entries and HTML emails. These filters can be set on mimetypes but can also be matched with specific headers. The configuration contains a few examples and shows the flexibility and power of this feature:

text/plain=colorize
text/calendar=calendar
message/delivery-status=colorize
message/rfc822=colorize
#text/html=pandoc -f html -t plain | colorize
text/html=html | colorize
#text/*=bat -fP --file-name="$AERC_FILENAME"
#application/x-sh=bat -fP -l sh
#image/*=catimg -w $(tput cols) -
#subject,~Git(hub|lab)=lolcat -f
#from,thatguywhodoesnothardwraphismessages=wrap -w 100 | colorize

Email Authoring

Email authoring is an absolute pleasure. I really like the fact I can now compose my emails in vim.

Address Book

The address book was something I struggled with quite a bit. Like many things, theoretically this is handled quite elegantly in aerc: it defers address book organisation to an external program. This is done by allowing the user to specify a search query to the address book software in question. The results of that query are then used for autocomplete in relevant header fields. The default program aerc uses for this is abook. My problems with this software were twofold: first (at least in my tests) it did not seem to support names with non-ascii characters. Since I frequently converse with German people, this made this software useless to me. Also, there wasn’t an easy way to get new addresses from the email program to abook, since it can not parse a complete email and take out all of the addresses. This might have been solvable with custom filters, but I didn’t want to put much time into this since it didn’t work with non-ascii characters anyway. The second piece of software I tried was aercbook. This had the same problem with non-ascii characters as abook, but it does support parsing all header fields for email addresses. I then found a third option, called emailbook-janet. This is a rewrite of aercbook that is unfortunately not as performant, but it solves the ascii problem. Using keybinds, one can even select emails, parse them and automatically add all addresses to the address book:

aa = :pipe -m emailbook.janet /home/biggs/.config/aerc/address_book --parse --all<Enter>

One limitation of this software (and aercbook) is that they are not really “address books”. They simply save the name and email of contacts, to make autocomplete work. This is all I need at the moment, so that works out. If I ever need a more powerful solution, I might switch over to khard or some similar solution. Who knows, I might even set up an LDAP Server :)

PGP Support

One big problem I’ve had with Thunderbird is its strange PGP/GPG support. The biggest annoyance is its separate keychain. This makes using Thunderbird with something like a smartcard finicky at best and impossible at worst (at least, I’ve never got it to work properly). Aerc also has its own keychain, but it is entirely optional. The default mode is to just use the GPG keychain. You can set a specific key ID to sign messages, or just let aerc look up the key by email. There are options for automatically signing all emails and to use opportunistic encryption. All in all, as long as you have a working GPG setup, this works flawlessly.

General Configuration

Here are some general features that did not really fit into a separate category, but that I wanted to highlight nonetheless.

The first is that aerc supports signatures. Hardly a surprising or groundbreaking feature for an email client, but nice nonetheless. Signatures are just text files that can be included for each account with signature-file=. Alternatively, aerc can also execute an external command to create a signature.

Aerc also supports email templates. I haven’t looked at those in detail yet, but it seems like a nice feature.

It also tries to intelligently remove repetitions of ‘Re:’ prefixes in email headers (in multiple languages). To accomplish this, it uses a regex that can be optionally overwritten or extended.

Two other nice (but kinda default) features of aerc are its forget subject and attachment reminder. The first just checks if the subject is empty, but the second one is actually quite clever, and very easily extensible: it’s just a regex! Specifically, this one:

^[^>]*attach(ed|ment)

This does several smart things, one of them is filtering the line out if it begins with a ‘>’ (which would mean that someone else mentioned an attachment, not you). Because I also write a lot of emails in German, I decided to extend this regex so it also works in German (NB: the regex is case-insensitive and multi-line by default):

^[^>]*(attach(ed|ment))|an(ge)?h[aä]ngt?

All in all, I’ve really enjoyed using aerc over the last few days. It seems really polished and it is a pleasure to use. I think I’ll stick with this email client for a while longer and see how it goes.

For me notifications should ideally have the following properties:

• Stay somewhere until they are manually dismissed (even after a reboot)
• Not be on the screen all the time
• Be sorted in reverse chronological order
• Be easily accessible (via a shortcut or the GUI)

Some OSes get this down quite well. One that comes to mind is Windows 10/11. The one that does this the best (in my opinion) is Android (looking at you, Apple. Whats up with the weird notification pop-ups???). The current state on Linux is a far cry from that reality.

When I found about Rofication I thought this tool was written for me personally. The tool is quite bare-bones, but it brings everything that I need to make a functional “notification center” in Sway.

My Setup

Rofication consists of 2 parts, the daemon and an example GUI client. After installing it, using it in sway is just the matter of autostarting it and adding an appropriate keybind:

exec rofication-daemon.py
bindsym $mod+n exec rofication-gui.py

After that we are ready to receive and view our notifications. They are persisted across reboots and can be interacted with via different key bindings (eg. Alt+s to dismiss).

This gets us up and running, but are missing one more component: a widget that shows us how many notifications we have. Ideally, it should also change color if we have notifications, so that we can ascertain this at a glance.

One part of this is already provided by the original author, in the form of rofication-statusi3blocks.py. This simply prints out the number of notifications and could be easily integrated into a widget. To make our widget change color depending on the number of notifications, we need to extend this a bit. The bar I’m using (Waybar), expects a simple JSON object in the following form:

{
  "text": "$text",
  "alt": "$alt",
  "tooltip": "$tooltip",
  "class": "$class",
  "percentage": "$percentage"
}

NB: Waybar expects this to be in a single line, I just formatted it this way for demonstration purposes.

To make this work, we pass the name of a CSS class, and change the waybar config to make that class use a specific colour. Using jq, it is quite simple to “extend” the original script to output JSON instead:

#!/usr/bin/env sh

rofication-statusi3blocks.py | \
    jq --unbuffered \
    --compact-output \
    '{"text": .,"alt": ., "tooltip": "notifications", "class": (if . > 0 then "multiple" else "none" end)}'

After that, we can add a new module to our waybar-config:

{
    "custom/rofication": {
        "format": "{} ",
        "exec": "notif",
        "interval": 5,
        "return-type": "json",
        "on-click": "rofication-gui.py"
    }
}

Add the appropriate CSS code as well:

#custom-rofication.multiple {
    background-color: #eb4d4b;
}

After that we should have a working setup!

I’ve been using this setup for quite a while now and I’m really happy with it. It is pretty much an ideal notification setup for me. The only thing I’m missing is a timestamp for the specific notifications. Maybe this is possible, but I have not found an option to enable this yet.

Hope you enjoyed reading this article!

To play this game in multiplayer, you need a dedicated server. No problem I thought, I’ll just set it up quickly. This decision sent me on a 3-day trip of trial, despair and ultimately, success. I’ll use this blog post to document the problems I went through, as well as to generally muse about the state of this game.

Note: This article might be a bit harsh at times. It is not intended to disparage any single individual that was involved in the development of this game. Also keep in mind that the game is in its early stages, so we will hopefully see some improvement in these areas.

I first installed this game using steamCMD. After that was done, I started configuring the service with systemd. Usually, the first thing I look at, is how the program is started normally. The server version of Project Zomboid is a Java application that is started by a bash script.

#!/bin/bash

INSTDIR="`dirname $0`" ; cd "${INSTDIR}" ; INSTDIR="`pwd`"

if "${INSTDIR}/jre64/bin/java" -version > /dev/null 2>&1; then
        echo "64-bit java detected"
        export PATH="${INSTDIR}/jre64/bin:$PATH"
        export LD_LIBRARY_PATH="${INSTDIR}/linux64:${INSTDIR}/natives:${INSTDIR}:${INSTDIR}/jre64/lib/amd64:${LD_LIBRARY_PATH}"
        JSIG="libjsig.so"
        LD_PRELOAD="${LD_PRELOAD}:${JSIG}" ./ProjectZomboid64 "$@"
elif "${INSTDIR}/jre/bin/java" -client -version > /dev/null 2>&1; then
        echo "32-bit java detected"
        export PATH="${INSTDIR}/jre/bin:$PATH"
        export LD_LIBRARY_PATH="${INSTDIR}/linux32:${INSTDIR}/natives:${INSTDIR}:${INSTDIR}/jre/lib/i386:${LD_LIBRARY_PATH}"
        JSIG="libjsig.so"
        LD_PRELOAD="${LD_PRELOAD}:${JSIG}" ./ProjectZomboid32 "$@"
else
        echo "couldn't determine 32/64 bit of java"
fi
exit 0

This script seems to be written by someone that is not very familiar with bash/shell scripting. There are several anti-patterns here (re-assigning a “constant”, using exit 0 at the end of the script, no error handling, using backticks, no double quotes, etc). Many of these errors are picked up (and optionally automatically fixed) with shellcheck --diff start-server.sh:

--- a/start-server.sh
+++ b/start-server.sh
@@ -7,7 +7,7 @@
 #
 ############

-INSTDIR="`dirname $0`" ; cd "${INSTDIR}" ; INSTDIR="`pwd`"
+INSTDIR="$(dirname "$0")" ; cd "${INSTDIR}" || exit ; INSTDIR="$(pwd)"

 if "${INSTDIR}/jre64/bin/java" -version > /dev/null 2>&1; then
        echo "64-bit java detected"

What’s stranger is the way this script detects the java version. It does not actually parse the output of the version information, even though it executes the relevant command (java -version). Instead, it checks where Steam installed the bundled JRE and uses the name of the path as an indication for the version.

This script could be replaced by a systemd service completely, but I decided to keep it for now. Time will tell whether this is a good idea.

systemd

I created a simple systemd service file to execute the server. The first problem presented itself immediately: The application asked for the administrator password on stdin. I reran the binary without systemd to be able to enter this password.

After creating the service and adding some hardening options, I found out after some research that the server does not implement signal handling. The developers explicitly say that running the binary in the foreground is the only supported method, since the server can only be shut down safely using the console commands. To circumvent this, I decided to create a systemd socket that creates a named pipe. This pipe is connected to the stdin of the process. This means I can send commands to the server to manage it, even when it is run via systemd. The socket file looks like this:

[Socket]
ListenFIFO=%t/zomboid.sock
SocketMode=0660
SocketUser=zomboid
SocketGroup=games
RemoveOnStop=true

[Install]
WantedBy=sockets.target

The corresponding options in the service file are:

[Unit]
Description=Zomboid Headless Server
After=network-online.target
Requires=zomboid.socket

[Service]
WorkingDirectory=/srv/zomboid/game
ExecStart=/srv/zomboid/game/start-server.sh
Type=exec
User=zomboid
StandardInput=socket
StandardOutput=journal
StandardError=journal
Sockets=zomboid.socket
ExecStop=/bin/sh -c "echo quit > /run/zomboid.sock"

By modifying the ExecStop= option, we can make sure that the server is stopped cleanly when using systemctl stop zomboid.service.

The developers have signalled (no pun intended) that they want to fix this issue, which is very commendable. I hope a fix is implemented soon.

Logging and logfiles

The strange decisions continue when it comes to logging. I did not expect the application to honor all best practices, but I did expect some kind of standards compliance, or at least configurability.

The application does 2 types of logging: it outputs messages to stdout. This is reasonable, but the logging format itself is very strange:

[03-11-23 14:20:55.140] LOG  : Multiplayer , 1699017655140> 1,021,965,239> [MPStatistics] mem usage notification threshold=8,160,437,760.
[03-11-23 14:20:57.820] LOG  : Network     , 1699017657820> 1,021,967,919> [03-11-23 14:20:57.820] > ZNet: SSteamSDK -> SZombienet: OnPolicyResponse.
[03-11-23 14:20:57.821] LOG  : Network     , 1699017657821> 1,021,967,920> [03-11-23 14:20:57.821] > ZNet: OnPolicyResponse.
[03-11-23 14:20:57.821] LOG  : Network     , 1699017657821> 1,021,967,921> [03-11-23 14:20:57.821] > ZNet: SZombienet -> SSteamSDK: BSecure.
[03-11-23 14:20:57.822] LOG  : Network     , 1699017657822> 1,021,967,921> [03-11-23 14:20:57.822] > ZNet: Zomboid Server is VAC Secure.

I don’t know much about logging facilities in Java, but this is horrible. It does not adhere to the syslog standard, the date format is strange, the default severity is called LOG, it uses fixed width fields, each line has three timestamps in different formats (some lines even have four).

In my opinion, this logging system would be improved by just using System.out.println(). At least then formatting could be handled by the logging daemon and messages would be actually readable.

I know that logging is complicated topic, but this solution seems to use the worst of both worlds.

The second problem is that the application also writes this information to several log files in a custom folder (Logs/). There is no way to turn this off or change the path. The logs have strange file names (03-11-23_14-20-04_DebugLog-server.txt) and are rotated by the application itself (!) using a strange algorithm: the application creates folders for each day and log files are moved into it after each application restart. This makes rotating the logs using something like logrotate supremely difficult, if not entirely impossible.

For this reason, I decided to simply discard these log files. The messages are saved in the systemd journal anyway, I simply do not need these files. The first solution I thought of is using something like nullfsvfs (eg: “/dev/null as a filesystem”). This seemed very interesting, but ultimately overkill. I decided to use the simpler, sledgehammer solution:

Deleting all files in Logs/ every time the service is stopped.

This can be done via one line in the systemd service:

ExecStopPost=/bin/sh -c "rm -rf /srv/zomboid/Zomboid/Logs/*"

Logs that don’t follow a single event stream and are not rotateable are useless at best and a risk (disk space) at worst. I really hope the developers think about implementing logging in a more standards-compliant and sensible way. Adhering to the twelve factor guidelines on logging should be a huge improvement. If this is not possible, at least allow users to choose where they want logs to go (eg: console, file, etc). (This would contradict the 12 Factor App Guidelines, but it would at least be better than the status quo. Also logfile names should not have a timestamp (that’s what filesystem metadata is for). Thirdly, don’t try to rotate logs yourself. Let system administration utilities (eg: logrotate, journald) handle this for you.

Complete file

In conclusion, this is the complete systemd file for the service:

[Unit]
Description=Zomboid Headless Server
After=network-online.target
Requires=zomboid.socket

[Service]
WorkingDirectory=/srv/zomboid/game
ExecStart=/srv/zomboid/game/start-server.sh
User=zomboid
StandardInput=socket
StandardOutput=journal
StandardError=journal
Type=exec
Sockets=zomboid.socket
ExecStop=/bin/sh -c "echo quit > /run/zomboid.sock"
ExecStopPost=/bin/sh -c "rm -rf /srv/zomboid/Zomboid/Logs/*"

# hardening
ProtectControlGroups=true
RestrictSUIDSGID=true
ProtectClock=true
ProtectHome=true
ProtectKernelLogs=true
ProtectKernelModules=true
ProtectProc=invisible
RemoveIPC=true
PrivateDevices=true
ProtectHostname=true
NoNewPrivileges=true
CapabilityBoundingSet=
RestrictNamespaces=true
SystemCallArchitectures=native
LockPersonality=true
RestrictRealtime=true
SystemCallErrorNumber=EPERM
UMask=0077
RestrictAddressFamilies=AF_INET AF_INET6
PrivateUsers=true
PrivateTmp=true
ProtectKernelTunables=true
MemoryDenyWriteExecute=false
SystemCallFilter=@system-service
SystemCallFilter=~@resources
SystemCallFilter=~@privileged
PrivateNetwork=false
ProtectSystem=strict
ReadWritePaths=/srv/zomboid/Zomboid

[Install]
WantedBy=multi-user.target

With the hardening options active, we can get it down to quite a nice score:

→ Overall exposure level for zomboid.service: 1.1 OK 🙂

Networking

After starting the service first tries to open its ports on the router via UPnP. Another anti-feature in my opinion: UPnP should be disabled by default and left as a concious choice for the users that really want it (in my opinion, you should never actually want/need UPnP, so it might even be better to disable this “feature” completely).

Since my router does not implement UPnP, I disabled it and opened the requisite ports in my firewall (16261 and 16262).

I did not want my server to appear in the games global server browser, so I decided to create a DNS record that points to the server. I could then tell my friends to use that name when they want to connect. Imagine my surprise when I tested this, and the client still could not connect to the server. After some digging I found the following:

ss -unlp | grep 1626

UNCONN 0      0                          0.0.0.0:16261      0.0.0.0:*    users:(("ProjectZomboid6",pid=1660231,fd=40))
UNCONN 0      0                          0.0.0.0:16262      0.0.0.0:*    users:(("ProjectZomboid6",pid=1660231,fd=42))

The application only supports IPv4. To remedy this, I created a custom record that only resolves my IPv4 address. With this in hand I tried again, to no avail. Turns out, the client does not support name resolution at all.

At this point I gave up trying to find a nice solution and instead chose a solution. I told my friends to do a nslookup/dig of my IPv4-only record and that they should use the output of that in the client.

I would urge the developers to implement IPv6 and support name resolution. This should really make the user experience better for servers that can’t or don’t want to be in the global server browser.

User management

User management is another strange beast. A specific server has local users that are not connected to the steam users at all. These users have a username and a password. There is also an optional server password. To limit access, we can either configure a server password or configure whitelisting. If there is a whitelist, the server administrator must add all of the users and their passwords beforehand. As far as I can tell, there is no way for the user to change their password later. Since this is a huge hassle, I decided to simply create a server password and disable whitelisting.

Conclusion

First of all, a big thank you to the developers of Project Zomboid. I very much look forward to playing it. It looks like a nice game and I am glad that the server is now working. This was way harder than it needed to be though, and I wonder how people that have less technical knowledge than me could set this up sensibly. I really hope the developers put some time and effort into these things. These are fundamental problems, that fortunately should have easy solutions. By programming your applications according to relevant standards, you can save yourself a lot of hassle in the long run.

This post is about a song, more precisely two songs. It is about Aïcha by Khaled and the English-language cover of that song by Outlandish. Although the version by Outlandish is a cover, the meaning of the song is radically transformed because of musical and lyrical choices. To understand what I mean we first have to understand a few things about the original song and about its genre, Raï. I will also briefly introduce Outlandish.

Raï and Khaled

Raï is a form of Algerian folk music that originates in Oran. Going into all of the historical development of this genre and explaining all sociological aspects is something that I don’t have any competency for. For the sake of this article, a few things are important. Raï is associated with a countercultural movement in Algeria that often clashed with established morals and societal expectations. The lyrics are often political and often critical of social problems. They also contain elements that Algerian society at that time considered raunchy or vulgar. Musically it is a synthesis of traditional Islamic, traditional secular and western music. This creates a very distinct sound.

Khaled is the personification of modern Raï. He was particularly popular in the 80s and 90s, and still enjoys success in France and Algeria.

Aïcha itself was written Jean-Jacques Goldman. There are two versions, a French version and a bilingual version (French and Arabic). The Arabic parts were written by Khaled himself.

Outlandish

Outlandish is a Danish hip-hop group. When the song was written, they consisted of three members: Waqas Ali Qadri, Lenny Martinez and Isam Bachiri.

Analysis

If you haven’t already please take a moment to listen to both songs right now and to read the lyrics. After you’ve done that, continue reading. I will start with a short musical comparison, and then get to the actual meat of this post: a comparison of the lyrics.

The cover takes some musical elements from the original, but they are a distinct genre. While the original is a Raï song, the cover is a mixture R&B and pop. Musically, I prefer the original but I don’t really have the musical talent to talk about interesting differences between the songs.

NB: For the sake of brevity, I won’t translate the french parts of the song. If you don’t understand, either learn french or try one of the numerous automatic translation providers.

Lyrically, the songs are radically different. To highlight this, let’s take the time to go through the lyrics of both of them. We’ll start with the original.

The Original

Comme si je n'existais pas
Elle est passée à côté de moi
Sans un regard, reine de Saba
J'ai dit Aïcha prends tout est pour toi
Voici les perles, les bijoux
Aussi l'or autour de ton cou
Les fruits bien mûrs au goût de miel
Ma vie Aïcha si tu m'aimes

J'irai où ton souffle nous mène
Dans les pays d'ivoire et d'ébène
J'effacerai tes larmes, tes peines
Rien n'est trop beau pour une si belle

Here we have a narrator that describes our titular Aïcha, who seems to be quite a special person. She is compared to the biblical and mythical Queen of Sheba. This comparison is compounded by references to ivory and ebony wood, which are products that are produced in the places our queen is supposed to hail from. The narrator promises her all kinds of expensive material gifts: pearls, gold, et cetera.

Je dirais les mots les poèmes
Je jouerais les musiques du ciel
Je prendrais les rayons du soleil
Pour éclairer tes yeux de rêves

In the second part of the song, the narrator now shifts his focus to immaterial courtship. He will write her poems, sing her songs, even catch sunlight for her. After this part we get a musical and lyrical shift, as the titular Aïcha suddenly starts answering our narrator.

Elle a dit garde tes trésors
Moi je vaux mieux que tout ça
Des barreaux sont des barreaux même en or
Je veux les mêmes droits que toi
Et du respect pour chaque jour, moi je ne veux que l'amour

This twist clearly illustrates the theme of the song. It is a song about love and courtship, but also about freedom and the social status of women in MENA countries. What is important to Aïcha is not being wooed with material or immaterial gifts. She plainly states she wants the same rights as men, in a relationship but also in society in general. With this, the song manages to be subversive and exemplifies the kind of social commentary that is typical of its genre. The Aïcha in this song is strong-willed and tells us exactly what she wants.

The Cover

Without further ado, let us delve into the cover version. The first part is already quite different from the original.

So sweet, so beautiful
Everyday like a queen on her throne
Don't nobody knows how she feels
Aïcha, lady one day it will be real (let's do it)
She moves, she moves like a breeze
I swear I can't get her out of my dreams
To have her shining right here by my side (shining like a star)
I'd sacrifice all them tears in my eyes

Apart from leaving out the specific reference to the queen of Sheba, one thing is immediately apparent: the perspective of this song is different. We no longer have a narrator that talks to Aïcha, but one that describes her.

She holds her child to her heart
Makes her feel like she is blessed from above
Falls asleep underneath her sweet tears
Her lullaby fades away with his fears

The second part starts deviating completely from the original. Gone are any references to promises or courtship. Instead, Aïcha is here described as a mother with a child, a far cry from the image of Aïcha in the original. After that, we arrive at the same musical shift as in the original, but with radically different imagery:

She needs somebody to lean on
Someone body, mind & soul
To take her hand, to take her world
And show her the time of her life, so true
Throw the pain away for good
No more contemplating boo
No more contemplating boo

Gone is Aïchas rejection of courtship. Gone is the social commentary. Gone is the specific reference to women’s rights. Also, markedly:

Gone is Aïchas voice.

She is not the one speaking. She has no active part in this song. Instead we are being told by the narrator what she (supposedly) wants: someone to lean on, someone to take her hand. Someone to show her the time of her life. This Aïcha needs to be taken care of, for she certainly can not take care of herself. This is a far cry from the powerful woman in the original, who actively demands things from the narrator. Khaleds Aïcha is active. Outlandishs Aïcha is entirely passive.

Outlandish manages to quite successfully subvert the central themes of the song: longing, freedom and equality. The song is reduced to a much simpler theme: a desirable but unhappy woman, who needs love to be happy.

The crowning jewel of the song can be found in the last verse. Here the narrator tells us what is desirable about her.

Lord knows the way she feels
Everyday in his name she begins
To have her shining here by my side
I'd sacrifice all them tears in my eyes

This verse can be seen as the ultimate conclusion of the song. What makes our Aïcha most desirable? Her devotion to God of course! The song ends with an explicit reference to God and the difference to the original could not be more marked. We have gone from a secular song written by a french Jew and performed by an Algerian Raï singer to something that could be found on an evangelical hip-hop sampler album.

Not surprisingly all three members of Outlandish describe themselves as religious.

To make something clear: I have nothing against religion, or religious music for that matter. I consider myself a Christian and have even been known to listen to some Christian music from time to time. But this cover is something else. It does not feel like a cover. It feels more and more like a deliberate subversion of the themes of the original. It makes me genuinely angry every time I think about it.

I hope you enjoyed my little analysis of this song. This thing has been on my mind for quite a while and I’m glad I finally got it out. If this wasn’t your cup of tea, I’m pretty sure that the next article will be more technical.

Normally, I encrypt my main root partition with LUKS while my boot partition stays unencrypted. I do this because encrypting the boot partition has downsides: it only supports LUKS 1 which would mean entering 2 passphrases. Not encrypting the boot partition means being open to evil maid attacks. But even encrypting the boot partition does not save one from this vector, because the ESP stays unencrypted. Secure boot solves this problem: it only allows booting operating systems that were signed and makes sure that the “boot chain” is secure. This does not prevent all attacks, but provides a reasonable hurdle for most use cases.

Configuring secure boot enables another convenience feature: unlocking the root partition with a key from the TPM rather than a passphrase. To do this, we need to configure the following components:

• systemd-boot
• mkinitcpio
• secure boot
• systemd-cryptenroll

systemd-boot

NB: This part is not strictly necessary, I think it would theoretically be possible to configure GRUB to do the same thing, but it “fits” well into the rest of the setup.

sudo bootctl install

After installing it, we need to add some configuration. First, we edit the file /boot/loader/loader.conf.

timeout 0
console-mode max
default arch.conf
editor no

Then we create the default boot entry (arch.conf).

title   Arch Linux
linux   /vmlinuz-linux
initrd  /intel-ucode.img
initrd  /initramfs-linux.img
options root=/dev/mapper/root rw loglevel=3 quiet splash

To update our Bootloader every time the kernel or systemd is updated, we have 2 options. The first option is using systemd-boot-update.service. This updates the bootloader on the next reboot. This option won’t work for our setup, since we need to generate and sign the boot loader files before we reboot. To solve this issue, I installed systemd-boot-pacman-hook (AUR). There are other solutions for this problem, but this seemed the simplest. It also works well with the workflow of sbctl that I will go into later in this article.

mkinitcpio

After installing the bootloader, we need to make sure that our disk can be decrypted. We do this by configuring the initramfs. I will be using the default tool under Arch Linux (mkinitcpio). If you’re using dracut the setup should be similar, but I have not tested it.

There are 2 ways of decrypting a drive with mkinitcpio: the encrypt or the sd-encrypt hook. The sd-encrypt hook uses systemd, and it is the one I will be using. To configure this hook, first we need to configure /etc/crypttab.initramfs. This file gets added to the initramfs and contains the devices that should be decrypted by systemd-cryptsetup-generator:

root	UUID=<label>	none	tmp2-device=auto

The last column specifies that we want this drive to be automatically unlocked using a key stored in the TPM. If you want to use a passphrase, omit the last two columns.

We also need to set the correct hooks. To do that, we edit /etc/mkinitcpio.conf and change the HOOKS variable to the following:

HOOKS=(base systemd autodetect modconf kms keyboard sd-vconsole block sd-encrypt filesystems fsck)

Secure boot

This is the most finicky party of the setup, and the exact way of doing this depends on your hardware manufacturer/UEFI implementation. To use secure boot, there are two strategies: using your own keys, or using a signed bootloader. Most distros take the second route. I will be implementing the first one.

Note: Doing this can brick your device. Please make sure you understand what you are doing and don’t blindly follow the instructions in this article.

It’s a good idea to back up the variables, in case something goes wrong. These can then be restored later, if your UEFI supports it.

for var in PK KEK db dbx ; do efi-readvar -v $var -o old_${var}.esl ; done

After that, we need to put the UEFI in “Setup Mode”. This mode is activated when the Platform Key (PK) is deleted. After deleting the key and rebooting, we can generate our own keys with sbctl.

sudo sbctl create-keys

After that, we can enroll these keys into the UEFI. The -m also enrolls Microsofts keys. This is necessary on many motherboards, since they use these keys to sign their firmware. Only leave this option out if you know what you are doing.

sudo sbctl enroll-keys -m

After that, we can sign our kernel image and boot loader.

sudo sbctl sign -s /boot/vmlinuz-linux
sudo sbctl sign -s /boot/EFI/systemd/systemd-bootx64.efi

Since I use fwupd I signed that one as well.

sudo sbctl sign -s /boot/EFI/Arch/fwupdx64.efi

After they are added once, the files will be automatically resigned every time the kernel or systemd are updated. This is done via a pacman hook. For that reason, we installed systemd-boot-pacman-hook earlier: this automatically regenerates systemd-bootx64.efi when systemd-boot is updated. After that, it will be signed by the pacman hook of sbctl.

systemd-cryptenroll

Before we generate the key, we need to think about when we want systemd to use the TPM. This is configured using so-called Platform Configuration Registers (PCRs). One of these is the secure boot state (7). This means systemd will only unlock the drive if secure boot is enabled and no secure boot errors have been reported. This is the default setting. Some guides on the internet also use PCR 0 (firmware). To me personally, this does not make much sense. I’m not an expert, but I would think that we have to inherently trust the firmware anyway, since a malicious firmware could theoretically also falsely report PCR 0. If anyone has a good answer to this, feel free to contact me.

In the meantime, I’ve decided to not use PCR 0 since it would be quite annoying (rebinding every time the firmware is updated).

With that discussion out of the way, we can generate our key.

sudo systemd-cryptenroll --tpm2-device=auto /dev/sdX

If you’re paranoid, you can add a PIN as well with --tpm2-with-pin=true.

It’s also sensible to generate a recovery key and store it somewhere safe. This can be used when systemd can not decrypt the drive from the TPM.

sudo systemd-cryptenroll --recovery /dev/sdX

I’m very happy with this setup. This problem has been in the back of my mind for some years now, and I’m glad I finally tried to solve it. I will probably be using this setup on all of my future Linux machines.

The first solution that came to mind were alternative Youtube frontends. I found a list with different options that I had a look at. Most of these solutions support accounts. That would mean I could create an account on their public instance, and then use that instance to watch the videos. This seemed like a nice idea at first, but in such cases I’m more at ease with self-hosting. Most of these services support that though. I started diving into the documentation of these different services and the more I did, the less I became interested. In general these services are very resource-heavy, requiring about 4G of RAM. They are also frequently buggy. One service claims that to function: “[the application] must be restarted often, at least once a day, ideally every hour.” That doesn’t inspire much confidence. Another service only runs as a docker container, which is something I try to avoid as much as possible.

At this point I realised I was falling into a rabbit hole, and getting nowhere. At this point, I asked some of my friends and one of them pointed me towards this:

https://www.youtube.com/feeds/videos.xml?channel_id=UCyhnYIvIKK_--PiJXCMKxQQ

If we open that link, we get an Atom feed. These feeds can be plugged into an RSS reader, which gives a great overview over all the channels that I’m interested in. There’s just one tiny hitch: we need the channel ID. This is not the same thing as the name of the channel and isn’t “shown” normally. Because I didn’t need to find the ID of many channels I just ended up using a web tool to get the IDs. This worked pretty well. Apparently, there are other ways, as outlined in this SO discussion.

After gathering all channel IDs, we just have to plug them into an RSS reader. Since I hadn’t used one for quite a while, I took this opportunity to find a new (preferably TUI-based) one. I started my search with the excellent List of applications on the Arch Wiki and found canto. From their website:

“Canto is an Atom/RSS feed reader for the console that is meant to be quick, concise, and colorful. It’s meant to provide a minimal, yet information packed interface. No navigating menus. No dense blocks of unreadable white text. An interface with almost infinite customization and extensibility using the excellent Python programming language.”

That sums it up pretty well. I won’t go through all of the features Canto has, because most of them are pretty standard for an RSS/Atom reader, but I want to highlight the specific features that make it ideal for this use case. Canto allows us to set a default browser to open links, and also has a built-in reader, but its killer feature is its so-called “smart link plugin”. This plugin makes it possible to open specific file types or URLs with specific programs. This means I can use canto to open the Youtube entries with umpv, and all other entries with my normal browser. To make this possible, we just need to copy the plugin into our configuration directory, and change some settings:

cp /usr/lib/canto/plugins/smartlink.py ~/.config/canto/plugins/
vim ~/.config/canto/plugins

To change the settings, we just change the HANDLERS variable.

HANDLERS = [
    { "match-file" : "image data",
      "handler" : "feh",
    },
    { "match-file" : "PDF",
      "handler" : "evince",
    },
    { "match-url" : "youtube.com",
      "handler": "umpv",
    }
]

Now we can press f on an entry to open it with umpv, which adds it to our mpv queue. If we press g, the entry is opened in the browser we configured.

I’ve been using this setup for a few days now, and I’m very happy with it. It is exactly what I wanted: lightweight, simple, uses mpv for playback and terminal-based.

So, now we have to power of Lua at our disposal, how can we deal with this problem?

Our web application sends a request in the following format:

{
  "Username": "admin",
  "Pw": "secretpassword"
}

To extract the information, we use lua-resty-reqargs. This takes the request, and returns three values (get, post, and files). These are presented as lua tables. We can then look at the values inside the post table to get our Username value. If we find a certain string inside our variable, we send the client a 403.

local get, post, files = require "resty.reqargs"
ngx.status = ngx.HTTP_OK
local user_normalised = post.Username:lower()

if string.find(user_normalised, "admin") then
    ngx.status = 403
    ngx.exit(ngx.HTTP_FORBIDDEN)
end

Our Nginx config has the same location block, with the “jump” to our @with_admin pseudo-location.

location /authenticate {
    error_page 403 = @with_admin;
    access_by_lua_file conf/auth/application.lua;
    proxy_pass http://application;
}

location @with_admin {
    allow 192.168.1.0/24;
    allow 192.168.10.0/24;
    deny all;
    proxy_pass http://application;
}

Pretty easy, huh? I look forward to all the additional features OpenResty has to offer.

The simplest example is a web application that exposes its admin interface on a single URI. This is easy, because we can just define that URI as a location block, and then put the specific security options in that block.

upstream application {
    server unix://var/run/application.socket;
}

server {
    server_name application.example.com;

    location /admin/ {
        allow 192.168.1.0/24;
        deny all;
        proxy_pass http://application;
    }

    location / {
        proxy_pass http://application;
    }

}

This works very well if your application only exposes admin access on a certain endpoint. Most applications don’t do this. Frequently they will have separate login and admin panel URLs. So even if you protect the admin panel, login as an administrative user is still possible.

To solve this problem, we first have to create a location block for the login endpoint. In this location block we disallow logging in from the internet for administrative users. To figure out how to do that, we have to understand the login process.

Most web applications will use a POST request with application/x-www-form-urlencoded. In this case, the body of the HTTP message is a string and contains all information as key-value pairs, separated by ampersands (&). In the case of a login, this is something like:

userName=admin&password=rosebud

We need the name of the key to parse it into Nginx. The best way of doing that is with the developer tools of your favorite browser. Use the Network tab while logging in, and just look at the request body of the relevant POST request (this is also a good way to get the URL for your location block).

Nginx can’t natively parse the bodies of these requests, but there is a module that can. Please refer to the GitHub page for installation instructions.

This module is super easy to use. We just specify the key of the value we want to have, and the module will save it in an Nginx variable.

set_form_input $userName;

After we have parsed the correct variable, we just need to determine if someone is trying to log on as our admin user. We can do that with an if statement.

NB: if statements are notorious in Nginx as something you need to be careful about: there is even a whole article about how if is evil in location contexts. If you want to fully understand why I do what I do, please read that piece of documentation.

tldr; it’s fine to use it in location contexts if there’s no alternative module, and if we only return to another location in the body of the if statement.

This is realised by defining our target location as an error page. We then return that error. In my first version, my location block looked like this:

location /user/login {
    set_form_input $userName;

    error_page 403 = @with_admin;
    if ($user_name = admin) {
        return 403;
    }
    proxy_pass http://application;
}

This location blog has a problem though: can you figure out what it is?

The problem is that this will allow an attacker to put some whitespace around the string. This will not match our string, and since most web applications strip whitespace in login fields, the login will still work.

Thanks to Besen for spotting this problem!

It’s better to use substring matching. This will stop an attacker from using any weird characters before or after the string, that might be stripped by the application itself:

location /user/login {
    set_form_input $userName;

    error_page 403 = @with_admin;
    if ($user_name ~ admin) {
        return 403;
    }
    proxy_pass http://application;
}

After that we just need to define a location block that our return statement will jump to. In that block we define our security settings, and pass the credentials on to our proxied application.

location @with_admin {
    allow 192.168.1.0/24;
    deny all;
    proxy_pass http://application;
}

As a recap, the full configuration will look something like this:

upstream application {
    server unix://var/run/application.socket;
}

server {
    server_name application.example.com;

    location /admin/ {
        allow 192.168.1.0/24;
        deny all;
        proxy_pass http://gitea;
    }

    location /user/login {
        set_form_input $userName;

        error_page 403 = @with_admin;
        if ($user_name ~ admin) {
            return 403;
        }
        proxy_pass http://application;
    }

    location @with_admin {
        allow 192.168.1.0/24;
        deny all;
        proxy_pass http://application;
    }

    location / {
        proxy_pass http://application;
    }
}

In that way, we protect our admin interface and only allow logging in as the admin user from a LAN.

This will not work for all web applications. Some applications use JSON to communicate with their backend. As far as I know Nginx does not have a native way to deal with this. To deal with this, we will use OpenResty. This is an application that turns Nginx into a lua application server. I will write about this topic in a future blog post, so stay tuned if this is something that interests you!

Binding modes deactivate all normal bindings when switching to them. Only the bindings defined in the binding mode are active.

set $mode_launcher "Start: [f]irefox [c]hat [k]eepassxc [q]uit"

mode $mode_launcher {
    bindsym f exec firefox
    bindsym c exec ~/.config/sway/scripts/social.sh
    bindsym k exec keepassxc

    bindsym Return mode "default"
    bindsym Escape mode "default"
    bindsym $mod+o mode "default"
    bindsym q mode "default"
}

bindsym $mod+o mode $mode_launcher

After going into the mode with $mod+0, we can start the applications with the relevant keys. The mode can be exited using Return, Esc, $mod+o or q.

Path-based activation of units.

This feature allows systemd to monitor files or paths, and start a unit when the path exists, or something in it is changed. It has different options:

• PathExists: triggers if path exists.
• PathExistsGlob: triggers if a file that matches a glob exists.
• PathModified: triggers if path is changed (every write).
• PathChanged: triggers if path is changed (not every write, only when file is closed).
• DirectoryNotEmpty: triggers if directory contains at least one file.

This is done (in typical systemd fashion), by creating a .path file that corresponds to a specific service file.

I’ve been using it for some of my units that have their data storage on an external mount point, that due to encryption, is only mounted manually after booting the machine.

One service that uses this is my Jellyfin instance. So, to make sure Jellyfin only starts after the volume is mounted (eg. the path exists), we can do the following thing:

First, we create a .path file:

systemctl edit --full --force jellyfin.path

In that path file, we include the following lines:

[Path]
PathExists=/media/zfsnas/cryptset/smb/shared

[Install]
WantedBy=multi-user.target

This is pretty much the simplest form of creating a .path unit. See the manpage for more options.

The important part is the WantedBy= in the install section. This (like with all units) allows us to enable the units, and will then start them when it enters multi-user mode.

We can then enable the .path file, and disable the regular .service file:

systemctl disable jellyfin.service
systemctl enable jellyfin.path

After that, jellyfin.service will wait for the specified path to exist before starting. I really like this feature, and I can already think of a few things that this would be handy for (monitoring files and directories, automating backups to external drives, etc).

I’ll keep you posted if I do anything interesting with it!

The program is called Wayland Overlay Bar (wob). It can be used to create simple progress/status bars for programs. The nice thing about it, is that it just takes an integer on a named pipe, and displays the value of that integer as a graphical bar.

Installation and configuration is very simple. After installing the program, you can create a named pipe, and connect that pipe to the stdin of wob:

mkfifo /tmp/wobpipe
tail -f /tmp/wobpipe | wob

For convenience, wob also provides a systemd socket file. The command below creates a wob pipe under $XDG_RUNTIME_DIR/wob.socket that is accessible to the current user.

systemctl --user enable wob.socket --now

I mainly use it for volume and brightness control. To make it work in sway, only a slight modification of my commands was necessary:

set $WOBSOCK $XDG_RUNTIME_DIR/wob.sock

bindsym XF86AudioLowerVolume exec pamixer --decrease 5 && pamixer --get-volume > $WOBSOCK
bindsym XF86AudioRaiseVolume exec pamixer --increase 5 && pamixer --get-volume > $WOBSOCK
bindsym XF86MonBrightnessDown exec xbacklight -dec 5 && xbacklight -get > $WOBSOCK
bindsym XF86MonBrightnessUp exec xbacklight -inc 5 && xbacklight -get > $WOBSOCK

I love the simplicity and flexibility of this program. I really see myself using this often in the future.

In this article I’ll try to give some information that is actually based on reality. It is always difficult to give “generic” advice, because security advice should always be tailored to a specific situation. I will try to give some advice that will fit in this situation, and add caveats and further information when necessary. First, we’ll talk about some ways you can anonymise your “research” about abortion, then we’ll talk about period tracking and what apps I would personally recommend.

NB: I can only judge these apps according to their security/privacy. I don’t purport to know much about their usability.

Also, I will try to be as non-technical as possible in this article. This means that this article will oversimplify several concepts, to hopefully make them clear to non-technical users. I might follow this article up with some more technical analysis of period tracking apps.

I will also link several articles from the EFFs excellent Surveillance Self-Defense program.

Caveat: app stores and mobile platforms

One big caveat for this article are app stores that are primarily found on Android and iOS. Please keep in mind that on the Play Store and the App Store you are always logged in with your account. So there will be record of you downloading and installing the app. Under Android you can use an alternative app store called F-Droid. This allows you to download and install applications without a login. I don’t use iOS so I don’t know this for sure, but if I understand the Apple ecosystem correctly, something like this doesn’t exist for that platform.

The other caveat is that mobile platforms will save a backup of your apps and their data in the cloud (Google Drive/iCloud). For iCloud, you can disable using iCloud for individual Apps. For Android this depends on your android version/phone manufacturer.

Think about these things if you decide to use certain apps, like a period tracker.

Researching topics

On the internet, you will find two types of advice on how to anonymise your browsing. One will suggest using a VPN, the other using Tor. Both of these tools have their own advantages and disadvantages, and each have to be used correctly to be effective. I personally prefer Tor, since VPNs have some very big downsides that aren’t immediately apparent to non-technical users. I still mention them here, because they are often the first type of tool people hear about, and I want to caution against them.

Please also note that any anonymising way falls flat if you identify yourself via another way. This should be a no-brainer, but if you log into a website via a VPN or Tor, you are no longer “anonymous” to that website.

VPN

VPNs are a technology that can be used for different purposes. In our case, a VPN anonymises your traffic to your ISP. Because of the way the internet works, your ISP can see all websites you visit, but not what you do on the websites itself. They see that you visited google.com, but not what you search. They will see that you visited youtube.com, but not what videos you watch, etc.

If this is a concern for you, you can use a VPN. This will make your ISP unable to see which websites you visit. But this will just “shift” the traffic to your VPN provider, which essentially (for these purposes) becomes your ISP. This means you need to trust your VPN provider as much as your ISP, if not more. This means you really need to research which VPN you want to use. In general, the way I understand the situation in the US, your data will not be safe from the government because courts can get to that data via subpoena. So you will want to find a VPN provider that doesn’t save any data. But even if your provider says they don’t save any data, there is no way to know that they are not lying. Because if they have data, they will have to give it away. Another way is to use a provider where jurisdiction by US courts does not extend. This is difficult to gauge.

One advantage a VPN has over Tor, is that it tunnels your complete traffic, not only your web browsing.

In general I would not recommend a VPN, unless you know what you are doing and you fall in the category I describe above.

For more information, you can read the EFFs guide on choosing the right VPN.

Tor

This is a protocol and a browser that completely anonymises your traffic. It is impossible for anyone to uniquely identify the websites you visit, and it is also impossible for the websites to identify you. Tor can be downloaded on the website of the Tor Project. I would recommend using the Tor Browser. In this application, the protocol is bundled with a special browser that also has several different anonymising settings. Tor is different from a VPN in that it encrypts your connection in multiple layers, and then sends it along multiple other computers before it reaches the website you are opening. This means your traffic is fully anonymised (there are some things to be considered still, these are described in the Tor FAQ).

One caveat about Tor is that it can be quite slow. It bounces your connection around several other computers, so you will have quite high latency (“ping”) and slower download speeds. This means that you will “reach” websites slower and they will also “render” slower on your device. Another is that it only works for that Browser. Other traffic (from programs on your computer) will not be anonymous.

Here are some guides on how to install Tor Browser:

• Windows
• MacOS
• Android
• iOS

For mobile platforms also make sure you understand the caveats.

Period tracking

I’ve seen advice ranging from “delete all of your period trackers” to “this specific app is perfectly safe and you should definitely use it”.

The first advice (delete all of your apps) is of course very safe. It’s also absolutely bonkers, and it doesn’t help users at all. In my opinion, if you want to blurt that hot take into the public sphere, you might as well not say anything at all.

Now that we’ve dealt with that, some apps. As you can guess, I’m a bit out of my depth with this topic. I’m not a health expert, and I can’t meaningfully test these apps. There seem to be two “big” apps that are currently circulating. I’ll write my opinion for each one. The first one (Clue) seems to be the more popular one, and the second one (Drip) seems to be the more privacy-focused one.

Clue

Clue is a commercial period and fertility tracking app from Berlin. They are currently heavily marketing their app to a US audience via Twitter posts and press releases. In these press releases they especially keep hammering home that they are a European company that falls under GDPR. GDPR is a big topic that I can’t address in this article, but suffice to say it is a quite strict data protection regulation that sets rules how EU companies have to handle data from EU residents. It sets a lot of restrictions and some rights that consumers have. There are two important things to know about this. The first one is:

GDPR does not apply to non-EU residents and citizens.

While Clue probably does not treat “American” data different from “EU” data, because that would just mean more work, they are perfectly able to do so. In practice this caveat likely does not matter. There is another problem that is likely more important:

GDPR has exceptions for law enforcement.

In Germany (where Clue is based), courts can request documents and information. Companies then have to provide this information. As far as I understand, this also applies to US law enforcement, because of Mutual Legal Assistance Treaties. In general, Germany does not require that something is illegal in Germany for them to serve document requests through these treaties.

Clue is currently misleading about this topic. They claim that they will only serve EU and German authorities. This is technically correct, but does not matter in practice. If i understand the process correctly, US law enforcement will contact German law enforcement, that will then contact the company. My information about this process is purely theoretical. I’m not a lawyer, so I don’t know if this actually happens, or will ever happen. I just want to shine a light on the fact that it is not as easy as “we are in the EU and GDPR is very good”. GDPR does not protect you as an non-resident and non-citizen of the EU. GDPR also does not protect you from criminal and civil liability!

All in all, this does not mean you should absolutely not use Clue. There are still some good arguments for using it over US-based applications. Just think about the consequences of using it, and try to think about your specific threat model.

Drip

Drip is a privacy-focused and open source application. Open source means that the source code for the application can be read and changed by anyone. Their privacy policy is refreshingly short. This is because they don’t collect any data. All data is kept locally on the device, and can be deleted from inside the app. This means that they can’t give any data to law enforcement, even if there is a subpoena. (An example of this strategy can be seen in the cases where Signal was subpoenaed) The app is funded by Mozilla and the German government (through Prototype Fund).

The app has some problems that may or may not be a deal breaker:

• The newest version is only available via Google Play

They also provide instructions on their website to manually install the latest version, so this problem can be partly circumvented.

• There is no iOS version currently

The development team is working on an iOS version, but it is not available at the moment. Even if they create an iOS version, there will not a be way to download this version anonymously (see caveats for more information).

An additional deal breaker for an American audience might be that this app does not currently allow you to enter Temperature in Fahrenheit. There have been some feature requests for this, but there is currently no one working on this feature (since the team seems to be busy with releasing an iOS version).

Conclusion

The internet contains a lot of information. Some of it is good, but a lot of it is bad, especially when people work themselves up to a frenzy. I hope that the advice in this article will be useful to people. Good luck, and stay safe out there!

I decided the best way of doing this would be via NextCloud. I have a NextCloud server, which syncs important files between my devices. This would be the perfect tool for “deploying” the new ISO to all of the devices that it is needed on.

Nextcloud has an API for their WebDAV implementation. This allows us to use WebDav operations to download, upload, and change files on the server. This is pretty much ideal for our use case. Since WebDAV is an extension of HTTP, I can use my favourite API client (curl + bash).

#!/usr/bin/env bash
USERNAME="clouduser"
FILE="spart.iso"

curl -u "${USERNAME}:${PASSWORD}" \
    https://cloud.example.com/remote.php/dav/files/clouduser/ISO/"$FILE" \
    -T "$BUILD_DIR/out/$FILE"

Our client will need a password. To provide this, we can create an Application Password in NextCloud. After generating it, we will use Drone to provide the password to our script. We don’t want to store our password in plain text, so we can use drones secret management. Secrets are defined per repository under settings -> secrets. After adding our secret, we can use it in our pipeline.

kind: pipeline
type: exec
name: default

steps:
- name: upload iso to nextcloud
  commands:
    - ./extra/upload/upload-to-nextcloud.sh
  environment:
    BUILD_DIR: /media/fast/cryptset/builds/spart
    PASSWORD:
        from_secret: nc_password

trigger:
  branch:
  - main

Drone continues to be a very useful tool, and I look forward to doing more stuff with it. I hope you enjoyed reading this article!

Unity has some support for CLI workflows, but it is a bit limited, and difficult to work with. Since my server runs headlessly, I will try to run the application completely headless as well.

The first problem was getting the correct Unity version. Unity is quite picky about its versions, and I wanted to make sure I got the same minor release. Since I’m on Arch, there are 2 different ways of getting the version. I could use the AUR package and change the PKGBUILD to the version I want. Or I could use the Unity Hub. This would also enable me to install different versions, and manage the modules for each installed version. I decided to use the unity hub, because of the higher flexibility. It can be installed with this AUR package.

X problems

The first problem that presented itself is when I tried to run Unity Hub in headless mode. It just errors out with the following message:

(unityhub-bin:192687): Gtk-WARNING **: 13:06:59.647: cannot open display:

The problem is that Unity Hub tries to initialize an X display even if it doesn’t actually output any graphics. To fix this, we can provide it with a virtual X server. One option is to use Xvfb. This provides a virtual display server that performs all graphical operations in virtual memory without showing any screen output. With this we can “trick” unityhub into thinking we have a graphical display. Arch provides this package with xorg-server-xvfb. After installing it, we can use the convenience script called xvfb-run:

xvfb-run unityhub --headless help

After that, we can install the Unity version we need (2021.2.0f1 in our case). Because this is not one of the “recommended” versions that Unity Hub includes by default, we also need to specify the changeset. I got the specific changeset from the Unity archive. I just looked at the URL for the windows Unity editor downloads and used the changeset from there. In my case this is 4bf1ec4b23c9. We also install the module to create linux builds. (--childmodules automatically installs al child modules of the selected modules) This makes the full command the following:

xvfb-run unityhub --headless install --version 2021.2.0f1 --changeset 4bf1ec4b23c9 --module linux-il2cpp --childmodules

Activation

After installing we need to activate our Unity license. This is a bit complicated, and the exact setup depends highly on the specific use case. In my case, I’m using the free, non-commercial licenses that are tied to a Unity account. So I can’t just specify a serial number to activate my license (that only works for pro licenses). The other option is to specify your account details on the command line. Since I’m running this whole process in a build pipeline, I don’t want to do this. So I decided to follow the steps for offline activation. In the first step, we need to generate a license activation file for our specific editor version. Unity Hub installs the editor into the users home directory at $HOME/Unity/Hub/Editor/2021.2.0f1/Editor/Unity

Unity -batchmode -nographics -createManualActivationFile

This will create a .alf file in the current directory. We then need to download this file to a computer with a web browser. In the web browser we can then request a Unity license file at the following URL. After copying the license file to our server again, we can activate it with the following command:

Unity -batchmode -nographics -manualLicenseFile <file>

After activating Unity, we can start trying to build our application. This took quite some trial and error, since the Unity CLI is limited in its functionality, and not particularly well documented. There are some options we always need to add if we want to do a headless build:

• -batchmode: runs all unity workflows sequentially
• -nographics: does not initialize any graphics (eg: the editor window)
• -accept-apiupdate: Make sure that the APIUpdater runs
• -quit: quit after running all batch mode workflows

Unity also has some generic build arguments and targets:

• Standalone
• Win
• Win64
• OSXUniversal
• Linux64
• iOS
• Android
• WebGL
• WindowsStoreApps
• tvOS

This doesn’t allow us to control more specific build options (subtargets, scenes, etc). To do this, we need to write a C#-Script that defines the options. This script needs to be in a specific folder in your project folder (Assets/Editor). In that script you can specify your needed build options. My script looks like this:

using System.Collections;
using System.Collections.Generic;
using UnityEditor;
using UnityEngine;

public static class BuildSystem
{
    static string[] sceneList = { "Assets/Scenes/SampleScene.unity" };

    public static void BuildLinuxHeadless()
    {
        var report = BuildPipeline.BuildPlayer(new BuildPlayerOptions()
        {
            locationPathName = "../target/server",
            scenes = sceneList,
            target = BuildTarget.StandaloneLinux64,
            subtarget = (int)StandaloneBuildSubtarget.Server
        });

        Debug.Log($"Build result: {report.summary.result}, {report.summary.totalErrors} errors");
        if (report.summary.totalErrors > 0)
            EditorApplication.Exit(1);
    }
}

After that, we can execute the script with -executeMethod.

Unity -batchmode -nographics -projectPath <project_path> -executeMethod BuildSystem.BuildLinuxHeadless -accept-apiupdate -quit

The build should spit out a working player after that. If you want to customise build options, just adjust the script accordingly.

Pipeline

This is an example Drone pipeline based on this script, and a systemd service file that starts the server.

game-server.service

[Unit]
Description=game Server
After=network-online.target

[Service]
ExecStart=/srv/user/game/game-server
Type=exec
Environment="TERM=xterm"
WorkingDirectory=/srv/user/game/
User=user

[Install]
WantedBy=default.target

.drone.yml

kind: pipeline
type: exec
name: default

steps:
- name: copy build files
  commands:
  - install -m 750 -d /var/lib/drone-runner-exec/builds/game
  - install -m 750 -d /var/lib/drone-runner-exec/builds/game/src
  - install -m 750 -d /var/lib/drone-runner-exec/builds/game/target
  - cp -r game_project/* /var/lib/drone-runner-exec/builds/game/src
- name: build
  commands:
  - export PATH=$PATH:/var/lib/drone-runner-exec/Unity/Hub/Editor/2021.2.0f1/Editor
  - export HOME=/var/lib/drone-runner-exec
  - export XDG_CACHE_HOME=$HOME/.cache
  - Unity -batchmode -nographics -projectPath $PROJECT_PATH -executeMethod BuildSystem.BuildLinuxHeadless -accept-apiupdate -quit &> $HOME/builds/game/build.log
  environment:
    PROJECT_PATH: /var/lib/drone-runner-exec/builds/game/src
    TARGET_PATH: /var/lib/drone-runner-exec/builds/game/target
- name: stop server
  commands:
  - sudo systemctl stop game-server.service
- name: copy server files
  commands:
  - install -m 750 -g builders -d "$DEPLOY_PATH"
  - install -m 750 -g builders -d "$DEPLOY_PATH/game-server_Data"
  - install -m 750 -g builders "$TARGET_PATH/game-server" "$DEPLOY_PATH"
  - install -m 750 -g builders "$TARGET_PATH/UnityPlayer.so" "$DEPLOY_PATH"
  - cp -r $TARGET_PATH/game-server_Data/* "$DEPLOY_PATH/game-server_Data/"
  environment:
    TARGET_PATH: /var/lib/drone-runner-exec/builds/game/target
    DEPLOY_PATH: /srv/user/game
- name: copy service file
  commands:
  - sudo install -m 644 game-server.service /etc/systemd/system/
- name: reload systemd daemon
  commands:
  - sudo systemctl daemon-reload
- name: start service
  commands:
  - sudo systemctl start game-server.service

Some things to consider about the drone pipeline:

The unity editor needs certain variables set to build correctly. Notably the following:

• $HOME
• $XDG_CACHE_HOME

Also, Unity is very verbose when building. In fact, it is so verbose that the web console of my drone server couldn’t handle it. I decided to write the output to a file. That way, I can at least see the log of the last build and why it went wrong. Of course, you can always make this part more complicated, add some kind of versioned logging facility. But I’ll tackle that when I need such a functionality.

I hope you enjoyed reading this article. Now go out there and create your own Unity build pipelines!

• simple data rescue
• forensics
• creating backups
• partitioning
• unlocking computers

Obviously, they have many more use cases.

I’ve been such a big fan of them, that I have quite some collection of live distros and ISOs. This also shows my biggest problem: there are a lot of different options to choose from, and none of them do everything I want. A lot of them are infrequently updated, have very specific usages or are just a bit weird. There are some exceptions to this rule (looking at you, Parted Magic), but they do charge a subscription fee to use the software. So I decided to look into doing this myself.

I had been experimenting with Archiso for a while to create custom live systems. I even created some for previous employers. But the problem I was having was essentially the same as my collection of other ISOs: They were “lying around” everywhere. I had no centralized way of building/deploying them. With an ever growing collection of ISOs an ever growing mountain of flash drives emerged.

I decided to solve this problem by creating a single distro that I could use for everything. If I needed a new tool, I would just add it. I combined this with a flash drive that I keep on my key chain. I decided to do this with Archiso again, since I have some experience with it, and I like the build system. The two things that really tie everything together though, are Gitea (A self-hosted Git forge) and Drone (a self-hosted CI platform). This allows me to automate the building process.

NB: Setting up Archiso is out of the scope of this article. Please see this excellent article on the Arch Wiki to get started.

Building the ISO

Normally, we would just build the ISO locally with mkarchiso. With drone, we can automate this process. I will show my drone file as an example, but drone has a lot more options. Check out the documentation if I whet your appetite.

The drone workflow consists of one or more pipelines that each contain steps. Each step has a name, and one or more associated commands. Let’s go through the steps one by one.

kind: pipeline
type: exec
name: default

steps:
- name: create chroot
  commands:
  - mkdir /var/lib/drone-runner-exec/buildroot/
  - mkarchroot /var/lib/drone-runner-exec/buildroot/root base-devel

This command creates a chroot that we use to build our AUR packages in. For Archiso, we need to manually build AUR packages and add them to a local repository.

- name: download aur packages
  commands:
  - auracle download nwipe
  - auracle download unixbench
  - auracle download stress-ng

Here we download our aur packages using the excellent auracle

- name: build stress-ng
  commands:
  - cd stress-ng
  - makechrootpkg -c -r /var/lib/drone-runner-exec/buildroot/
- name: build unixbench
  commands:
  - cd unixbench
  - makechrootpkg -c -r /var/lib/drone-runner-exec/buildroot/ -- --syncdeps
- name: build nwipe
  commands:
  - cd nwipe
  - makechrootpkg -c -r /var/lib/drone-runner-exec/buildroot/
- name: build owper
  commands:
  - cd extra/owper/
  - makechrootpkg -c -r /var/lib/drone-runner-exec/buildroot/ -- --syncdeps

After that, we build the packages in a clean chroot.

- name: add packages to repo
  commands:
  - find . -name '*pkg.tar.zst' -exec cp '{}' /var/lib/drone-runner-exec/builds/spart/aur_repo \;
  - cd /var/lib/drone-runner-exec/builds/spart/aur_repo/
  - repo-add localaur.db.tar.gz *.pkg.tar.zst

We add the packages to our local repo that pacman can use.

- name: build iso
  commands:
  - sudo mkarchiso -v -w "/var/lib/drone-runner-exec/builds/spart/work" -o "/var/lib/drone-runner-exec/builds/spart/out" -A SPART -L SPART -P SergeantBiggs .

Building the ISO.

- name: chown iso
  commands:
  - sudo chown -R drone-runner-exec:builders "/var/lib/drone-runner-exec/builds/spart/out/"
- name: move iso
  commands:
  - mv /var/lib/drone-runner-exec/builds/spart/out/*.iso /var/www/open.sgnt.link/iso/spart.iso

After that, we move the ISO to a web server. We can then download it once it is finished.

- name: delete work directory, repo and chroot
  commands:
  - sudo rm -rf /var/lib/drone-runner-exec/builds/spart/work/
  - sudo rm -rf /var/lib/drone-runner-exec/buildroot/
  - rm -rf /var/lib/drone-runner-exec/builds/spart/aur_repo/*
  when:
    status:
    - failure
    - success

After everything is finished, we delete the work directory and the chroot. The “when” part makes sure that this command is always executed, even if the build fails.

Here is the complete file, for reference:

kind: pipeline
type: exec
name: default

steps:
- name: create chroot
  commands:
  - mkdir /var/lib/drone-runner-exec/buildroot/
  - mkarchroot /var/lib/drone-runner-exec/buildroot/root base-devel
- name: download aur packages
  commands:
  - auracle download nwipe
  - auracle download unixbench
  - auracle download stress-ng
- name: build stress-ng
  commands:
  - cd stress-ng
  - makechrootpkg -c -r /var/lib/drone-runner-exec/buildroot/
- name: build unixbench
  commands:
  - cd unixbench
  - makechrootpkg -c -r /var/lib/drone-runner-exec/buildroot/ -- --syncdeps
- name: build nwipe
  commands:
  - cd nwipe
  - makechrootpkg -c -r /var/lib/drone-runner-exec/buildroot/
- name: build owper
  commands:
  - cd extra/owper/
  - makechrootpkg -c -r /var/lib/drone-runner-exec/buildroot/ -- --syncdeps
- name: add packages to repo
  commands:
  - find . -name '*pkg.tar.zst' -exec cp '{}' /var/lib/drone-runner-exec/builds/spart/aur_repo \;
  - cd /var/lib/drone-runner-exec/builds/spart/aur_repo/
  - repo-add localaur.db.tar.gz *.pkg.tar.zst
- name: build iso
  commands:
  - sudo mkarchiso -v -w "/var/lib/drone-runner-exec/builds/spart/work" -o "/var/lib/drone-runner-exec/builds/spart/out" -A SPART -L SPART -P SergeantBiggs .
- name: chown iso
  commands:
  - sudo chown -R drone-runner-exec:builders "/var/lib/drone-runner-exec/builds/spart/out/"
- name: move iso
  commands:
  - mv /var/lib/drone-runner-exec/builds/spart/out/*.iso /var/www/open.sgnt.link/iso/spart.iso
- name: delete work directory, repo and chroot
  commands:
  - sudo rm -rf /var/lib/drone-runner-exec/builds/spart/work/
  - sudo rm -rf /var/lib/drone-runner-exec/buildroot/
  - rm -rf /var/lib/drone-runner-exec/builds/spart/aur_repo/*
  when:
    status:
    - failure
    - success

trigger:
  branch:
  - main

And that’s that. I really love this setup. It enables me to add new features (packages, settings, etc) to my live system and build the ISO automatically. Afterwards, I can just download the image and copy it to my flash drive.

I hope you enjoyed reading this article!

NB: This is intended as an easy way to share files for classic “desktop” virtualization settings. It’s not intended for servers, where there are multiple users on the host or the guests.

We’ll be using the same method for both guests, because it works on Linux and on Windows. Instead of the “traditional” way (sharing over the network with SMB, NFS, or 9p), we’ll we using a file system developed by libvirt called virtiofs. Virtiofs was developed for precisely this purpose (“Unlike existing approaches, it is designed to offer local file system semantics and performance.”). This makes file sharing easier to configure, since we don’t need to set up a server. It also should offer very good performance.

Host

To set this up, we first need a folder on the host. This can be done in several ways. I decided to just create a folder in /mnt, and change its ownership to my user:

sudo mkdir /mnt/vfs_share
sudo chown foo:foo /mnt/vfs_share

Then we need to add a virtiofs device.

Virtiofs needs shared memory to work. This can be enabled in the hardware configuration window. Navigate to Hardware -> Memory and select Enable shared memory.

After that, open virt-manager and navigate to the hardware settings of the virtual machine. Click on Add Hardware.

A dialog box should pop up. In the pane on the left, click on Filesystem

Enter the necessary settings.

The settings are as follows:

For the Driver, we keep the default (virtiofs). The source path is the path on our host filesystem. The target path is a bit of a misnomer. It’s not actually a path, just an identifier that we use as a mount point in our guest file system. This will become clear when we mount the file system in the guest later.

If you want a more detailed overview of these options and some alternatives, check out the libvirt knowledge base on virtiofs

Guest

Linux

After that, it’s time to configure the guest. First we create a folder in our users home directory that we will mount the share into.

mkdir ~/share

After that, we can mount the share with:

sudo mount -t virtiofs share /home/foo/share

If we want to make this permanent, we can just add an entry to /etc/fstab

share   /home/foo/share       virtiofs        defaults        0       0

After rebooting, the share should be mounted! If your user on the host and the guest are the same, you don’t even need to worry about any other permissions. Otherwise, change the folder with chmod to suit your needs.

Windows 10

Windows 10 can also use virtiofs, but it needs some additional work. First, we need to install 2 pieces of software:

• WinFSP
• Virtio drivers

This installation can be done in two ways: manually or with chocolatey.

Chocolatey

If you already have Chocolatey installed, it’s as simple as:

choco install virtio-drivers winfsp

If you don’t have Chocolatey installed, I highly recommend that you check it out. It’s a package manager for Windows that integrates quite well into the system. It makes maintaining software on Windows a lot easier! Installation instructions are here.

During the installation of the virtio-drivers it will pop up a dialog box asking if you trust the drivers. Confirm the dialog box to install them.

Manual install

Download WinFSP here and install it. The default options are fine if you just want to use the drivers.

After that, you need to download the latest ISO from the following archive. After downloading, mount the ISO. Then open Device Manager. Select the appropriate device. It should appear under Other devices as a Mass storage controller

Right-click on the device and select Update driver. From that menu, select Browse my computer for drivers. Browse to the location of the ISO, and select the directory of the drivers.

Configuration

After that, it’s a good idea to create a service for the drivers, so they are automatically loaded during system startup.

sc.exe create VirtioFsSvc binpath="C:\ProgramData\Chocolatey\bin\virtiofs.exe" start=auto depend="WinFsp.Launcher/VirtioFsDrv" DisplayName="Virtio FS Service"
sc.exe start VirtioFsSvc

If you’ve installed manually, find the location of the .exe and change the binpath.

Benchmarks

All Benchmarks were done on a Dell Latitude 5401 with the following specs:

Linux

For the benchmark, I simply copied over a 10 GB file with cp, measuring it with time. That gave me the following result:

real    0m12.532s
user    0m0.002s
sys     0m5.785s

I also transferred the same file with rsync, to get a progress bar. That way I could get a feel for the approximate speed of the transfer. The transfer speed was around 250-300 MB/s consistently.

Windows

On Windows, I also copied the file in 2 ways. First using the windows File transfer dialog. I got up speeds of about 550-700 MB/s. After that, I tried the same using robocopy (A Microsoft tool for robust file copying).

-------------------------------------------------------------------------------
   ROBOCOPY     ::     Robust File Copy for Windows
-------------------------------------------------------------------------------

  Started : Friday, 11 March 2022 10:23:20
   Source : C:\Users\foo\Downloads\
     Dest = Z:\

    Files : *.*

  Options : *.* /DCOPY:DA /COPY:DAT /R:1000000 /W:30

------------------------------------------------------------------------------

                           1    C:\Users\foo\Downloads
100%        New File              10.0 g        10gig.img

------------------------------------------------------------------------------

               Total    Copied   Skipped  Mismatch    FAILED    Extras
    Dirs :         1         0         1         0         0         0
   Files :         1         1         0         0         0         2
   Bytes :  10.000 g  10.000 g         0         0         0   797.0 k
   Times :   0:00:18   0:00:18                       0:00:00   0:00:00


   Speed :           566020993 Bytes/sec.
   Speed :           32387.981 MegaBytes/min.
   Ended : Friday, 11 March 2022 10:23:39

Here we end up with roughly the same speed as the windows file dialog.

Conclusion

I’m very pleased with these results. The process of setting it up is quite easy (although not as “seamless” as the solutions Virtualbox and VMWare provide). The performance seems fantastic, at first glance. For now, I’m very happy with it. I will be testing this further. If I hit any snags or discover something else that’s interesting, I’ll let you know.

The built-in options for hardening are quite extensive, and can best be compared to something like firejail. They both have similar capabilities, but firejail focuses more on desktop applications, whereas systemd hardening applies to systemd units. The hardening options are configured in the units service file, in the [Service] section.

Let’s look at the different options with an example: a service file I set up a while ago for a Discord bot. The bot is a self-hosted version of evobot.

# /etc/systemd/system/boombot.service
[Unit]
Description=Music Bot
After=network-online.target

[Service]
ExecStart=/usr/bin/node /srv/bot/DiscordBots/BoomBot/index.js
Type=simple
WorkingDirectory=/srv/bot/DiscordBots/BoomBot

# Hardening
PrivateDevices=true
PrivateTmp=true
ProtectControlGroups=true
ProtectSystem=full
ProtectKernelTunables=true
RestrictSUIDSGID=true
User=bot

[Install]
WantedBy=multi-user.target

As you can see, I already set up some basic hardening options, but we can certainly do better than that.

In hardening systemd units, two resources are particularly useful. The first one is systemd-analyze security <name>.service. This tool gives an overview of a lot of useful hardening options, whether they are turned on or not, and a score based on how many options are active. The score is based on a weighted system, where hardening options are given some points that are subtracted if they are activated. The scale goes from 10 (worst) to 0 (best).

The other resource is the man page for systemd.exec(5). It includes all possible hardening options (in the section SANDBOXING). Most options are described in great detail. This enables us to make informed choices about our hardening.

To start, let’s have a look at the output of systemd-analyze security boombot.service.

  NAME                                                        DESCRIPTION                                                             EXPOSURE
✗ RemoveIPC=                                                  Service user may leave SysV IPC objects around                               0.1
✗ RootDirectory=/RootImage=                                   Service runs within the host's root directory                                0.1
✓ User=/DynamicUser=                                          Service runs under a static non-root user identity
✗ CapabilityBoundingSet=~CAP_SYS_TIME                         Service processes may change the system clock                                0.2
✗ NoNewPrivileges=                                            Service processes may acquire new privileges                                 0.2
✓ AmbientCapabilities=                                        Service process does not receive ambient capabilities
✗ ProtectClock=                                               Service may write to the hardware clock or system clock                      0.2
✗ CapabilityBoundingSet=~CAP_SYS_PACCT                        Service may use acct()                                                       0.1
✗ CapabilityBoundingSet=~CAP_KILL                             Service may send UNIX signals to arbitrary processes                         0.1
✗ ProtectKernelLogs=                                          Service may read from or write to the kernel log ring buffer                 0.2
✗ CapabilityBoundingSet=~CAP_WAKE_ALARM                       Service may program timers that wake up the system                           0.1
✗ CapabilityBoundingSet=~CAP_(DAC_*|FOWNER|IPC_OWNER)         Service may override UNIX file/IPC permission checks                         0.2
✗ CapabilityBoundingSet=~CAP_LINUX_IMMUTABLE                  Service may mark files immutable                                             0.1
✗ CapabilityBoundingSet=~CAP_IPC_LOCK                         Service may lock memory into RAM                                             0.1
✗ ProtectKernelModules=                                       Service may load or read kernel modules                                      0.2
✗ CapabilityBoundingSet=~CAP_SYS_MODULE                       Service may load kernel modules                                              0.2
✗ CapabilityBoundingSet=~CAP_SYS_TTY_CONFIG                   Service may issue vhangup()                                                  0.1
✗ CapabilityBoundingSet=~CAP_SYS_BOOT                         Service may issue reboot()                                                   0.1
✗ CapabilityBoundingSet=~CAP_SYS_CHROOT                       Service may issue chroot()                                                   0.1
✗ SystemCallArchitectures=                                    Service may execute system calls with all ABIs                               0.2
✗ CapabilityBoundingSet=~CAP_BLOCK_SUSPEND                    Service may establish wake locks                                             0.1
✗ MemoryDenyWriteExecute=                                     Service may create writable executable memory mappings                       0.1
✗ RestrictNamespaces=~user                                    Service may create user namespaces                                           0.3
✗ RestrictNamespaces=~pid                                     Service may create process namespaces                                        0.1
✗ RestrictNamespaces=~net                                     Service may create network namespaces                                        0.1
✗ RestrictNamespaces=~uts                                     Service may create hostname namespaces                                       0.1
✗ RestrictNamespaces=~mnt                                     Service may create file system namespaces                                    0.1
✗ CapabilityBoundingSet=~CAP_LEASE                            Service may create file leases                                               0.1
✗ RestrictNamespaces=~cgroup                                  Service may create cgroup namespaces                                         0.1
✗ RestrictNamespaces=~ipc                                     Service may create IPC namespaces                                            0.1
✗ ProtectHostname=                                            Service may change system host/domainname                                    0.1
✗ CapabilityBoundingSet=~CAP_(CHOWN|FSETID|SETFCAP)           Service may change file ownership/access mode/capabilities unrestricted      0.2
✗ CapabilityBoundingSet=~CAP_SET(UID|GID|PCAP)                Service may change UID/GID identities/capabilities                           0.3
✗ LockPersonality=                                            Service may change ABI personality                                           0.1
✗ RestrictAddressFamilies=~AF_PACKET                          Service may allocate packet sockets                                          0.2
✗ RestrictAddressFamilies=~AF_NETLINK                         Service may allocate netlink sockets                                         0.1
✗ RestrictAddressFamilies=~AF_UNIX                            Service may allocate local sockets                                           0.1
✗ RestrictAddressFamilies=~…                                  Service may allocate exotic sockets                                          0.3
✗ RestrictAddressFamilies=~AF_(INET|INET6)                    Service may allocate Internet sockets                                        0.3
✗ CapabilityBoundingSet=~CAP_MAC_*                            Service may adjust SMACK MAC                                                 0.1
✗ RestrictRealtime=                                           Service may acquire realtime scheduling                                      0.1
✗ ProtectSystem=                                              Service has very limited write access to the OS file hierarchy               0.1
✗ CapabilityBoundingSet=~CAP_SYS_PTRACE                       Service has ptrace() debugging abilities                                     0.3
✗ CapabilityBoundingSet=~CAP_SYS_(NICE|RESOURCE)              Service has privileges to change resource use parameters                     0.1
✓ SupplementaryGroups=                                        Service has no supplementary groups
✓ CapabilityBoundingSet=~CAP_SYS_RAWIO                        Service has no raw I/O access
✓ PrivateTmp=                                                 Service has no access to other software's temporary files
✓ PrivateDevices=                                             Service has no access to hardware devices
✗ CapabilityBoundingSet=~CAP_NET_ADMIN                        Service has network configuration privileges                                 0.2
✗ ProtectProc=                                                Service has full access to process tree (/proc hidepid=)                     0.2
✗ ProcSubset=                                                 Service has full access to non-process /proc files (/proc subset=)           0.1
✗ ProtectHome=                                                Service has full access to home directories                                  0.2
✗ CapabilityBoundingSet=~CAP_NET_(BIND_SERVICE|BROADCAST|RAW) Service has elevated networking privileges                                   0.1
✗ CapabilityBoundingSet=~CAP_AUDIT_*                          Service has audit subsystem access                                           0.1
✗ CapabilityBoundingSet=~CAP_SYS_ADMIN                        Service has administrator privileges                                         0.3
✗ PrivateNetwork=                                             Service has access to the host's network                                     0.5
✗ PrivateUsers=                                               Service has access to other users                                            0.2
✗ CapabilityBoundingSet=~CAP_SYSLOG                           Service has access to kernel logging                                         0.1
✓ DeviceAllow=                                                Service has a minimal device ACL
✓ KeyringMode=                                                Service doesn't share key material with other services
✓ Delegate=                                                   Service does not maintain its own delegated control group subtree
✗ SystemCallFilter=~@clock                                    Service does not filter system calls                                         0.2
✗ SystemCallFilter=~@cpu-emulation                            Service does not filter system calls                                         0.1
✗ SystemCallFilter=~@debug                                    Service does not filter system calls                                         0.2
✗ SystemCallFilter=~@module                                   Service does not filter system calls                                         0.2
✗ SystemCallFilter=~@mount                                    Service does not filter system calls                                         0.2
✗ SystemCallFilter=~@obsolete                                 Service does not filter system calls                                         0.1
✗ SystemCallFilter=~@privileged                               Service does not filter system calls                                         0.2
✗ SystemCallFilter=~@raw-io                                   Service does not filter system calls                                         0.2
✗ SystemCallFilter=~@reboot                                   Service does not filter system calls                                         0.2
✗ SystemCallFilter=~@resources                                Service does not filter system calls                                         0.2
✗ SystemCallFilter=~@swap                                     Service does not filter system calls                                         0.2
✗ IPAddressDeny=                                              Service does not define an IP address allow list                             0.2
✓ NotifyAccess=                                               Service child processes cannot alter service state
✓ ProtectControlGroups=                                       Service cannot modify the control group file system
✓ PrivateMounts=                                              Service cannot install system mounts
✓ CapabilityBoundingSet=~CAP_MKNOD                            Service cannot create device nodes
✓ ProtectKernelTunables=                                      Service cannot alter kernel tunables (/proc/sys, …)
✓ RestrictSUIDSGID=                                           SUID/SGID file creation by service is restricted
✗ UMask=                                                      Files created by service are world-readable by default                       0.1

→ Overall exposure level for boombot.service: 7.6 EXPOSED 🙁

As you can see in the last line, systemd is not very happy with our hardening options. Let’s turn that frown upside down!

The first group of “options” I want to tackle is denying access to the file system. We already had ProtectSystem= set to “full”. With this, only certain parts of the OS are mounted read-only for the service (/usr, /boot, /etc). If we set it to strict, this extends to the complete file hierarchy. If we want to allow the service to write to certain paths, we can specify them manually with ReadWritePaths=. Since this bot doesn’t store any data, let’s set it to strict, and see if it still works. There are also other ways to restrict access to the file system. Let’s enable the following settings:

ProtectSystem=strict		# Disable write to entire file hierarchy
PrivateDevices=true		# Only allow access to pseudo-devices (eg: null, random, zero) in separate namespace	
PrivateTmp=true			# Mount /tmp in own namespace
ProtectKernelLogs=true		# Disable access to Kernel Logs
ProtectProc=invisible		# Disable access to information about other processes
PrivateUsers=true		# Disable access to other users on system
ProtectHome=true		# Disable access to /home
UMask=0077			# set umask

Let’s have a look at how far we’ve come with these options:

→ Overall exposure level for boombot.service: 6.7 MEDIUM 😐

That’s already a lot better! Let’s see if we can improve it. The next step is restricting access to the system.

RestrictNamespaces=true         		# Disable creating namespaces
LockPersonality=true            		# Locks personality system call
NoNewPrivileges=true            		# Service may not acquire new privileges
ProtectKernelModules=true       		# Service may not load kernel modules
SystemCallArchitectures=native  		# Only allow native system calls
ProtectHostname=true            		# Service may not change host name
RestrictAddressFamilies=AF_INET AF_INET6        # Service may only use IP address families
RestrictRealtime=true				# Disable realtime privileges
ProtectControlGroups=true			# Disable access to cgroups
ProtectKernelTunables=true			# Disable write access to kernel variables
RestrictSUIDSGID=true				# Disable setting suid or sgid bits
ProtectClock=true                               # Disable changing system clock

→ Overall exposure level for boombot.service: 4.7 OK 🙂

We have our first smile! Let’s keep going!

The next 2 options are very powerfull, but it is a bit harder to decide how to set them. They are CapabilityBoundingSet and SystemCallFilter.

The first option restricts (as its name suggests) capabilities. These are “privileges” that a process has. Examples of these capabilities include:

• Read/write access to the Kernel audit log
• Access to privileged BPF operations
• Block system suspend
• chown arbitrary files
• kill arbitrary processes
• change MAC address
• Bind a socket to a port less than 1024
• and many more

Many of these capabilities are not needed for normal processes, and they can be especially dangerous because they often ignore permissions (eg: CAP_CHOWN ignores filesystem permissions). Let’s disable all of them.

CapabilityBoundingSet=

You can also apply a whitelist, or invert the selection with ~ (eg: ~CAP_SYS_PTRACE).

→ Overall exposure level for boombot.service: 2.8 OK 🙂

Now for the hardest part (in my opinion): SystemCallFilter. This is a very powerful tool, as it allows you to restrict the system calls that the process can call. This isn’t easy to configure, and might need frequent updates, as the process gains new features.

To make this process easier, systemd includes some “groupings”. These are prepended with an ‘@’, and include several groups of system calls. Examples of these include

• @swap
• @timer
• @reboot
• @raw-io
• etc.

For simplicity, we’ll use a special grouping, @system-service. This includes a “set of system calls used by common services, excluding any special purpose calls”. The man page recommends this as a starting point for custom lists. We also deny 2 other groups that systemd-analyze security recommends:

SystemCallFilter=@system-service
SystemCallFilter=~@privileged @resources # System calls related to super-user

After setting these options, we arrive at our final assessment:

→ Overall exposure level for boombot.service: 1.1 OK 🙂

That seems like a pretty nice verdict! Our process is now locked down quite well. Of course, there are other things that can still be done, like setting a more specific system call filter, but this is a good start. I’ll include the full service file here, so you can have a look at it and use it as a template for your own unit files.

# /etc/systemd/system/boombot.service
[Unit]
Description=Music Bot
After=network-online.target

[Service]
ExecStart=/usr/bin/node /srv/bot/DiscordBots/BoomBot/index.js
Type=simple
WorkingDirectory=/srv/bot/DiscordBots/BoomBot
User=bot

# Hardening
## File System
ProtectSystem=strict
PrivateDevices=true
PrivateTmp=true
ProtectKernelLogs=true
ProtectProc=invisible
PrivateUsers=true
ProtectHome=true
UMask=0077

## System
RestrictNamespaces=true
LockPersonality=true
NoNewPrivileges=true
ProtectKernelModules=true
SystemCallArchitectures=native
ProtectHostname=true
RestrictAddressFamilies=AF_INET AF_INET6
RestrictRealtime=true
ProtectControlGroups=true
ProtectKernelTunables=true
RestrictSUIDSGID=true
ProtectClock=true
RemoveIPC=true

## Capabilities and syscalls
CapabilityBoundingSet=
SystemCallFilter=@system-service
SystemCallFilter=~@privileged @resources

[Install]
WantedBy=multi-user.target

I hope you enjoyed reading this article, and I hope you get inspired to have a closer look at your services and lock them down!

I will be migrating the posts from the WordPress instance. This might take a few weeks. I thought about using a tool to convert the pages automatically, but I’ve decided against it. A tool can’t convert the pages cleanly, and I don’t have that many blog posts anyway (one example of laziness actually having benefits ^^). Also, this will give me a good opportunity to get to know Hugos ins and outs, so it’s a good exercise nonetheless. I’m not sure whether I want to backdate the new posts or not. On the one hand I like the idea of starting with a “clean slate”, but on the other hand I don’t want to lose the information.

One thing that will definitely be lost, is the comment section. Since I never got many comments (aside from very friendly bots reminding me about the great benefits of Viagra and Cialis pills), I don’t think I lost anything. Maybe I will bring back the comment section at some point, but for now I’m not planning on it. If you really want to tell me something, just write me an email!

When I was browsing LWN a few days ago, I noticed something in the release notes for the new systemd v250:

   *  Support for encrypted and authenticated credentials has been added.
      This extends the credential logic introduced with v247 to support
      non-interactive symmetric encryption and authentication, based on a
      key that is stored on the /var/ file system or in the TPM2 chip (if
      available), or the combination of both (by default if a TPM2 chip
      exists the combination is used, otherwise the /var/ key only). The
      credentials are automatically decrypted at the moment a service is
      started, and are made accessible to the service itself in unencrypted
      form. A new tool 'systemd-creds' encrypts credentials for this
      purpose, and two new service file settings LoadCredentialEncrypted=
      and SetCredentialEncrypted= configure such credentials.

      This feature is useful to store sensitive material such as SSL
      certificates, passwords and similar securely at rest and only decrypt
      them when needed, and in a way that is tied to the local OS
      installation or hardware.

This looks like quite a nice feature. Let’s try it out!

Like the release notes say, systemd-creds will use a TPM2 chip if it is available. If it isn’t available, it will just use the secret in /var/lib/systemd/credential.secret. So let’s first check if we have a TPM enabled. This can by reading the value of the following file:

cat /sys/class/tpm/tpm0/tpm_version_major

If TPM 2.0 is available, the output of the command will be “2”.

First we need to encrypt our old credentials. This can be done with systemd-creds. This command takes an input file and an output file. The input file is our old unencrypted credential file, the output file will be the encrypted one:

systemd-creds encrypt secrets/old/token_unencrypted secrets/token

After doing that, we just need to change the setting in our .service file (this is an example file for a bot that I’m running on my server):

[Unit]
Description=My Service
After=network-online.target

[Service]
ExecStart=/usr/bin/important-service
Type=exec
user=foo
WorkingDirectory=/home/foo

# Hardening
PrivateDevices=true
ProtectControlGroups=true
ProtectSystem=full
ProtectKernelTunables=true
RestrictSUIDSGID=true
PrivateTmp=true
PrivateUsers=true
ProtectClock=true
ProtectHome=true
ProtectKernelLogs=true
ProtectKernelModules=true
ProtectProc=noaccess
# Everything is RO, we can only write to ReadWritePaths
ProtectSystem=strict
RemoveIPC=true
UMask=0077
ProtectHostname=true
NoNewPrivileges=true

CapabilityBoundingSet=
RestrictNamespaces=true
RestrictAddressFamilies=AF_INET AF_INET6

# Secrets
LoadCredentialEncrypted=token:/home/foo/secrets/token


[Install]
WantedBy=multi-user.target

NB: the ID of the token (the name in front of the colon). Needs to be the same as the file name of the secret.

After that, everything should work fine, right? Not really. When I started the unit for the first time, it immediately crashed. Can you guess what the problem was? I’ll even give you the error logs from the unit:

Jan 11 14:19:57 examplehost service[13245]: ERROR:tcti:src/tss2-tcti/tcti-device.c:442:Tss2_Tcti_Device_Init() Failed to open specified TCTI device file /dev/tpmrm0: Operation not permitted
Jan 11 14:19:57 examplehost service[13245]: ERROR:tcti:src/tss2-tcti/tctildr-dl.c:154:tcti_from_file() Could not initialize TCTI file: libtss2-tcti-device.so.0
Jan 11 14:19:57 examplehost service[13245]: ERROR:tcti:src/tss2-tcti/tcti-device.c:442:Tss2_Tcti_Device_Init() Failed to open specified TCTI device file /dev/tpm0: Operation not permitted
Jan 11 14:19:57 examplehost service[13245]: ERROR:tcti:src/tss2-tcti/tctildr-dl.c:154:tcti_from_file() Could not initialize TCTI file: libtss2-tcti-device.so.0
Jan 11 14:19:57 examplehost service[13245]: WARNING:tcti:src/util/io.c:252:socket_connect() Failed to connect to host 127.0.0.1, port 2321: errno 111: Connection refused
Jan 11 14:19:57 examplehost service[13245]: ERROR:tcti:src/tss2-tcti/tcti-swtpm.c:591:Tss2_Tcti_Swtpm_Init() Cannot connect to swtpm TPM socket
Jan 11 14:19:57 examplehost service[13245]: ERROR:tcti:src/tss2-tcti/tctildr-dl.c:154:tcti_from_file() Could not initialize TCTI file: libtss2-tcti-swtpm.so.0
Jan 11 14:19:57 examplehost service[13245]: WARNING:tcti:src/util/io.c:252:socket_connect() Failed to connect to host 127.0.0.1, port 2321: errno 111: Connection refused
Jan 11 14:19:57 examplehost service[13245]: ERROR:tcti:src/tss2-tcti/tctildr-dl.c:154:tcti_from_file() Could not initialize TCTI file: libtss2-tcti-mssim.so.0
Jan 11 14:19:57 examplehost service[13245]: ERROR:tcti:src/tss2-tcti/tctildr-dl.c:254:tctildr_get_default() No standard TCTI could be loaded
Jan 11 14:19:57 examplehost service[13245]: ERROR:tcti:src/tss2-tcti/tctildr.c:428:Tss2_TctiLdr_Initialize_Ex() Failed to instantiate TCTI
Jan 11 14:19:57 examplehost service[13245]: ERROR:esys:src/tss2-esys/esys_context.c:69:Esys_Initialize() Initialize default tcti. ErrorCode (0x000a000a)
Jan 11 14:19:57 examplehost systemd[13245]: Failed to initialize TPM context: tcti:IO failure
Jan 11 14:19:57 examplehost systemd[13244]: sergeantbot-staging.service: Failed to set up credentials: Protocol error
Jan 11 14:19:57 examplehost systemd[13244]: sergeantbot-staging.service: Failed at step CREDENTIALS spawning /srv/bot/prod/service: Protocol error

The problem was with one of my hardening options. Specifically with PrivateDevices=true. If the option is set systemd only gives access to a very limited subset of pseudo-devices (null, zero, pty). So we need to do this differently. One possibility would be to disable this option. But I’m not a big fan of that. Another is to use the DevicePolicy option. There we can specify a device policy. There are 2 options: strict or closed. Closed allows access to pseudo-devices, and strict disallows all access. Since having access to certain pseudo-devices is not a problem, we’ll set this to closed for now. Then we can allow specific devices with the DeviceAllow option. Just look for the device that throws the error in the logs (tpm0 and tpmrm0 in my case), and allow it explicitly:

DevicePolicy=closed
DeviceAllow=/dev/tpm0
DeviceAllow=/dev/tpmrm0

After that, restart the service and everything should work fine.

That concludes our quick dive into systemd credentials. I really like this feature and I think it is a very good way of passing secrets to a service (definitely better than using environment variables!)

I hope you enjoyed the article!

When I configured the services, I did it inconsistently because I didn’t really understand the different authentication options. Most of the time I would just create a username and password, contact the server over TCP/IP, and call it a day. This meant tracking users + passwords in external password managers, and writing long passwords in configuration files. I knew about postgresql sockets but never really used them. That is, until I discovered peer authentication. This allows postgres to authenticate users locally. You can then specify which users have access to which databases in pga_hba.conf. This allows you to easily add new users and give them permissions for databases, without having to fuck around with passwords. You can configure it by adding the specific users and databases that you want to activate peer authentication for.

For example: if you configure a gitlab instance, and want to give the gitlab user access to the gitlab database, add the following to pg_hba.conf:

# TYPE  DATABASE        USER            ADDRESS                 METHOD

# "local" is for Unix domain socket connections only
local   all             postgres                                peer

# All other users that can access the socket
local   foo             foo                                     peer
local   bar             bar                                     peer
local   gitlab          gitlab                                  peer

This provides an easy way to ensure only specific users are allowed to access specific databases. It has the added benefit of being faster and more reliable, since it uses a Unix socket. I also removed the network authentication for localhost, and disabled listening on a network socket, since that is no longer needed.

/var/lib/postgres/data/postgresql.conf

listen_addresses = ''

Most of the tutorials I found on the internet were quite old (suggesting to RSA with 1024 bits!). So I thought of writing a quick tutorial that uses settings that are a bit more modern. I’ll be describing 2 methods. One with PuTTY, the other one with the built-in BSD OpenSSH client under Windows 10.

Built-in OpenSSH

This is the recommended way of doing things, and it’s not fundamentally different from configuring SSH on *NIX platforms. First, open a PowerShell prompt. In that window, type:

ssh-keygen -t ed25519

-t ed25519 tells OpenSSH to create an ed25519 key. This is a form of elliptic curve DSA. It is very fast and also more secure than “small” (1024-2048 bit) RSA keys. I would recommend this algorithm, although you can also use standard NIST ECDSA.

OpenSSH should start a dialog. If you’ve never created an SSH-Key, you can just hit enter to choose the default path. It’s not important where you save the key, just make sure it’s in a place only your user can access (the default location is fine for that). After you enter the path, it will ask you for a passphrase. It’s recommended to choose a passphrase, although it’s not strictly necessary if you just use the key for this functionality, since it doesn’t really have high stakes. If you plan on using the key for other things, I would recommend choosing a passphrase.

After that, go into the path were it saved the key, and open the public key file (in my case id_ed25519.pub). Then send the text string to your sysadmin. He will enter that key in his authorized_keys file.

PuTTY

This is the “old-school” way of doing things. If you’re running Windows 10, I would recommend you use the built-in OpenSSH client.

First you need to download and install PuTTY. After installing it, we first need to generate a key. After that, we’ll create a profile for each command. That means we can just select and open the profile to execute our commands. To generate a key, you need to execute a program called PuTTYgen. The correct options are shown in the following screenshot. We want an ed25519 key, so choose EdDSA under Type of key to generate, and make sure ed25519 is selected as the type.

After that, click generate. Generation should be very fast on a modern computer. After generating you will see the following:

You can just copy the public key out of the box at the top. Send that to your sysadmin. Afterwards, you can choose a passphrase and klick on save private key. Choose an appropriate location (c:\User\username.ssh is a good place). You can also save the public key, but that one can also be generated from the private key later, so you don’t need to do it at this time.

After that, open up PuTTY. Enter the appropriate hostname into the dialog (ask your sysadmin for the correct FQDN). After that, we need to do 2 things. First we need to manually specify the SSH key. For that, go into Connection -> SSH -> Auth. In the last option, we can specify the private key file. Just hit browse, and select the private key.

Because PuTTY tries to connect to an interactive SSH immediately (which isn’t allowed for security reasons), we need to configure some additional things. The best way to do this is to create 3 sessions for each of our 3 commands. To associate a command with a specific connection, we go to Connection -> SSH. There we can specify a remote command.

After that, go back to Session and define a session name under Saved Sessions. After that, save the session.

Repeat the process for the other commands. After that, you can just open putty, load the session and click on Open to execute the command. Now everything should work. Good luck and have fun with Valheim!

The supply is mounted internally, and then connected to a plug that is mounted to the panel. This cable uses a right-angle-connector. The problem with this is, I didn’t think of the fact that my power supply mounts the other way. So it needs a connector that is angled towards the other side. Finding a right-angle connector wasn’t that difficult, but I didn’t find one with a panel-mountable socket on the other side. So, for a long time I just left the cable dangling out of the back. I decided to rectify this a few days ago.

My first idea was to zombify both cables I had by cutting them up and then soldering them back together. If this would be a low-voltage project, I wouldn’t think twice about it. But I’ve always felt a bit uneasy with high voltage. And I wasn’t sure if it’s OK to just solder in that case. So I asked around in my social circles if anyone knew an electrician. My good friend, the Broom Besen told me: “I’m not an electrician, but when I worked on construction sites I always did it that way”. Very reassuring. So, I decided to listen to his advice. We decided the best way would be to crimp the cable onto a panel-mounted plug.

So that’s what I did.

I know this is a bit different from my other projects, but I hope you liked reading it!

When I initially set it up, I had just one (Desktop!) HDD: A Seagate Barracuda with 8TB of storage. For the filesystem, I chose Btrfs at the time because I heard some good things about it, and looking through the features it seemed to do what I wanted. But the longer I used it, the more problems creeped up. It doesn’t have native filesystem encryption, so I had to use LUKS. It supports Quotas, but doesn’t have a good way of displaying how much storage a specific subvolume/snapshot actually uses. It supports snapshots, but not too many of them. I also thought that subvolumes were very neat, so I created a lot of them to give each “application” a unique path that I tried to cram into the FHS-Philosophy. This caused more trouble than it solved though, because now I had a lot of different paths that I had to write down. I also made a bash script to mount all of these different subvolumes, and had to frequently look inside this script to keep all of the paths and subvolumes together. And with every Service I added, I needed to update this mounting script. This wasn’t really hard or complicated, but very annoying.

#!/bin/bash

DISK1='/dev/sde1'
CNAME='HDD_1'
ROOT_PATH='/mnt'
JELLYFIN="jellyfin"
NAS_ROOT="/media/smbnas/"
BACKUP_ROOT="backup"


echo "Opening disk"
sudo cryptsetup open "$DISK1" "$CNAME" || echo 'failed'

echo "Mounting btrfs root"
sudo mount "/dev/mapper/$CNAME" "$ROOT_PATH/$CNAME" -o compress=zstd,autodefrag
|| echo 'failed'

echo "Mounting btrfs subvol 'jellyfin'"
sudo mount "/dev/mapper/$CNAME" "$NAS_ROOT/$JELLYFIN" -o
compress=zstd,autodefrag,subvol="/$JELLYFIN" || echo 'failed'

echo "Mounting btrfs subvol 'backup/biggs'"
mount "/dev/mapper/$CNAME" "$NAS_ROOT/$BACKUP_ROOT/biggs" -o
compress=zstd,autodefrag,subvol="/$BACKUP_ROOT/biggs" || echo 'failed'

echo "Mounting nextcloud subvol"
mount "/dev/mapper/$CNAME" "/var/nextcloud" -o
compress=zstd,autodefrag,subvol="/nextcloud" || echo 'failed'

echo "Mounting XXX subvol"
mount "/dev/mapper/$CNAME" "/var/www/XXX.sergeantbiggs.net" -o
compress=zstd,autodefrag,subvol="/tiktok" || echo "failed"

The script I used. It’s a bit of a mess.

I also created another script to automate snapshot creation. This was to protect the files on my SMB shares. If you delete a file from an SMB share on the client side, it’s gone. With this, I would have a chance to restore things if they were accidentally deleted. This made listing the subvolumes a complete shitshow, because the list was completetely cluttered with the snapshots. So a lot of Btrfs tools were just not working very well in my situation and I had to rely on scripts, both self-written and third-party.

The Move

That’s why I decided to move to ZFS. In this article, I will take you on the journey I faced to do this. I decided to this at the same time that I did a major hardware upgrade. Up to that point I just had one Desktop HDD. I decided to replace it with 3 WD Reds, each 8 TB in size. I will be running these in RAIDZ, an improved version of RAID5 that removes the write hole (a general problem in RAID 5, something Btrfs hasn’t been able to solve yet). It is also generally very efficient.

So, how will we go about this whole thing? There are 2 options:

Option 1 would be to replace all the Btrfs subvolumes with the ZFS equivalent (Datasets) and mount them at the exact same locations. The advantage would be that I don’t have to reconfigure the applications that use the storage. The disadvantage would be that my services would be down while the data transfers (and for over 5 TB, that would take a while).

Option 2 would be to mount both volumes, transfer the data and reconfigure the applications after the transfer is complete. The advantage here would that I would just have very short interruptions in service.

I decided to go for option 2. This has the other advantage that it forces me to organise the data in a different way, which is hopefully more sensible than scores of subvolumes that mount to different folders somewhere in the hierarchy.

First I had to decide how I wanted to install ZFS. If you don’t know about this problem, here’s a short explainer. ZFS uses a free software license, the CDDL This is a so-called copyleft license (a license which restricts using the code for proprietary software). One problem with this is that it is incompatible with the GPL. This means ZFS can’t be distributed with the Linux kernel. There is a third-party kernel module, called zfsonlinux, which we will be using. For Arch Linux, there are a few options of getting this kernel module. The recommended way of getting it is by installing a patched kernel that includes the module. The advantage is that this is the easiest solution, and installation/upgrade (which, for all intents and purposes, is always the same in Arch!) is very fast and not any different to upgrade a normal kernel. The disadvantage is that I have to wait for the maintainers to release new versions of the modified kernel. This can sometimes take months. Since I want to keep this server very up to date (and close to upstream) for security reasons, I didn’t like this option very much. The other option would be to use DKMS. This recompiles the module into the kernel every time there is a kernel update. Although the update takes slightly longer, this has the advantage of not having to wait for the maintainers to release a new patched kernel. The DKMS version is installed with the following command: pacman -S zfs-dkms linux-headers. After installing, we can start to configure ZFS.

ZFS Configuration

First we create a pool for our hard drives. Zfs-on-linux recommends using device ids when creating pools that are smaller than 10 devices. The device ids on Arch Linux can be found by showing the contents of /dev/disk/by-id.

lrwxrwxrwx 1 root root  9 Jun 26 13:53 ata-Intenso_SSD_Sata_III_AA000000000000016985 -> ../../sda
lrwxrwxrwx 1 root root 10 Jun 26 13:53 ata-Intenso_SSD_Sata_III_AA000000000000016985-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Jun 26 13:53 ata-Intenso_SSD_Sata_III_AA000000000000016985-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Jun 26 13:53 ata-Intenso_SSD_Sata_III_AA000000000000016985-part3 -> ../../sda3
lrwxrwxrwx 1 root root  9 Jun 26 13:53 ata-WDC_WD80EFBX-68AZZN0_VRG0M7DK -> ../../sdc
lrwxrwxrwx 1 root root  9 Jun 26 13:53 ata-WDC_WD80EFBX-68AZZN0_VRG8G6LK -> ../../sdb
lrwxrwxrwx 1 root root  9 Jun 26 13:53 ata-WDC_WD80EFBX-68AZZN0_VRG8ZJNK -> ../../sdd

In my case, this is /dev/sdb, /dev/sdc, and /dev/sdd. Using the device ids has the advantage that Linux changes the “classic” identifiers (sd[a-z]) if the boot order is different. So adding more disks, or just a USB drive, can change these. This is obviously something we want to avoid. So, to create our pool we use the following command:

zpool create -f -m /media/ zfsnas raidz ata-WDC_WD80EFBX-68AZZN0_VRG0M7DK ata-WDC_WD80EFBX-68AZZN0_VRG8G6LK ata-WDC_WD80EFBX-68AZZN0_VRG8ZJNK

If the command is successful, there should be no output. We can check the status of our pool with zpool status. The output should look like this:

  pool: zfsnas
 state: ONLINE
config:

        NAME                                   STATE     READ WRITE CKSUM
        zfsnas                                 ONLINE       0     0     0
          raidz1-0                             ONLINE       0     0     0
            ata-WDC_WD80EFBX-68AZZN0_VRG0M7DK  ONLINE       0     0     0
            ata-WDC_WD80EFBX-68AZZN0_VRG8G6LK  ONLINE       0     0     0
            ata-WDC_WD80EFBX-68AZZN0_VRG8ZJNK  ONLINE       0     0     0

errors: No known data errors

So the pools are automatically imported on boot we need to enable 2 systemd services.

systemctl enable zfs-import-cache
systemctl enable zfs-import.target

After that, we want to create datasets. These look like folders, but allow us to use other features, like mounting them under a different path and setting quotas. We create these datasets with zfs create <nameofzpool>/<nameofdataset>. In our case we also want to enable encryption. So the command looks like this

zfs create -o encryption=on -o keyformat=passphrase zfsnas/cryptset

ZFS asks us for a passphrase, which we then need to enter twice. After that we can start to create datasets for our different services. In my case, I have one dataset for my Nextcloud server, one for my SMB NAS, and one for the backups of my different clients. We also set quotas for each dataset.

zfs create zfsnas/cryptset/smb
zfs create zfsnas/cryptset/nextcloud
zfs create zfsnas/cryptset/backup
zfs set quota=8TB zfsnas/cryptset/smb
zfs set quota=200GB zfsnas/cryptset/nextcloud
zfs set quota=4TB zfsnas/cryptset/backup

After we have prepared the datasets, we transfer everything from the old disk to the new array. I’m doing this with rsync.

Reconfiguring Services

After transferring everything, we need to reconfigure our services. The one I was most afraid of was nextcloud. There is this help article that begins with this scary looking disclaimer:

First of all: Changing data directory after installation is not officially supported. Consider re-installing Nextcloud with new data directory, if you did not use it too much/added users/created shares/tags/comments etc.

That doesn’t inspire much confidence. The problem is that NC stores information about all files in its database. So this database to be either manually updated (error-prone) or rebuilt (losing basically all metadata like shares, comments, etc). I decided to do the second one, since I’m not a DBA by any stretch of the imagination, and I didn’t have many shares. If someone gets their data access cut off, they’ll complain anyway. Then I’ll be informed and can create a new share.

The first thing I did was change the location of the data directory. To do this, we need to change /etc/webapps/nextcloud/config.php. It contains a variable called datadirectory. After I changed this variable, I ran occ maintenance:repair for good measure. To my surprise, this was all that I needed. All files (and share links) were there. For Samba, I just changed the config (smb.conf) to the new location.

After that, everything worked. I’m looking forward to the new NAS, and I hope I will lose some of the hassle I had with Btrfs.

Very frequent updates.

My friends (and I) soon grew weary of constant reminders to update the server so I set up a way to do it automatically. First, the organisational problem. When can I restart the server safely, and when does it disturb the least amount of users?

Since I’m a big believer in democracy I decided to use a doodle. Well, not doodle specifically but the DFN Terminplaner, an implementation of Framadate. Maybe I’ll host it on my own server some day? If I do, dear reader, I’ll be sure to tell you about it in a future blog post. But for now, back to the poll. After polling my friends, the results were pretty clear:

Not quite the typical maintenance time for servers, right? Alas, I don’t really care about that. If they are happy with it, I’m happy with it. So I settled on 12:00 to make sure they are sleeping (should work in 99% of cases).

Now I need a way to automatically run the update command.

The “classic” Linux/Unix way would be to use cron. But why use something boring that was invented for V7 Unix (almost 42 years ago!)? systemd can do something similar, with some caveats. So let’s just do that. We will all have to bow to Poettering and Sievers some day, even if we lose our home directories in the process. The old gods are dead. The new ones are just as terrifying. They could only be stopped by a combined uprising of devuan (remember those guys?) and gentoo users. Or maybe that one guy that approached me during a Linux install party because he wanted to use eduroam on his NixOS machine without using wpa_supplicant.

Anyway.

I decided to use systemd timers.

With this approach we need two systemd units. A service that runs the program and a timer that specifies the time.

update-valheim.service:

[Unit]
Description=Update Valheim server
After=network-online.target

[Service]
WorkingDirectory=/home/steam/Valheim/
ExecStart=/home/steam/update_valheim.sh
Type=oneshot

[Install]
WantedBy=multi-user.target

update-valheim.timer:

[Unit]
Description=Valheim updater timer

[Timer]
OnCalendar=*-*-* 12:00:00

[Install]
WantedBy=timers.target

The line OnCalendar=*-*-* 12:00:00 specifies that we want to execute it every day at 12:00.After that, we just start/enable the timer

systemctl --user start update-valheim.timer
systemctl --user enable update-valheim.timer

The script to actually update the server has the following content:

#!/bin/bash

OUTPUT=$(steamcmd +login anonymous +force_install_dir /home/steam/Valheim +app_update 8966 60 +quit | tail -n 1)

# If app is out of date, restart Service
if ! [[ "$OUTPUT" == "Success! App '896660' already up to date." ]]; then
        echo "Restarting valheim"
        systemctl --user restart valheim.service
fi

If steamCMD detects that the app is already up to date it will output Success! App '896660' already up to date.

We don’t restart the server, if steamCMD didn’t change any of the files. If it did, we restart the service with systemd. And that’s that. A pretty simple solution for a pretty annoying problem. Hopefully it works. If it doesn’t I will of course update you, dear reader!

Setting up your own server is pretty easy, although there were some pitfalls. To document those, and also to show how I did it differently (because of course I did), I’m writing this blog post.

The first thing you need to do is to install SteamCMD. This is a CLI-client for Steam that you need to install the dedicated server. After you install it, it’s a good idea to create a separate user for your games:

useradd -m steam

I added an directory in the home folder of the steam user called Valheim. To install it to that folder, you need to enter the following command

steamcmd +login anonymous +force_install_dir /home/steam/Valheim +app_update 896660 +exit

You can get the ID (896660) from this steam database. Just search for Valheim, and it should come up. You can also use the command above to update your server.

Most guides recommend you use the Bash script inside the folder to start the server.

export templdpath=$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=./linux64:$LD_LIBRARY_PATH
export SteamAppId=892970


echo "Starting server PRESS CTRL-C to exit"

# Tip: Make a local copy of this script to avoid it being overwritten by steam.
# NOTE: Minimum password length is 5 characters & Password cant be in the server name.
# NOTE: You need to make sure the ports 2456-2458 is being forwarded to your server through your local router & firewall.
./valheim_server.x86_64 -name "server_name" -port 2456 -world "world_name" -password "your_password"

export LD_LIBRARY_PATH=$templdpath

I decided against it, because we might as well use a systemd service file. After all, that’s what they’re designed for! The systemd service file should look a little like this:

[Unit]
Description=Valheim Headless Server
After=network-online.target

[Service]
WorkingDirectory=/home/steam/Valheim/
ExecStart=/home/steam/Valheim/valheim_server.x86_64 -name "server_name" -port 2456 -world "world_name" -password "your_password"
KillSignal=SIGINT
Environment="LD_LIBRARY_PATH=./linux64:$LD_LIBRARY_PATH"
Environment="SteamAppId=892970"
Environment="TERM=xterm"

[Install]
WantedBy=multi-user.target

One important thing to note is the line Environment="TERM=xterm". For some reason, unity games need TERM set to xterm. Otherwise the game doesn’t work. Since I’m using tmux (with TERM set to tmux-256color), I have to set it to xterm. I also set the other Environment variables that were in the Bash script.

One thing you need to do is change your firewall to allow ports 2456-2458. If you use nftables it should look like this.

net filter {
    chain input {
        type filter hook input priority 0;
        policy drop;
        iif lo accept

        # Early dropping of invalid packets:
        ct state invalid drop

        # Accept traffic originating from us
        ct state established,related accept

        # Valheim
        tcp dport 2456-2458 accept
        udp dport 2456-2458 accept
    }

    chain output {
        type filter hook output priority 0;
        policy accept;
        oif lo accept
    }
}

If you had a previous world you need to copy it over. On windows, it can be found in C:\Users\user\AppData\LocalLow\IronGate\Valheim\worlds. In that folder you should have two files. You need to copy those two files into ~/.config/unity3d/IronGate/Valheim/worlds. If you don’t have that folder, you can just run the binary once to create it.

After that, you’re technically ready to go. I was faced with one more problem though: the Server uses about 30% of my total CPU resources, even when no one is connected! This is a needless waste of my resources, since my machine runs 24/7. So I decided I would only start it when people actually wanted to play. My electric bill will thank me.

But when I relayed this decision to my friends, the wailing and lamentations began. “How will we be able to play it whenever we want? We can’t live with this! We need a way to start the server ourselves!!”. My assurances of a swift reaction when contacted via Discord, fell on deaf ears. So I wracked my brain. How can I make sure my (mischievous) friends can start (and stop!) the server themselves, without giving them shell access to my machine?

My solution consisted of two components

• systemd/User
• SSH Keys + the authorized_keys file

Systemd/User is a way of creating systemd units that can be controlled by the user. They are placed in the users home directory (~/.config/systemd/user/). They can then be controlled by passing the --user flag when running systemd commands. This allowed me to circumvent giving them root/sudo access, which is necessary for normal systemd operation.

NB: If you want to use systemd/User configuration, you have to configure lingering. this makes sure that the service is not stopped after the user has logged out:

loginctl enable-linger steam

After that I needed to find a way for them to execute commands without being able to log in. And SSH has a way of doing that! In the authorized_keys file (which contains the public keys for authentication) you can set a custom command that will be executed when they connect via SSH. So for each user (key) you get the following line in authorized_keys.

# Limited users (can only start, stop, and query status of valheim server)
command="./manage_valheim.sh $SSH_ORIGINAL_COMMAND" ssh-rsa KEY_GOES_HERE

The variable $SSH_ORIGINAL_COMMAND contains the command they enter on the command line when using SSH. This command can be either start, stop, or status. The command is parsed with manage_valheim.sh, a Bash script.

#!/bin/bash

SCTL="systemctl --user"

function usage() {
	echo "Usage: start stop status"
}

case $1 in
	start)
		$SCTL start valheim.service
		;;
	stop)
		$SCTL stop valheim.service
		;;
	status)
		$SCTL status valheim.service
		;;
	*)
		echo "Command not recognized!"
		usage
		;;
esac

The script parses user input and then starts the appropriate action (start, stop, or status). This way, the users that have public keys in authorized_keys can “manage” the server without having complete shell access.

Now my friends are happy, because they have an easy, user friendly way of managing the server. I’m happy because they don’t have any more access than they need. And, perhaps most importantly, my electric bill is happy.

Thanks for the read, I hope you enjoyed it. And I hope it will help you in setting up your own Valheim server. And one last thing: Play Valheim! It’s a great game, shout-out to the Developer IronGate Studio. You guys did a great job!

A health monitoring system with built in notifications.

I’m using a notification system that uses Email and also a Telegram bot. The bot is especially handy, since I get the notifications pretty much instantly when an alarm is raised, and I receive notifications on the go. One day I received some puzzling alarms.

These three messages in succession were strange. Normally I get some packet loss and disk backlog on the system. I use it as a NAS, and this can happen when transferring large files. But the last message is really strange. 11% packet loss? That’s huge! So I started wondering what it was. I logged into the Netdata web interface. The CPU graph immediately caught my eye.

What were these strange spikes? I thought the best Idea would be to just login to the server and check it locally. So I fired up htop (still the best task manager) and just waited for the spike. Unfortunately, since the spikes are very short, I wasn’t able to take a screenshot. But the culprit was soon found. It was Factorio!

Factorio is a survival/management game and I occasionally play it with some friends of mine, hence the server.

After I found out that Factorio was the culprit, I looked into it a bit further. With journalctl -u -f factorio.service I looked at the live log of the service.

31 04:01:15 stern systemd[1]: factorio.service: Scheduled restart job, restart counter is at 3474.
Dec 31 04:01:15 stern systemd[1]: Stopped Factorio game headless server.
Dec 31 04:01:15 stern systemd[1]: Started Factorio game headless server.
Dec 31 04:01:15 stern factorio[100820]:    0.000 2020-12-31 04:01:15; Factorio 1.0.0 (build 54889, linux64, headless)
Dec 31 04:01:15 stern factorio[100820]:    0.000 Operating system: Linux
Dec 31 04:01:15 stern factorio[100820]:    0.000 Program arguments: "/usr/bin/factorio" "--server-settings" "/etc/factorio/server-settings.json" "--use-server-whitelist" "--server-whitelist" "/etc/factorio/server-whitelist.json" "--server-adminlist" "/etc/factorio/server-adminlist.json" "--start-server-load-latest"
Dec 31 04:01:15 stern factorio[100820]:    0.000 Read data path: /usr/share/factorio
Dec 31 04:01:15 stern factorio[100820]:    0.000 Write data path: /var/lib/factorio/.factorio [82445/117495MB]
Dec 31 04:01:15 stern factorio[100820]:    0.000 Binaries path: /usr
Dec 31 04:01:15 stern factorio[100820]:    0.010 System info: [CPU: Intel(R) Celeron(R) J4105 CPU @ 1.50GHz, 4 cores, RAM: 7607 MB]
Dec 31 04:01:15 stern factorio[100820]:    0.010 Environment: DISPLAY=<unset> WAYLAND_DISPLAY=<unset> DESKTOP_SESSION=<unset> XDG_SESSION_DESKTOP=<unset> XDG_CURRENT_DESKTOP=<unset> __GL_FSAA_MODE=<unset> __GL_LOG_MAX_ANISO=<unset> __GL_SYNC_TO_VBLANK=<unset> __GL_SORT_FBCONFIGS=<unset> __GL_YIELD=<unset>
Dec 31 04:01:15 stern factorio[100820]:    0.010 Running in headless mode
Dec 31 04:01:15 stern factorio[100820]:    0.014 Loading mod core 0.0.0 (data.lua)
Dec 31 04:01:15 stern factorio[100820]:    0.122 Loading mod base 1.0.0 (data.lua)
Dec 31 04:01:16 stern factorio[100820]:    0.691 Loading mod base 1.0.0 (data-updates.lua)
Dec 31 04:01:16 stern factorio[100820]:    0.974 Checksum for core: 2630831588
Dec 31 04:01:16 stern factorio[100820]:    0.974 Checksum of base: 3509992273
Dec 31 04:01:17 stern factorio[100820]:    1.347 Prototype list checksum: 3301461508
Dec 31 04:01:17 stern factorio[100820]:    1.444 Info PlayerData.cpp:68: Local player-data.json available, timestamp 1609383665
Dec 31 04:01:17 stern factorio[100820]:    1.444 Info PlayerData.cpp:75: Cloud player-data.json unavailable
Dec 31 04:01:17 stern factorio[100820]:    1.447 Factorio initialised
Dec 31 04:01:17 stern factorio[100820]:    1.448 Info HttpSharedState.cpp:54: Downloading https://auth.factorio.com/api-login?api_version=4
Dec 31 04:01:17 stern factorio[100820]:    2.217 Info AuthServerConnector.cpp:41: Auth server returned error: {"data":{},"error":"login-insufficient-membership","message":"Your account doesn't own the game.  Please buy it to log in.  If you are playing through Steam, please check your Steam account is linked to your Factorio account"}
Dec 31 04:01:17 stern factorio[100820]:    2.217 Error CommandLineMultiplayer.cpp:345: Hosting multiplayer game failed: Your account doesn't own the game. Please buy it to log in.
Dec 31 04:01:17 stern factorio[100820]:    2.223 Info ServerMultiplayerManager.cpp:136: Quitting multiplayer connection.
Dec 31 04:01:17 stern factorio[100820]:    2.223 Info ServerMultiplayerManager.cpp:769: updateTick(4294967295) changing state from(Ready) to(Closed)
Dec 31 04:01:18 stern factorio[100820]:    2.288 Goodbye

Note the restart counter, which is currently at 3474! The log had millions of entries like this, and the service was restarting every few seconds. Apparently it was doing this for weeks/months now. The reason was that my Steam account wasn’t linked to my Factorio account anymore. After relinking, it worked again.

So, what did we learn from this experience. First of all, look at your systemd units, and how they are configured. The problem here was two-fold. First off, the AUR package for factorio-headless isn’t… ideal. It’s been out of date for years and does really weird things. The systemd unit it ships makes no sense in a lot of ways. IMHO, shipping a systemd unit for something like this makes no sense anyway. Anyone who sets up a Factorio server will likely have a specific way of configuring it. When I first installed the AUR package I had to change a lot to the systemd unit to make it work in my situation. I’ll include the unit below, maybe you can spot the mistake.

[Unit]
Description=Factorio game headless server
Documentation=http://www.factorio.com/
After=network.target

[Service]
User=factorio
Group=factorio
EnvironmentFile=/etc/conf.d/factorio
WorkingDirectory=/var/lib/factorio
ExecStart=/usr/bin/factorio --server-settings /etc/factorio/server-settings.json --use-server-whitelist --server-whitelist /etc/factorio/server-whitelist.json --server-adminlist /etc/factorio/server-adminlist.json --start-server-load-latest
TimeoutStopSec=30
KillSignal=SIGINT
RestartSec=10
Restart=on-failure

[Install]
WantedBy=multi-user.target

So, did you catch it? It’s line 15, Restart=on-failure. I’ve never been a big fan of autorestarting units, but for something like Factorio it makes no sense at all. It’s not a critical service. And if the service fails, I want to know why, and I don’t want it to just autorestart over and over. Which brings me to the second problem:

I didn’t notice. I just installed the service, configured it until it worked and didn’t give it much thought afterwards. This really goes to show the benefit of knowing how systemd works and writing your own units, instead of just copying other peoples work, who may or may not know what they are doing. Even as I am writing this article, I noticed another mistake in the service file.

The line After=network.target should be After=network-online.target. Here’s why.

To cite the page:

“network.target has very little meaning during start-up. It only indicates that the network management stack is up after it has been reached. Whether any network interfaces are already configured when it is reached is undefined. Its primary purpose is for ordering things properly at shutdown.”

network-online.target makes sure the network is actually up, and it is the one you want when you use a service that needs access to the internet (e.g. a Factorio server).

In this experience, I learned a few things:

• Look at your systemd units (ideally configure them manually for things like this)
• Monitoring tools are nice, Netdata is very nice
• If your monitoring tools have alarms, configure them so you get them somewhere where you can actually see them

I have to add that I don’t know if the error was actually connected to Factorio, since the service had this problem for a few weeks/months. But the error is what made me curious, and made me take a closer look.

I hope you enjoyed this little piece. May it help you in your future endeavors!

That said, I very much like the Pi Zero, especially the Zero W. It’s very small, very versatile and very very cheap. It’s especially handy if you want to do things with microelectronics. And it’s exactly what I need for the second function of my DeskJet: the scanning function.

Getting a fully functioning scanning device has been a journey. One of epic proportions. One of heartbreak and failure, but ultimately of success.

The story begins with the fact that SANE doesn’t really have a windows client. Yes, there are deprecated pieces of software floating around SourceForge (who remembers those guys, eh?), but I really don’t like the idea of using one of those. Especially since I will then be dependent on this software for years. So I started writing my own scanning “interface”. I decided to go with a HTTP server, since it is platform-independent. It also allows me to add new features and write my own “API”. The server has a PHP script which just executes a command that scans the image. I would not recommend this approach, because it has a lot of security problems, but it works quite well for me. I’m not too worried about this for my own setup, since the server is only exposed to the LAN and is protected with a password. I don’t recommend this for any kind of production setup though.

The finished scan is saved on an SMB share. The clients in the network can access this share. The share is read-only, so users can’t delete files of other users. So the share doesn’t get filled with thousands of documents, the files in the folder will be periodically deleted. This is done with a simple Bash script that is periodically executed with a cron job.

For this setup you will need: a web server + a PHP implementation. I’m using NGINX and PHP-FPM. You also need some network share for the scanned files.

First we install php-fpm and nginx and configure them accordingly. I’ll include examples of the configuration files.

server {
	server_name scan.example.lan;
	listen 443 ssl http2;
	ssl_certificate /etc/ssl/cert.pem;
	ssl_certificate_key /etc/ssl/key.pem;

	auth_basic "Scan Web Interface";
	auth_basic_user_file scan_authfile;

	root /var/www/scan.example.lan;
	location / {
		index               index.html;
	}

	location ~ \.php$ {
		# 404
		try_files $fastcgi_script_name =404;

		include fastcgi_params;

		fastcgi_pass                        unix:/run/php-fpm/php-fpm.sock;
		fastcgi_index                       index.php;
		fastcgi_buffers                     8 16k;
		fastcgi_buffer_size                 32k;

		fastcgi_param DOCUMENT_ROOT         $realpath_root;
		fastcgi_param SCRIPT_FILENAME       $realpath_root$fastcgi_script_name;
	}

The web interface itself is as simple as possible, because web development is not my forté. Look at the HTML and PHP as a minimalistic example to get you started:

<!DOCTYPE html>
<html>
    <head>
        <meta charset="UTF-8">
        <title>Best scanning page ever</title>
        <link href="/styles.css" rel="stylesheet" type="text/css" media="all">
    </head>
    <body>
         <form action="index.php" method="post">
            <input type="submit" name="scan" value="Click to scan">
        </form>
    </body>
</html>

<?php
$datetime = date("Y-m-d_His");
exec("scanimage -o /media/smb/scan/${datetime}.jpg --format=jpeg --resolution 300");
echo "Scan done";
?>

The share definition in smb.conf:

[scan]
comment = Scan Share
path = /media/smb/scan/
valid users = @users
public = no
writable = no
printable = no

The bash script that periodically deletes the folder. Because the name of the file is the date, it is relatively trivial to delete all files that are older than a certain time.

#!/bin/bash
shopt -s nullglob

SCAN_FOLDER=/media/smb/scan/

function error_exit() { echo "Error, exiting!"; exit 1; }

function delete_snapshots() {
CUTOFF="$1"
for scan_file in *; do
    snapshot_date=$(echo "$scan_file" | awk -F '_' '{ print $1 }')
    # If date is invalid, exit
    date --date "$snapshot_date" &> /dev/null || error_exit

    if [[ $(date --date "$snapshot_date" +%s) -lt $(date --date "$CUTOFF" +%s) ]]; then
        rm "$scan_file"
        echo "Deleted file: $scan_file"
    fi
done
}

cd "$SCAN_FOLDER" || error_exit

# Delete files that are older than one day
delete_snapshots "1 day ago"

Some settings for php-fpm need to be changed. This is necessary, because php-fpm includes is hardened by default. One of these is the fact that it can’t access physical devices on the system (the files in /dev/). Instead it only gets access to a set of virtualized pseudo-devices (e.g. /dev/random). This is a very sensible default because in 99% of cases you really don’t need that access when running a web server. But in our case we need access to the scanner. To change these defaults, we need to change the systemd service file for php-fpm.

To do this run systemctl edit php-fpm.service and add the following lines:

[Service]
PrivateDevices=false

We also need to add the user http (or whatever user runs your nginx instance) to the scanner group so the server has the necessary permissions to scan:

usermod -aG scanner http

If you’re wondering how the Raspberry Pi plays into all of this, don’t despair dear reader! It will all become apparent now. The problem with this setup is that the scanner is in the living room. So every time I want to scan a document, I have to:

1. Walk to the scanner
2. Put a page in the scanner
3. Walk back to my computer
4. Navigate to the web interface
5. Click on Scan
6. Repeat for every page

This is highly annoying, and I wanted a simpler solution. One that enables me to scan while standing next to the scanner. Then I can just scan all pages and copy them from the share when I’m done. The first thing that sprang to mind was to write an App/Shortcut for my Smartphone. While this would work well, there are some things I don’t like about this solution. I often don’t have my phone in my pocket when I’m home, because I don’t need it. I would have to take the phone out, navigate to the page, input the credentials, etc. The ideal solution would be a button on/next to the scanner. I thought about a lot of solutions. At first I wanted a button that is connected over USB. But I didn’t really know how to deal with authentication so I dropped the idea.

After giving it some thought, I decided to go to the master of bizarre solutions for difficult problems: my good friend radow. They once unironically suggested to implement a print server by setting up an email server, which then prints the attachments of the emails it receives. This “implementation” was posited as a solution to the problem “I need a networked printer for me and my 3 roommates”. We also once filed down the flat blade on a DVI-I plug, so it would fit into a DVI-D socket.

Spoiler alert: it did not work.

After some deliberation we came to the conclusion that my best bet would be some kind of networked solution, since I already have the HTTP interface. So I settled on using a raspberry pi. That way I can solder a button to the GPIO pins. When the button is pressed a script is executed that initiates a scan.

#!/usr/bin/env python3

import sys
import RPi.GPIO as GPIO
import requests
from time import sleep

PIN_BUTTON = 18
GPIO.setmode(GPIO.BCM)

GPIO.setup(PIN_BUTTON, GPIO.IN, pull_up_down=GPIO.PUD_DOWN)

while True:
    try:
        GPIO.wait_for_edge(PIN_BUTTON, GPIO.RISING)
        requests.post('https://scan.example.lan/index.php', auth=('user', 'password'))
        sleep(0.5)
    except KeyboardInterrupt:
        GPIO.cleanup()
        sys.exit()

To deal with power I decided to use a mountable micro usb socket that extends to a usb plug. We can then just plug in another micro usb cable into the back of the printer.

I don’t have any kind of dremeling tool, so I just decided to drill a round hole into the plastic and mount it that way. It is an ugly solution, but it works.

So that’s the project finished. It took quite a while, and was very frustrating at times, but it all worked out in the end. I’m very happy with the results.

Generally I don’t like to stick random Wifi devices in my LAN, and certainly not proprietary devices that will probably never have their firmware updated. But I don’t want to drag this thing out of a closet every time I want to print something.

Luckily I have a solution for that! I have a small server sitting in my living room. It’s running Arch Linux (Of course, what else?). So if I can connect that printer to the Server via USB, I can then send print jobs to the server. So let’s get started doing that.

If you want to follow along, here’s what you need to have at home

• A HP DeskJet 3750
• A headless server, running Arch Linux (not technically necessary, but I will judge you if you use any other Distro)
• A USB Cable (A to B)

So, have you got all that stuff? Great, let’s begin.

We will be using a combination of Samba, CUPS and Nginx. We use hplip for the printer drivers. First we install the packages.

pacman -S cups samba nginx hplip

If you use Arch Linux, don’t forget to start and enable the services.

Let’s dive into the configuration files!

For CUPS, we can just leave the configuration file as is. We will be running the web server on localhost, and then access it using NGINX. Why, you ask? Don’t despair, dear reader. This will become apparent in just a few seconds.

I’ll include the NGINX configuration file. Maybe you can already guess it?

server {
    server_name cups.example.lan;
    listen 443 ssl http2;
    ssl_certificate /etc/ssl/cert.pem;
    ssl_certificate_key /etc/ssl/key.pem;

    auth_basic "CUPS Admin Interface";
    auth_basic_user_file cups_authfile;

    location / {
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Server $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass http://localhost:631;
        proxy_http_version 1.1;
        proxy_pass_request_headers on;
        proxy_set_header Connection "keep-alive";
        proxy_store off;
    }
}

Yes reader, you guessed it correctly! If we run the CUPS web interface behind NGINX, we can protect it with a password. This is a handy way to protect web interfaces with passwords, and to unify your web interface configuration.

So, after we add CUPS behind NGINX, we can add the printer. I won’t show this in this post, because it is ridiculously easy to do.

After we’ve added the printer in cups, we can add it to Samba. Then we can use the printer from any Windows device in the network.

[DJ-Printer]
printing = CUPS
comment = HP DeskJet 3750 InkJet Printer
printer = HP_DeskJet_3700_series
path = /var/spool/samba
browseable = no
guest ok = no
writable = no
printable = yes
user client driver = yes
valid users = @users

Now we just need to install the drivers on windows. In these cases I always go for the “basic” or “professional” drivers, since they don’t have as much bloatware. The drivers for this particular printer can be found here

After installing the drivers, we can add the printer. The easiest way to do this, is to go to Devices and Printers -> add a printer. In the next screen, click on The printer that I want isn't listed

After that click on Select a shared printer by name, and enter the name of the printer in the form \\hostname\printer_name. The printer name is the “share name” in the Samba configuration file (the one between square brackets). So in our example it would be \\hostname\DJ-Printer. In the next step, Windows will ask you to choose a driver. Just navigate the list and select the correct manufacturer and driver.

And now we have a fully working network printer!

In the next part of this miniseries, I will talk about the scan function. We will be adding a scan share, and the ability to scan directly from the printer. This will involve some hardware mods, since the printer doesn’t have a scan button. Stay tuned!