2022-01-11

Credential Management With Systemd

Systemd introduced a credential management system in version 247. It can be used to pass sensitive information from the system to the unit. It’s primary usage is passing configuration data, passwords, private keys, etc. The functionality is further explained in systemd.exec (5).

When I was browsing LWN a few days ago, I noticed something in the release notes for the new systemd v250:

   *  Support for encrypted and authenticated credentials has been added.
      This extends the credential logic introduced with v247 to support
      non-interactive symmetric encryption and authentication, based on a
      key that is stored on the /var/ file system or in the TPM2 chip (if
      available), or the combination of both (by default if a TPM2 chip
      exists the combination is used, otherwise the /var/ key only). The
      credentials are automatically decrypted at the moment a service is
      started, and are made accessible to the service itself in unencrypted
      form. A new tool 'systemd-creds' encrypts credentials for this
      purpose, and two new service file settings LoadCredentialEncrypted=
      and SetCredentialEncrypted= configure such credentials.

      This feature is useful to store sensitive material such as SSL
      certificates, passwords and similar securely at rest and only decrypt
      them when needed, and in a way that is tied to the local OS
      installation or hardware.

This looks like quite a nice feature. Let’s try it out!

Like the release notes say, systemd-creds will use a TPM2 chip if it is available. If it isn’t available, it will just use the secret in /var/lib/systemd/credential.secret. So let’s first check if we have a TPM enabled. This can by reading the value of the following file:

cat /sys/class/tpm/tpm0/tpm_version_major

If TPM 2.0 is available, the output of the command will be “2”.

First we need to encrypt our old credentials. This can be done with systemd-creds. This command takes an input file and an output file. The input file is our old unencrypted credential file, the output file will be the encrypted one:

systemd-creds encrypt secrets/old/token_unencrypted secrets/token

After doing that, we just need to change the setting in our .service file (this is an example file for a bot that I’m running on my server):

[Unit]
Description=My Service
After=network-online.target

[Service]
ExecStart=/usr/bin/important-service
Type=exec
user=foo
WorkingDirectory=/home/foo

# Hardening
PrivateDevices=true
ProtectControlGroups=true
ProtectSystem=full
ProtectKernelTunables=true
RestrictSUIDSGID=true
PrivateTmp=true
PrivateUsers=true
ProtectClock=true
ProtectHome=true
ProtectKernelLogs=true
ProtectKernelModules=true
ProtectProc=noaccess
# Everything is RO, we can only write to ReadWritePaths
ProtectSystem=strict
RemoveIPC=true
UMask=0077
ProtectHostname=true
NoNewPrivileges=true

CapabilityBoundingSet=
RestrictNamespaces=true
RestrictAddressFamilies=AF_INET AF_INET6

# Secrets
LoadCredentialEncrypted=token:/home/foo/secrets/token


[Install]
WantedBy=multi-user.target

NB: the ID of the token (the name in front of the colon). Needs to be the same as the file name of the secret.

After that, everything should work fine, right? Not really. When I started the unit for the first time, it immediately crashed. Can you guess what the problem was? I’ll even give you the error logs from the unit:

Jan 11 14:19:57 examplehost service[13245]: ERROR:tcti:src/tss2-tcti/tcti-device.c:442:Tss2_Tcti_Device_Init() Failed to open specified TCTI device file /dev/tpmrm0: Operation not permitted
Jan 11 14:19:57 examplehost service[13245]: ERROR:tcti:src/tss2-tcti/tctildr-dl.c:154:tcti_from_file() Could not initialize TCTI file: libtss2-tcti-device.so.0
Jan 11 14:19:57 examplehost service[13245]: ERROR:tcti:src/tss2-tcti/tcti-device.c:442:Tss2_Tcti_Device_Init() Failed to open specified TCTI device file /dev/tpm0: Operation not permitted
Jan 11 14:19:57 examplehost service[13245]: ERROR:tcti:src/tss2-tcti/tctildr-dl.c:154:tcti_from_file() Could not initialize TCTI file: libtss2-tcti-device.so.0
Jan 11 14:19:57 examplehost service[13245]: WARNING:tcti:src/util/io.c:252:socket_connect() Failed to connect to host 127.0.0.1, port 2321: errno 111: Connection refused
Jan 11 14:19:57 examplehost service[13245]: ERROR:tcti:src/tss2-tcti/tcti-swtpm.c:591:Tss2_Tcti_Swtpm_Init() Cannot connect to swtpm TPM socket
Jan 11 14:19:57 examplehost service[13245]: ERROR:tcti:src/tss2-tcti/tctildr-dl.c:154:tcti_from_file() Could not initialize TCTI file: libtss2-tcti-swtpm.so.0
Jan 11 14:19:57 examplehost service[13245]: WARNING:tcti:src/util/io.c:252:socket_connect() Failed to connect to host 127.0.0.1, port 2321: errno 111: Connection refused
Jan 11 14:19:57 examplehost service[13245]: ERROR:tcti:src/tss2-tcti/tctildr-dl.c:154:tcti_from_file() Could not initialize TCTI file: libtss2-tcti-mssim.so.0
Jan 11 14:19:57 examplehost service[13245]: ERROR:tcti:src/tss2-tcti/tctildr-dl.c:254:tctildr_get_default() No standard TCTI could be loaded
Jan 11 14:19:57 examplehost service[13245]: ERROR:tcti:src/tss2-tcti/tctildr.c:428:Tss2_TctiLdr_Initialize_Ex() Failed to instantiate TCTI
Jan 11 14:19:57 examplehost service[13245]: ERROR:esys:src/tss2-esys/esys_context.c:69:Esys_Initialize() Initialize default tcti. ErrorCode (0x000a000a)
Jan 11 14:19:57 examplehost systemd[13245]: Failed to initialize TPM context: tcti:IO failure
Jan 11 14:19:57 examplehost systemd[13244]: sergeantbot-staging.service: Failed to set up credentials: Protocol error
Jan 11 14:19:57 examplehost systemd[13244]: sergeantbot-staging.service: Failed at step CREDENTIALS spawning /srv/bot/prod/service: Protocol error

The problem was with one of my hardening options. Specifically with PrivateDevices=true. If the option is set systemd only gives access to a very limited subset of pseudo-devices (null, zero, pty). So we need to do this differently. One possibility would be to disable this option. But I’m not a big fan of that. Another is to use the DevicePolicy option. There we can specify a device policy. There are 2 options: strict or closed. Closed allows access to pseudo-devices, and strict disallows all access. Since having access to certain pseudo-devices is not a problem, we’ll set this to closed for now. Then we can allow specific devices with the DeviceAllow option. Just look for the device that throws the error in the logs (tpm0 and tpmrm0 in my case), and allow it explicitly:

DevicePolicy=closed
DeviceAllow=/dev/tpm0
DeviceAllow=/dev/tpmrm0

After that, restart the service and everything should work fine.

That concludes our quick dive into systemd credentials. I really like this feature and I think it is a very good way of passing secrets to a service (definitely better than using environment variables!)

I hope you enjoyed the article!

Credential Management With Systemd

Articles from blogs I read - Generated by openring

Copyleft licenses are not “restrictive” - Drew DeVault's blog (2024-04-19)

"No way to prevent this" say users of only language where this regularly happens - Xe Iaso's blog (2024-04-08)

Beyond GDPR - Ξ (2024-03-22)

A re-introduction to mkosi -- A Tool for Generating OS Images - Pid Eins (2024-01-10)