Running a rust program as a secure systemd service
Polkadot is an implementation of a node
for the Polkadot network written in Rust. In this post,
we’re going to go from an unbuilt codebase to a secure systemd
service.
The Polkadot client consists of a single binary file. It is executed and
configured with a variety of command-line flags. Users can choose to either
compile the binary themselves or fetch a pre-compiled binary from our
releases page. A common way
to run a polkadot node is to copy this binary to /usr/local/bin
and have a
systemd
service file that executes the binary with whatever command-line
flags are desired. While there are guides and advice on how to run a Polkadot
node (such as the fantastic Secure Validator
Setup), there are no
assumptions or expectations about how to invoke the binary.
Building Polkadot
Detailed instructions on how to build the polkadot binary can be found in the repository’s README.md.
Creating and securing a systemd service file
Basic Systemd Service
The majority of modern Linux distributions use systemd
as their init system. Systemd allows us to write what it calls ‘service files’
in order to run programs as services (also known as daemons). Running the program
as a systemd service means we can offload things such as logging, error handling, etc to systemd and not worry about them too much ourselves.
The most basic systemd service file for running polkadot looks like this:
[Unit]
Description=Polkadot node
[Service]
ExecStart=/usr/local/bin/polkadot
[Install]
WantedBy=multi-user.target
This will simply execute the polkadot binary with no additional command-line
flags. This gives us the advantage of logging output to journald
and
allows us to control the service with systemctl
, but there are plenty of
improvements that can be made. Let’s start with a few small ones.
[Unit]
Description=Polkadot Node
After=network.target
Documentation=https://github.com/paritytech/polkadot
[Service]
EnvironmentFile=-/etc/default/polkadot
ExecStart=/usr/bin/polkadot $POLKADOT_CLI_ARGS
User=polkadot
Group=polkadot
Restart=always
RestartSec=120
MemoryHigh=5400M
MemoryMax=5500M
[Install]
WantedBy=multi-user.target
First, we add the dependency After=
which ensures the service only attempts
to start after the system’s network is up. We also specify an
EnvironmentFile
, in which we can specify environment variables that are
available to the binary called in ExecStart
. Polkadot does not currently have
any support for reading its configuration from a config file, but we can use
this technique to specify a list of flags on startup. The environment file
contains the following as an example.
POLKADOT_CLI_ARGS="--database paritydb --wasm-execution Interpreted"
Flags can be added or removed to this environment variable as needed. Unfortunately,
it seems we can’t nest environment variables in this file. So for example, we can’t
have something like POLKADOT_CLI_ARGS="--name $(hostname)"
.
We also specify a user and group to run the service as - by default, the
service will run as root. If this can be avoided, it should be. We specify a
restart policy for the service - if the program crashes, it will always be
restarted but only after a pause of 120 seconds.
Finally, we add a restriction on the max memory the process can use - if it
hits the MemoryHigh
limit, any new memory is heavily reaped. If it hits the
MemoryMax
limit, the OOM killer will kill the process (thus triggering the
aforementioned restart mechanism).
This is a reasonable service file - we have some logic surrounding when to start
and restart the service, some basic privilege separation, and users can configure
the command-line flags used without having to edit the service file and run
systemctl daemon-reload
every time. Now we need to make it secure!
Securing the service file
Systemd comes with a wide variety of configurable options that can be applied
to services in order to improve the security - From basic things like
disallowing the service from gaining superuser privileges, to providing a
strict set of system calls the program is allowed to make. In order to find out
what improvements are even available, we can use the excellent
systemd-analyze
tool.
Note: The service file must be in-place in order to
run systemd-analyze
against it. Copy it to /etc/systemd/system/
and run
systemctl daemon-reload
each time you make an edit.
% systemd-analyze security --no-pager polkadot.service
NAME DESCRIPTION EXPOSURE
✗ PrivateNetwork= Service has access to the host's network 0.5
✓ User=/DynamicUser= Service runs under a static non-root use…
✗ CapabilityBoundingSet=~CAP_SET(UID|GID|P… Service may change UID/GID identities/ca… 0.3
<SNIP>
✗ CapabilityBoundingSet=~CAP_WAKE_ALARM Service may program timers that wake up … 0.1
✗ RestrictAddressFamilies=~AF_UNIX Service may allocate local sockets 0.1
→ Overall exposure level for polkadot.service: 9.2 UNSAFE 😨
Running systemd-analyze
generates a list of all the configuration options
that can be edited in order to reduce the exposure of the service, and sums
each unapplied option in order to generate a final exposure score. As can be
seen above, we scored 9.2/10 (lower is better) with a ranking of ‘UNSAFE’.
Yikes.
As an aside, I highly recommend running systemd-analyze security
with no
additional arguments. It will print an exposure summary for all running
services. You might be surprised at just how many services are considered
unsafe. I know I was.
The process of securing the service essentially consists of going through each
line of systemd-analyze
output and deciding whether that particular
restriction can be implemented without hindering the operation of your program.
For instance, the very first suggestion is PrivateNetwork=
, which restricts
whether or not the service can communicate on the host’s network. Since
Polkadot is a network service, we don’t really want to stop it being able
to communicate over the network.
More detailed descriptions of each suggestion can be found on the manpages
for systemd.exec
and systemd.resource-control
.
Here is the final polkadot.service
file after taking heed of systemd-analyze
’s
recommendations.
[Unit]
Description=Polkadot Node
After=network.target
Documentation=https://github.com/paritytech/polkadot
[Service]
EnvironmentFile=-/etc/default/polkadot
ExecStart=/usr/bin/polkadot $POLKADOT_CLI_ARGS
User=polkadot
Group=polkadot
Restart=always
RestartSec=120
MemoryHigh=5400M
MemoryMax=5500M
CapabilityBoundingSet=
LockPersonality=true
NoNewPrivileges=true
PrivateDevices=true
PrivateMounts=true
PrivateTmp=true
PrivateUsers=true
ProtectClock=true
ProtectControlGroups=true
ProtectHostname=true
ProtectKernelModules=true
ProtectKernelTunables=true
ProtectSystem=strict
RemoveIPC=true
RestrictAddressFamilies=AF_INET AF_INET6 AF_NETLINK AF_PACKET AF_UNIX
RestrictNamespaces=true
RestrictSUIDSGID=true
SystemCallArchitectures=native
UMask=0027
SystemCallFilter=@system-service
SystemCallFilter=~@clock @module @mount @reboot @swap @privileged
[Install]
WantedBy=multi-user.target
That’s quite a few new lines! Some of them may apply to a service you are
writing, some may not. It is important to understand how adding restrictions to
the execution environment of the service will impact the execution of your
program. For example, if I had added a suggestion from systemd-analyze
called MemoryDenyWriteExecute=true
(which stops the program from creating
memory mappings that are both executable and writable), Polkadot would run
just fine with the argument --wasm-execution Interpreted
but would crash when
invoking Polkadot with --wasm-execution Compiled
. It is important to
understand how adding restrictions to the execution environment of the service
will impact the execution of your program.
Running systemd-analyze
on the updated service file gives a much better result:
% systemd-analyze security --no-pager polkadot.service | tail -n1
→ Overall exposure level for polkadot.service: 1.9 OK 🙂
Now that we have Polkadot running as a service and we’re a lot more confident with the security of the service, it’s time to package and distribute.