Polkadot is an implementation of a node for the Polkadot network written in Rust. In this post, we’re going to go from an unbuilt codebase to a secure systemd service.

The Polkadot client consists of a single binary file. It is executed and configured with a variety of command-line flags. Users can choose to either compile the binary themselves or fetch a pre-compiled binary from our releases page. A common way to run a polkadot node is to copy this binary to /usr/local/bin and have a systemd service file that executes the binary with whatever command-line flags are desired. While there are guides and advice on how to run a Polkadot node (such as the fantastic Secure Validator Setup), there are no assumptions or expectations about how to invoke the binary.

Building Polkadot

Detailed instructions on how to build the polkadot binary can be found in the repository’s README.md.

Creating and securing a systemd service file

Basic Systemd Service

The majority of modern Linux distributions use systemd as their init system. Systemd allows us to write what it calls ‘service files’ in order to run programs as services (also known as daemons). Running the program as a systemd service means we can offload things such as logging, error handling, etc to systemd and not worry about them too much ourselves.

The most basic systemd service file for running polkadot looks like this:

[Unit]
Description=Polkadot node

[Service]
ExecStart=/usr/local/bin/polkadot

[Install]
WantedBy=multi-user.target

This will simply execute the polkadot binary with no additional command-line flags. This gives us the advantage of logging output to journald and allows us to control the service with systemctl, but there are plenty of improvements that can be made. Let’s start with a few small ones.

[Unit]
Description=Polkadot Node
After=network.target
Documentation=https://github.com/paritytech/polkadot

[Service]
EnvironmentFile=-/etc/default/polkadot
ExecStart=/usr/bin/polkadot $POLKADOT_CLI_ARGS
User=polkadot
Group=polkadot
Restart=always
RestartSec=120
MemoryHigh=5400M
MemoryMax=5500M

[Install]
WantedBy=multi-user.target

First, we add the dependency After= which ensures the service only attempts to start after the system’s network is up. We also specify an EnvironmentFile, in which we can specify environment variables that are available to the binary called in ExecStart. Polkadot does not currently have any support for reading its configuration from a config file, but we can use this technique to specify a list of flags on startup. The environment file contains the following as an example.

POLKADOT_CLI_ARGS="--database paritydb --wasm-execution Interpreted"

Flags can be added or removed to this environment variable as needed. Unfortunately, it seems we can’t nest environment variables in this file. So for example, we can’t have something like POLKADOT_CLI_ARGS="--name $(hostname)".

We also specify a user and group to run the service as - by default, the service will run as root. If this can be avoided, it should be. We specify a restart policy for the service - if the program crashes, it will always be restarted but only after a pause of 120 seconds. Finally, we add a restriction on the max memory the process can use - if it hits the MemoryHigh limit, any new memory is heavily reaped. If it hits the MemoryMax limit, the OOM killer will kill the process (thus triggering the aforementioned restart mechanism).

This is a reasonable service file - we have some logic surrounding when to start and restart the service, some basic privilege separation, and users can configure the command-line flags used without having to edit the service file and run systemctl daemon-reload every time. Now we need to make it secure!

Securing the service file

Systemd comes with a wide variety of configurable options that can be applied to services in order to improve the security - From basic things like disallowing the service from gaining superuser privileges, to providing a strict set of system calls the program is allowed to make. In order to find out what improvements are even available, we can use the excellent systemd-analyze tool.

Note: The service file must be in-place in order to run systemd-analyze against it. Copy it to /etc/systemd/system/ and run systemctl daemon-reload each time you make an edit.

% systemd-analyze security --no-pager polkadot.service
  NAME                                      DESCRIPTION                               EXPOSURE
✗ PrivateNetwork=                           Service has access to the host's network       0.5
✓ User=/DynamicUser=                        Service runs under a static non-root use…
✗ CapabilityBoundingSet=~CAP_SET(UID|GID|P… Service may change UID/GID identities/ca…      0.3
<SNIP>
✗ CapabilityBoundingSet=~CAP_WAKE_ALARM     Service may program timers that wake up …      0.1
✗ RestrictAddressFamilies=~AF_UNIX          Service may allocate local sockets             0.1

→ Overall exposure level for polkadot.service: 9.2 UNSAFE 😨

Running systemd-analyze generates a list of all the configuration options that can be edited in order to reduce the exposure of the service, and sums each unapplied option in order to generate a final exposure score. As can be seen above, we scored 9.2/10 (lower is better) with a ranking of ‘UNSAFE’. Yikes.

As an aside, I highly recommend running systemd-analyze security with no additional arguments. It will print an exposure summary for all running services. You might be surprised at just how many services are considered unsafe. I know I was.

The process of securing the service essentially consists of going through each line of systemd-analyze output and deciding whether that particular restriction can be implemented without hindering the operation of your program. For instance, the very first suggestion is PrivateNetwork=, which restricts whether or not the service can communicate on the host’s network. Since Polkadot is a network service, we don’t really want to stop it being able to communicate over the network.

More detailed descriptions of each suggestion can be found on the manpages for systemd.exec and systemd.resource-control.

Here is the final polkadot.service file after taking heed of systemd-analyze’s recommendations.

[Unit]
Description=Polkadot Node
After=network.target
Documentation=https://github.com/paritytech/polkadot

[Service]
EnvironmentFile=-/etc/default/polkadot
ExecStart=/usr/bin/polkadot $POLKADOT_CLI_ARGS
User=polkadot
Group=polkadot
Restart=always
RestartSec=120
MemoryHigh=5400M
MemoryMax=5500M
CapabilityBoundingSet=
LockPersonality=true
NoNewPrivileges=true
PrivateDevices=true
PrivateMounts=true
PrivateTmp=true
PrivateUsers=true
ProtectClock=true
ProtectControlGroups=true
ProtectHostname=true
ProtectKernelModules=true
ProtectKernelTunables=true
ProtectSystem=strict
RemoveIPC=true
RestrictAddressFamilies=AF_INET AF_INET6 AF_NETLINK AF_PACKET AF_UNIX
RestrictNamespaces=true
RestrictSUIDSGID=true
SystemCallArchitectures=native
UMask=0027
SystemCallFilter=@system-service
SystemCallFilter=~@clock @module @mount @reboot @swap @privileged

[Install]
WantedBy=multi-user.target

That’s quite a few new lines! Some of them may apply to a service you are writing, some may not. It is important to understand how adding restrictions to the execution environment of the service will impact the execution of your program. For example, if I had added a suggestion from systemd-analyze called MemoryDenyWriteExecute=true (which stops the program from creating memory mappings that are both executable and writable), Polkadot would run just fine with the argument --wasm-execution Interpreted but would crash when invoking Polkadot with --wasm-execution Compiled. It is important to understand how adding restrictions to the execution environment of the service will impact the execution of your program.

Running systemd-analyze on the updated service file gives a much better result:

% systemd-analyze security --no-pager polkadot.service | tail -n1
→ Overall exposure level for polkadot.service: 1.9 OK 🙂

Now that we have Polkadot running as a service and we’re a lot more confident with the security of the service, it’s time to package and distribute.