This is part of the Semicolon&Sons Code Diary - consisting of lessons learned on the job. You're in the unix category.

Last Updated: 2024-04-25

Big Idea

systemd starts up and supervises the entire system (hence the name...)

Concept of units

It is based around the notion of units - which have names and types. Unit configurations configuration is usually loaded directly from the file system (they correspond to file names).

Example: a unit avahi.service is read from a configuration file by the same name, and (should be) a unit encapsulating the Avahi daemon.

There are several kinds of units:

service: these are the most obvious kind of unit: daemons that can be started, stopped, restarted, reloaded
socket: this unit encapsulates a socket in the file-system or on the Internet. Each socket unit has a matching service unit, that is started if the first connection comes in on the socket or FIFO. Example: nscd.socket starts nscd.service on an incoming connection.
device: this unit encapsulates a device in the Linux device tree. If a device is marked for this via udev rules, it will be exposed as a device unit in systemd. Properties set with udev can be used as configuration source to set dependencies for device units.
mount: this unit encapsulates a mount point in the file system hierarchy. systemd monitors all mount points how they come and go, and can also be used to mount or unmount mount-points.
automount unit has a matching mount unit, which is started (i.e. mounted) as soon as the automount directory is accessed.
target: this unit type is used for logical grouping of units: instead of actually doing anything by itself, it simply references other units, which thereby can be controlled together. Examples for this are: bluetooth.target which is requested as soon as a bluetooth dongle becomes available and which simply pulls in bluetooth related services that otherwise would not need to be started: bluetoothd and obexd and suchlike.
snapshot: similar to target units snapshots do not actually do anything themselves and their only purpose is to reference other units. Snapshots can be used to save/rollback the state of all services and units of the init system. Primarily it has two intended use cases: to allow the user to temporarily enter a specific state such as "Emergency Shell", terminating current services, and provide an easy way to return to the state before, pulling up all services again that got temporarily pulled down.

Example unit file

[Unit]
Description=Apache Solr for Nextcloud's nextant app fulltext indexing

# systemd supports several kinds of dependencies between units. After/Before can be used to fix the ordering how units are activated.
After=syslog.target network.target remote-fs.target nss-lookup.target systemd-journald-dev-log.socket
Before=nginx.service

[Service]
Type=forking
User=solr
WorkingDirectory=/path/to/solr/server
# commands to start and stop
ExecStart=/path/to/solr/bin/solr start
ExecStop=/path/to/solr/bin/solr stop
Restart=on-failure
Environment=APP_ENV=production
Environment=DEBUG=false
PIDFile=/run/mydaemon/solr.pid

[Install]
WantedBy=multi-user.target

Explaining the unit file:

ExecStart - This is the command to run your program. Not that this should be one that forks by default. If it blocks in the foreground, use Type=simple
Environment - Extra env variables to add (1 per call to Environment - you can multiple entries
WantedBy - This is the dependencies handling mechanism in systemd. multi-user.target is the alternative for runlevel 3 in systemV world... this code will run after level 2 has been through. Often you see default.target which can be an alias for this. See exactly what it is with #systemctl get-default. Or set with systemctl set-default multi-user.target

Here's a table of the targets and their run levels:

# source: https://unix.stackexchange.com/questions/404667/systemd-service-what-is-multi-user-target/404671
Run Lvl Target Units                        Description
0       runlevel0.target, poweroff.target   Shut down and power off
1       runlevel1.target, rescue.target     Set up a rescue shell
2,3,4   runlevel[234].target,               Set up a non-gfx multi-user shell
        multi-user.target
5       runlevel5.target, graphical.target  Set up a graphical multi-user shell
6       runlevel6.target, reboot.target     Shut down and reboot the system

Run levels can be used, for example, to run some services only when a graphical system is there.

Relation to init.d?

It's the successor. Previously one put service scripts in /etc/init.d/ e.g. /etc/init.d/apache2 and called them e.g. /etc/init.d/apache2 status. A convenience for interacting with init.d was sudo service apache2 status. Init.d tended to use many shell scripts. However these are slow and brittle, therefore C programs are preferred in systemd.

How it's faster than predecessors?

Speed Increase 1: Starts less at boot

e.g. It won't start bluetoothd until a dongle is connected
e.g. Doesn't run printing daemon like CUPS until the user requests to print something

Speed Increase 2: Starts more in parallel

Previous init achieved inter-process-communication using sockets from p1 to p2. The issue was that p1 had to full boot before its socket was ready to pass into p2. This caused massive wasteful serialization.

The solution is clever: create the listening sockets before we actually start the daemons, and then just pass the socket during exec() to it. That way, we can create all sockets for all daemons in one step in the init system, and then in a second step run all daemons at once.

e.g. If you start syslog and various syslog clients at the same time, what will happen in the scheme pointed out above is that the messages of the clients will be added to the /dev/log socket buffer. As long as that buffer doesn't run full, the clients will not have to wait in any way and can immediately proceed with their start-up. As soon as syslog itself finished start-up, it will dequeue all messages and process them.

Other pluses:

Dependency management difficulties nearly cease to exist because dependencies between services will no longer necessarily have to be configured to allow proper parallelized start-up: if we start all sockets at once and a service needs another it can be sure that it can connect to its socket.
If a service dies, its listening socket stays around, not losing a single message. After a restart of the crashed service it can continue right where it left off.
If a service is upgraded we can restart the service while keeping around its sockets, thus ensuring the service is continuously responsive. Not a single connection is lost during the upgrade.

How to use?

See systemctl doc.

Vs. the service command?

service is a wrapper for either init.d or systemctl depending on the version.

How to install on a server?

Put a x.service file in /etc/systemd/system

If it needs a PID/socket file, create a dir related to that daemon under /run/mydaemon. Then chown of that sub-dir (NEVER /run itself though) to the user that will run your daemon

Reload with sudo systemctl daemon-reload

How to remove a service?

Remove the x.service file in /etc/systemd/system

Semicolon & Sons