Soft2Soft Ops Practical knowledge base
systemd

Why a systemd Service Gets Stuck in Activating and How to Fix It

4 views
systemd Linux диагностика сервисов

The activating state means that systemd has started the unit activation procedure but has not yet received confirmation that startup completed successfully. The fix depends on the service type: the process must either continue running, finish the startup script, send a readiness notification, or create a valid PID file.

The commands below apply to system-level systemd services. Run them with permissions sufficient to read the journal and manage the unit. Replace example.service with the name of the affected service.

1. Record the Actual Service State

First, retrieve a concise status without immediately trying to restart the service:

systemctl status example.service --no-pager -l

Then query the properties that define the current startup phase:

systemctl show example.service \
  -p LoadState \
  -p ActiveState \
  -p SubState \
  -p Result \
  -p Type \
  -p MainPID \
  -p ControlPID \
  -p ExecMainCode \
  -p ExecMainStatus \
  -p TimeoutStartUSec

A stuck startup usually has ActiveState=activating and one of the intermediate values in SubState, such as start, start-pre, start-post, or auto-restart.

  • ControlPID identifies the process running a control command, such as ExecStartPre= or a startup script.
  • MainPID identifies the main process if systemd has already been able to determine it.
  • Result and ExecMainStatus help distinguish an unfinished startup from repeated crashes and restart loops.

Do not increase TimeoutStartSec= until you understand the cause. A large or infinite timeout hides a stuck script, an incorrect service type, or a missing readiness notification, but does not fix the problem.

2. Find the Command Where Startup Stopped

Review the journal for the current system boot:

journalctl -u example.service --boot --no-pager

To view recent messages with precise timestamps, use:

journalctl -u example.service --boot \
  --since "-10 minutes" \
  -o short-precise \
  --no-pager

If the journal contains no useful message, inspect all unit commands:

systemctl cat example.service

Pay particular attention to these directives:

  • ExecStartPre= — preparation commands;
  • ExecStart= — the main startup command;
  • ExecStartPost= — actions performed after startup;
  • Type= — the condition systemd uses to consider the service started;
  • PIDFile= — the PID file path for some forking services;
  • TimeoutStartSec= — the maximum startup duration;
  • After=, Requires=, and Wants= — dependencies and startup ordering.

If SubState=start-pre, one of the ExecStartPre= commands is stuck. If SubState=start-post, inspect ExecStartPost=. These commands must finish before activation can continue.

You can retrieve the PID of the currently running control command as follows:

systemctl show example.service -p ControlPID -p MainPID

Then inspect the process with standard Linux tools:

ps -fp PID

Replace PID with the actual number. Check whether the command is waiting for user input, an unavailable file, a socket, a network resource, a lock, or a child process to exit.

3. Verify That Type Matches the Program’s Behavior

Type=simple

With Type=simple, the process started by ExecStart= must remain in the foreground. Do not launch it through a shell with &, and do not enable internal daemonization if the program can run in the foreground.

Problematic configuration:

[Service]
Type=simple
ExecStart=/usr/local/bin/example --daemon

If --daemon causes the program to spawn a background process and terminate the original process, systemd may track the wrong process. For a program that supports foreground mode, the preferred configuration is:

[Service]
Type=simple
ExecStart=/usr/local/bin/example --foreground

The exact option name depends on the program. Do not add --foreground unless the application supports it; check its built-in help or official documentation.

Type=notify

With Type=notify, the service remains in activating until systemd receives a READY=1 notification through the sd_notify mechanism. If the application does not implement this protocol, the selected type is incorrect.

Check the settings:

systemctl show example.service -p Type -p NotifyAccess

There are two valid solutions:

  1. Configure the application to send READY=1 after it is actually ready.
  2. Change Type=notify to a type that matches the program’s real behavior, such as simple or exec.

Do not replace a readiness notification with an arbitrary delay using sleep: startup time may vary, and systemd still will not receive a reliable readiness signal.

Type=oneshot

For Type=oneshot, the ExecStart= command must finish. While it is still running, the unit remains in the activating state. This type is suitable for a finite operation such as creating a directory, applying configuration, running a migration, or performing one-time setup.

A common mistake is running a persistent server process as oneshot:

[Service]
Type=oneshot
ExecStart=/usr/local/bin/example-server

If example-server is supposed to run continuously, use an appropriate type, usually simple, exec, or notify if the application supports it.

RemainAfterExit=yes does not force a stuck command to finish. This directive only allows the unit to remain active after the command completes successfully.

Type=forking

With Type=forking, the startup process must spawn a background process and then exit. If the startup command does not exit or the PID file does not appear at the expected path, the service may remain in activating for a long time.

Check the configuration:

systemctl show example.service -p Type -p PIDFile -p MainPID

Then make sure the path in PIDFile= matches the application’s own configuration. The PID file must contain the PID of the actual running main process. Do not create the PID file manually or write the PID of a shell or startup script into it.

If the application supports foreground mode, it is usually more reliable to disable internal daemonization and use Type=simple, exec, or notify.

4. Check Dependencies and Pending Jobs

The service may be waiting for another unit. View active systemd jobs:

systemctl list-jobs

Show the dependencies of the affected service:

systemctl list-dependencies example.service

To analyze startup ordering, use:

systemd-analyze critical-chain example.service

This command helps identify the chain of units that affected startup timing. It does not prove that the last displayed unit is stuck; compare the output with the journal and current jobs.

Inspect dependent mount, socket, network, and device units separately:

systemctl status dependency.unit --no-pager -l
journalctl -u dependency.unit --boot --no-pager

Replace dependency.unit with the name of the specific dependency. Do not add After=network-online.target automatically: it is justified only when the service genuinely requires a configured network before startup and the system correctly implements waiting for network-online.

5. Check the Unit Syntax and Effective Configuration

A file from /usr/lib/systemd/system or /lib/systemd/system may be overridden by a drop-in file from /etc/systemd/system. Therefore, inspect the merged configuration rather than only the original package file:

systemctl cat example.service

For static validation, use:

systemd-analyze verify /etc/systemd/system/example.service

If the unit is supplied by a package and modified through a drop-in, create or edit the override with the standard command:

systemctl edit example.service

For example, a service type and timeout correction may look like this:

[Service]
Type=simple
TimeoutStartSec=30s

This is only an example of the structure. The Type value must match the program, and the timeout must match an acceptable startup duration.

After changing unit files, reload the manager configuration:

systemctl daemon-reload

6. Stop the Stuck Startup and Verify the Fix

First, try a normal stop:

systemctl stop example.service

After fixing the configuration, clear the recorded failure state and start the service:

systemctl reset-failed example.service
systemctl start example.service

Verify the result in three ways:

systemctl is-active example.service
systemctl status example.service --no-pager -l
journalctl -u example.service --boot -n 50 --no-pager

A continuously running service is expected to reach the active state. For a successfully completed oneshot service, the final state depends on RemainAfterExit=: without it, the unit may transition to inactive, which does not by itself indicate an error.

Practical Diagnostic Sequence

  1. Retrieve ActiveState, SubState, Type, MainPID, and ControlPID.
  2. Review the unit journal for the current boot.
  3. Determine whether ExecStartPre, ExecStart, or ExecStartPost is stuck.
  4. Compare Type= with the application’s actual behavior.
  5. For notify, verify that READY=1 is sent.
  6. For oneshot, confirm that the command is expected to finish.
  7. For forking, verify that the parent process exits and that PIDFile= is correct.
  8. Check pending jobs and dependencies.
  9. Inspect the effective configuration with systemctl cat and validate the syntax with systemd-analyze verify.
  10. Run daemon-reload, start the service, and confirm the result using the status and journal.

Version Limitations

The available properties, exact intermediate states, and individual systemd features vary between distribution releases. On the installed system, the local manual pages take precedence:

man systemd.service
man systemd.unit
man systemctl
man journalctl
man systemd-analyze
man sd_notify

The links below point to the official systemd documentation for the latest published version. It may differ from the systemd version provided by a specific distribution.

Sources