Soft2Soft Ops Practical knowledge base
systemd

How to Clear a systemd Start Limit After Repeated Service Failures

1 views
systemd start-limit-hit Linux

A systemd lockout after a series of failed starts can be cleared with systemctl reset-failed. For a specific service, run sudo systemctl reset-failed example.service, then start the unit again. However, fix the cause of the failure first: resetting the state does not repair the application configuration or prevent the start limit from being reached again.

How to tell whether the start limit was reached

Start by checking the service status:

systemctl status example.service

Replace example.service with the actual unit name. When the start rate limit is reached, systemd messages may contain the line Start request repeated too quickly. A failed state alone does not mean that the limit was triggered: the process may have exited because of a configuration error, a missing file, incorrect permissions, an unavailable dependency, or a nonzero exit code.

Check the service journal:

journalctl -b -u example.service

If you need messages from more than the current system boot, remove the -b option:

journalctl -u example.service

Look for the original process error first, then for systemd manager messages about restart attempts. The start rate limit is usually the result of several failed attempts rather than the primary cause of the failure.

How to clear the lockout

After fixing the original error, reset the failed state and the start-limit counters associated with the specified unit:

sudo systemctl reset-failed example.service

Then start the service again:

sudo systemctl start example.service

Check the result:

systemctl status example.service
journalctl -b -u example.service

systemctl reset-failed does not repair the service or modify unit files. The command resets the failed state and the counters associated with the start limit. If the process keeps crashing, systemd may stop attempting to start it again after the configured limit is reached.

Check Restart and the start rate limit

To view the main unit file and any attached drop-in fragments, use:

systemctl cat example.service

systemctl cat is useful for finding where a particular directive came from, but it does not calculate the final value of every property used by the systemd manager. After reviewing the files, separately check the significant values actually loaded by systemd:

systemctl show example.service \
  -p Restart \
  -p RestartUSec \
  -p StartLimitIntervalUSec \
  -p StartLimitBurst

This lets you compare the contents of the main unit file and drop-in files with the properties that the manager actually applied to the loaded unit.

For automatic restarts, directives from the Restart= family are commonly used in the [Service] section. For example:

[Service]
Restart=on-failure
RestartSec=5s

Restart=on-failure instructs systemd to restart the service after certain unsuccessful terminations. RestartSec= sets the delay before the next attempt. The exact behavior depends on how the process terminates and on the systemd version, so verify it against the local manual page for the installed system:

man systemd.service

In modern systemd versions, the start rate limit is configured with unit parameters:

[Unit]
StartLimitIntervalSec=60s
StartLimitBurst=5

The values 60s and 5 are examples only. Choose limits based on the application's behavior, recovery time, and acceptable restart frequency.

These mechanisms serve different purposes: Restart= determines whether the process should be restarted, while StartLimitIntervalSec= together with StartLimitBurst= limits how frequently starts are allowed. Frequent restarts can quickly exhaust the permitted number of attempts.

How to change the settings with a drop-in

Do not edit a package-provided unit file unless necessary. To create a local override, open a drop-in:

sudo systemctl edit example.service

For example, you can increase the delay before a restart:

[Service]
Restart=on-failure
RestartSec=10s

Or change the start limit parameters:

[Unit]
StartLimitIntervalSec=120s
StartLimitBurst=5

After saving the changes, reload the unit files:

sudo systemctl daemon-reload

Check the original unit and the attached overrides:

systemctl cat example.service

Then check the loaded values:

systemctl show example.service \
  -p Restart \
  -p RestartUSec \
  -p StartLimitIntervalUSec \
  -p StartLimitBurst

After changing the configuration, reset the already accumulated start-limit state and start the service:

sudo systemctl reset-failed example.service
sudo systemctl start example.service

Finish the check by reviewing the service status and journal:

systemctl status example.service
journalctl -b -u example.service

When the start limit can be disabled

In systemd versions that support StartLimitIntervalSec=, a zero interval disables start rate limiting:

[Unit]
StartLimitIntervalSec=0

This should not be used as a universal fix. If the service crashes immediately while automatic restart is enabled, the manager can continue attempting to start it without the protection previously provided by the limit. This can create unnecessary load and a large number of repetitive journal entries.

Disabling the limiter makes sense only when that behavior is explicitly required by the operational design. In most cases, it is safer to fix the cause of the process termination and choose appropriate values for RestartSec=, the interval, and the number of permitted attempts.

Why reset-failed does not solve the problem

The process exits immediately again

After reset-failed, the counter may be successfully reset, but the application may exit with the same error on the next start. Check the journal:

journalctl -b -u example.service

If the application keeps its own logs outside journald, check those as well. Do not increase the limits merely to hide a process error that keeps recurring.

The wrong unit is being reset

Check the exact name:

systemctl status example.service

A single system may contain similarly named services, template units, and individual template instances. Reset the state of the unit that is actually reaching the limit.

The setting comes from a drop-in

Looking only at the file in the vendor directory may hide a local override. Use:

systemctl cat example.service

The command shows the original unit file and the attached drop-in fragments. Then use systemctl show to check the values of the relevant properties that are actually loaded.

The service is activated again by another mechanism

A unit may be started by more than a manual command. Dependencies and other systemd mechanisms can also cause a new activation. If a start seems unexpected, examine the manager messages immediately before the start and the dependencies associated with the unit, rather than looking only at the process's most recent error.

Account for the systemd version

Before changing the limits, determine the installed version:

systemd --version

The names and availability of individual directives have changed across systemd releases. In particular, older systems may use the former StartLimitInterval= name instead of the modern StartLimitIntervalSec=. Do not copy configuration from a modern system to an older server without checking it first: determine supported parameter names and the valid location for those settings from the documentation for the installed version.

Check the local manual pages:

man systemd.unit
man systemd.service
man systemctl

You can also determine which properties the current manager exposes for a particular loaded unit:

systemctl show example.service

If properties or directives from the examples are absent from the documentation for your version, do not add them blindly. Use the names and syntax documented in the local manual for the installed systemd version.

Practical diagnostic sequence

  1. Check the status with systemctl status example.service.
  2. Review the journal with journalctl -b -u example.service.
  3. Find the original cause of the application termination, not just the message indicating that start attempts were exhausted.
  4. Determine the version with systemd --version.
  5. View the main unit and drop-in files with systemctl cat example.service.
  6. Check the significant properties that are actually loaded with systemctl show.
  7. If necessary, fix the application, permissions, dependencies, file paths, or application configuration.
  8. If the restart policy needs to be changed, create a local drop-in with systemctl edit.
  9. After changing the unit configuration, run systemctl daemon-reload.
  10. Reset the state with systemctl reset-failed example.service.
  11. Start the service and check the status and journal again.

How to verify the result

A successful systemctl start command is not sufficient by itself: the process may exit a few seconds later and enter a restart loop again. Check the status and recent messages:

systemctl status example.service
journalctl -b -u example.service

If you changed the start policy, also confirm that systemd actually loaded the intended parameters:

systemctl show example.service \
  -p Restart \
  -p RestartUSec \
  -p StartLimitIntervalUSec \
  -p StartLimitBurst

The problem can be considered resolved when the service remains operational for a period appropriate to its normal behavior, the original error does not recur, and systemd no longer stops start attempts because the limit has been exceeded.

Final checklist

  • Confirm from the status and journal that the start limit was actually reached.
  • Find the root cause of the application failure.
  • Check the installed systemd version.
  • Review the main unit and drop-in files with systemctl cat.
  • Check the applied properties with systemctl show.
  • Verify version-dependent directives against the local man systemd.unit and man systemd.service pages.
  • Do not use reset-failed as a substitute for fixing the cause of the failure.
  • After changing unit files, run systemctl daemon-reload.
  • After fixing the issue, run systemctl reset-failed example.service and start the service.
  • Check the status, journal, and actually loaded parameters again.
  • Do not disable the start rate limiter without a justified operational reason.