systemctl — Why Your Service Won't Start After Reboot and How to Fix It

The Problem

A service is down in production. You need to check its status, read its logs, restart it, and make sure it comes back after a reboot. Or you have deployed a new application and need to create a systemd service so it starts automatically, restarts on failure, and logs to journald properly.

systemctl is the interface to systemd — the init system that manages services on modern Linux. This guide covers every operation you need for day-to-day service management.

Essential Service Operations

Start, stop, restart

# Start a service
sudo systemctl start nginx

# Stop a service
sudo systemctl stop nginx

# Restart a service (stop then start)
sudo systemctl restart nginx

# Reload configuration without stopping (not all services support this)
sudo systemctl reload nginx

# Restart only if already running
sudo systemctl try-restart nginx

# Reload if supported, otherwise restart
sudo systemctl reload-or-restart nginx

Check service status

# Full status with recent log lines
sudo systemctl status nginx

# Just check if active (for scripts)
systemctl is-active nginx

# Check if service is enabled (starts on boot)
systemctl is-enabled nginx

# Check if service failed
systemctl is-failed nginx

The status output tells you everything:

nginx.service - A high performance web server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2024-01-15 10:23:45 UTC; 3 days ago
    Process: 1234 ExecStartPre=/usr/sbin/nginx -t (code=exited, status=0/SUCCESS)
   Main PID: 1235 (nginx)
      Tasks: 5 (limit: 4915)
     Memory: 12.4M
        CPU: 1.234s
     CGroup: /system.slice/nginx.service

Enable and disable (boot behavior)

# Start on boot
sudo systemctl enable nginx

# Start on boot AND start immediately
sudo systemctl enable --now nginx

# Do not start on boot
sudo systemctl disable nginx

# Disable AND stop immediately
sudo systemctl disable --now nginx

# Prevent a service from being started (even manually)
sudo systemctl mask nginx

# Undo masking
sudo systemctl unmask nginx

Viewing Logs with journalctl

systemd captures all service output (stdout/stderr) in the journal:

# View all logs for a service
journalctl -u nginx

# View logs since last boot
journalctl -u nginx -b

# Follow logs in real-time (like tail -f)
journalctl -u nginx -f

# Last 50 lines
journalctl -u nginx -n 50

# Logs since a specific time
journalctl -u nginx --since "2024-01-15 10:00:00"
journalctl -u nginx --since "1 hour ago"
journalctl -u nginx --since today

# Logs between two times
journalctl -u nginx --since "2024-01-15 10:00" --until "2024-01-15 11:00"

# Only error-level and above
journalctl -u nginx -p err

# Output as JSON (for log aggregation)
journalctl -u nginx -o json --no-pager

Log priority levels

journalctl -u myapp -p emerg    # 0 - System is unusable
journalctl -u myapp -p alert    # 1 - Immediate action needed
journalctl -u myapp -p crit     # 2 - Critical conditions
journalctl -u myapp -p err      # 3 - Error conditions
journalctl -u myapp -p warning  # 4 - Warning conditions
journalctl -u myapp -p notice   # 5 - Normal but significant
journalctl -u myapp -p info     # 6 - Informational
journalctl -u myapp -p debug    # 7 - Debug-level messages

Journal disk usage

# Check journal size on disk
journalctl --disk-usage

# Vacuum old logs (keep only last 7 days)
sudo journalctl --vacuum-time=7d

# Vacuum by size (keep only 500MB)
sudo journalctl --vacuum-size=500M

Listing and Finding Services

# List all active services
systemctl list-units --type=service --state=active

# List all services (including inactive)
systemctl list-units --type=service --all

# List failed services
systemctl list-units --type=service --state=failed

# List enabled services (will start on boot)
systemctl list-unit-files --type=service --state=enabled

# Search for a service by keyword
systemctl list-units --type=service | grep docker

# Show all dependencies of a service
systemctl list-dependencies nginx

Creating Custom Service Files

Basic application service

# /etc/systemd/system/myapp.service
[Unit]
Description=My Application Server
Documentation=https://docs.myapp.com
After=network.target postgresql.service
Wants=postgresql.service

[Service]
Type=simple
User=appuser
Group=appuser
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/bin/server --port 8080
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5
StartLimitBurst=5
StartLimitIntervalSec=60

# Security hardening
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/opt/myapp/data /var/log/myapp
PrivateTmp=true

# Environment
Environment=NODE_ENV=production
EnvironmentFile=/opt/myapp/.env

# Logging
StandardOutput=journal
StandardError=journal
SyslogIdentifier=myapp

[Install]
WantedBy=multi-user.target

Activate the new service

# Reload systemd to pick up new/changed unit files
sudo systemctl daemon-reload

# Enable and start
sudo systemctl enable --now myapp

# Verify
sudo systemctl status myapp
journalctl -u myapp -f

Service with pre/post commands

[Service]
ExecStartPre=/opt/myapp/bin/migrate --check
ExecStart=/opt/myapp/bin/server
ExecStartPost=/usr/bin/curl -s http://localhost:8080/health
ExecStop=/opt/myapp/bin/server --graceful-shutdown
ExecStopPost=/opt/myapp/bin/cleanup
TimeoutStopSec=30

Timer-based service (replaces cron)

# /etc/systemd/system/backup.service
[Unit]
Description=Database Backup

[Service]
Type=oneshot
ExecStart=/opt/scripts/backup.sh
User=backup

# /etc/systemd/system/backup.timer
[Unit]
Description=Run database backup every 6 hours

[Timer]
OnCalendar=--* 00/6:00:00
Persistent=true
RandomizedDelaySec=300

[Install]
WantedBy=timers.target

sudo systemctl enable --now backup.timer
systemctl list-timers --all

Debugging Failed Services

# Check status (shows exit code and recent logs)
sudo systemctl status myapp

# View detailed failure information
journalctl -u myapp --since "5 min ago" --no-pager

# Check if the binary exists and is executable
ls -la /opt/myapp/bin/server

# Check if the user/group exists
id appuser

# Check port conflicts
ss -tlnp | grep 8080

# Run the ExecStart command manually as the service user
sudo -u appuser /opt/myapp/bin/server --port 8080

# Check SELinux denials (RHEL/CentOS)
ausearch -m avc --start recent

# Check resource limits
systemctl show myapp | grep Limit

Common failure patterns

# Exit code 217 — User not found
# Fix: Create the user specified in the unit file
sudo useradd -r -s /sbin/nologin appuser

# Exit code 203 — Exec format error
# Fix: Check the shebang line or binary architecture
file /opt/myapp/bin/server

# Exit code 200 — Namespace setup failed
# Fix: Remove security directives one by one to find the issue

# Status=226/NAMESPACE — Directory not accessible
# Fix: Check ReadWritePaths and WorkingDirectory permissions

Service Dependencies and Ordering

[Unit]
# Start after these services are started
After=network.target postgresql.service redis.service

# Require these services (fail if they fail)
Requires=postgresql.service

# Want these services (do not fail if they fail)
Wants=redis.service

# Start before these services
Before=nginx.service

# View dependency tree
systemctl list-dependencies myapp

# View reverse dependencies (what depends on this service)
systemctl list-dependencies myapp --reverse

Common Mistakes

Forgetting daemon-reload after editing unit files — systemd caches unit files. Without reloading, your changes are invisible.

Using Type=simple when the process forks — If your application daemonizes (forks to background), use Type=forking. Otherwise systemd thinks the service died immediately.

Hardcoding paths without checking the user — A service running as appuser cannot write to /var/log/ unless you explicitly allow it.

Not setting Restart=on-failure — Without this, a crashed service stays down until someone manually restarts it.

Ignoring StartLimitBurst — Without start limits, a service that crashes on startup will restart in an infinite loop, flooding your logs.

Using kill -9 instead of systemctl stop — Bypassing systemd means it does not know the service stopped, leading to confused state. Always use systemctl.

Quick Reference

Summary

systemctl is the single command for all service lifecycle operations — start, stop, enable, disable, and status. Pair it with journalctl for log access. When deploying applications, create proper unit files with restart policies, security hardening, and explicit dependencies. Always run daemon-reload after editing unit files, and use journalctl -u service -f as your primary debugging tool when services misbehave.

---

Frequently Asked Questions

How do I start, stop, and restart a service with systemctl?

Use systemctl start <service> to start, systemctl stop <service> to stop, and systemctl restart <service> to restart. Use systemctl reload <service> for services that support configuration reload without restart (like nginx). Check status with systemctl status <service> which shows running state, recent logs, and PID.

How do I make a service start automatically on boot?

Use systemctl enable <service> to configure a service to start at boot, and systemctl disable <service> to prevent it. Note that enable doesn't start the service immediately — use systemctl enable --now <service> to enable and start in one command. Check if a service is enabled with systemctl is-enabled <service>.

How do I create a custom systemd service?

Create a unit file at /etc/systemd/system/myapp.service with [Unit], [Service], and [Install] sections. Define ExecStart with the full path to your application, set User for the run-as user, add Restart=on-failure for automatic restart, and set WantedBy=multi-user.target. Then run systemctl daemon-reload to load the new service and systemctl enable --now myapp.

Why is my systemd service failing to start?

Check detailed status with systemctl status <service> and full logs with journalctl -u <service> -n 50. Common causes: wrong file path in ExecStart, permission issues (wrong user or file permissions), missing dependencies or environment variables, or the previous instance still holding a port. Use systemctl cat <service> to verify the unit file configuration.

What is the difference between systemctl and service commands?

systemctl is the modern command for systemd-based systems (CentOS 7+, Ubuntu 16.04+) offering full service lifecycle management, dependency control, and resource limits. The service command is a legacy SysVinit compatibility wrapper that still works but lacks advanced features. Always use systemctl on modern systems — service just calls systemctl internally anyway.

---

Related Resources

• Linux Commands Reference — 50+ Linux commands with production examples
• DevOps Interview Academy — Linux interview questions
• Linux Process Management Guide — Understanding processes behind services
• Cron Job Examples for Linux — Scheduling tasks alongside services
• Linux Performance Troubleshooting — Debugging unresponsive services