Linux Troubleshooting Interview — 20 'Debug This Live Server' Problems They Actually Ask

If you've been through DevOps interviews, you know the drill — they don't just ask "what does top do?" They throw a scenario at you and watch how you think through it. The difference between a junior and senior candidate isn't knowing the commands — it's knowing where to look first and why.

This guide covers 20 real-world Linux troubleshooting scenarios that come up repeatedly in DevOps interviews. For each one, we'll walk through the investigation like you're sitting in the hot seat.

---

Category 1: Disk & Storage

Scenario 1: Server Disk 100% Full But du Shows Only 20GB Used

Scenario: "Your monitoring alerts that a production server's root filesystem is 100% full. You SSH in and run du -sh / which reports only 20GB used, but df -h shows the 50GB disk is completely full. What's going on?"

Your Approach:

This is a classic deleted-file-still-held-open situation. When a process has a file open and you delete it, the filesystem doesn't reclaim the space until the process releases the file handle.

Commands Used:

# First, confirm the discrepancy
df -h /
# Output: /dev/sda1  50G   50G     0 100% /

du -sh /
# Output: 20G    /

# Find deleted files still held open by processes
lsof +L1
# Output:
# COMMAND   PID   USER   FD   TYPE DEVICE    SIZE/OFF NLINKS NODE NAME
# java    12345  app    1w   REG  8,1   32212254720  0  1234 /var/log/app.log (deleted)
# nginx    6789  www    5w   REG  8,1    5368709120  0  5678 /var/log/nginx/access.log (deleted)

Root Cause: A log rotation script deleted large log files, but the Java application and Nginx still had file handles open to the deleted files. The space (~30GB) is still allocated because the processes haven't released their file descriptors.

Fix:

# Option 1: Restart the processes (if you can afford downtime)
systemctl restart app-service
systemctl restart nginx

# Option 2: Truncate the file descriptor without restarting (zero-downtime)
# Find the fd number from lsof output, then:
echo "" > /proc/12345/fd/1

# Option 3: For future prevention, use copytruncate in logrotate
# /etc/logrotate.d/app
# /var/log/app.log {
#     daily
#     copytruncate
#     rotate 7
# }

---

Scenario 2: Can't Delete Files, Disk Still Full After Removing Large Files

Scenario: "You deleted a 10GB log file to free up space, confirmed it's gone with ls, but df still shows the same usage. What happened?"

Your Approach:

First thing I'd check — is the file truly gone, or just unlinked while a process holds it open? This overlaps with Scenario 1, but there's another sneaky cause: reserved blocks.

Commands Used:

# Check if any process still holds the deleted file
lsof | grep deleted

# If nothing shows up, check reserved blocks for root
tune2fs -l /dev/sda1 | grep -i reserved
# Output:
# Reserved block count:     1310720
# Reserved blocks uid:      0 (user root)

# That's 5% of your disk reserved for root — on a 100GB disk, that's 5GB
# you can't use as a normal user

# Reduce reserved blocks (careful in production!)
tune2fs -m 1 /dev/sda1

Root Cause: Two possibilities — either a process still holds the file open (check with lsof), or the ext4 filesystem has 5% reserved blocks for root that df counts as used space for non-root users.

Fix:

# If it's a held-open file:
# Find and restart the process, or truncate via /proc/PID/fd/

# If it's reserved blocks eating space:
# Reduce from 5% to 1% (safe for non-root filesystems)
sudo tune2fs -m 1 /dev/sda1

# Verify
df -h /

---

Scenario 3: Filesystem Suddenly Becomes Read-Only

Scenario: "Users report they can't write to the application data directory. You check and find the entire filesystem is mounted read-only. The server didn't reboot. What's your troubleshooting approach?"

Your Approach:

A filesystem going read-only mid-operation almost always means the kernel detected a hardware or filesystem error and remounted it read-only to prevent data corruption.

Commands Used:

# Confirm the read-only state
mount | grep sda1
# Output: /dev/sda1 on / type ext4 (ro,relatime,errors=remount-ro)

# Check kernel messages for the reason
dmesg | tail -50
# Output:
# [452632.123] EXT4-fs error (device sda1): ext4_journal_check_start:56: Detected aborted journal
# [452632.124] EXT4-fs (sda1): Remounting filesystem read-only

# Check disk health
smartctl -a /dev/sda
# Look for: Reallocated_Sector_Ct, Current_Pending_Sector, Offline_Uncorrectable

# Check filesystem logs
journalctl -k | grep -i "error\|EXT4\|I/O"

Root Cause: The disk has bad sectors or the RAID controller reported I/O errors. The kernel's errors=remount-ro mount option kicked in to protect data integrity.

Fix:

# Emergency: remount read-write (temporary, risky if hardware is failing)
mount -o remount,rw /

# Proper fix: schedule downtime and run fsck
# 1. Unmount the filesystem (may need single-user mode or boot from rescue)
umount /dev/sda1
fsck -y /dev/sda1

# 2. If SMART shows disk is dying, replace the disk
# 3. Check RAID status if applicable
cat /proc/mdstat
mdadm --detail /dev/md0

---

Scenario 4: /tmp Filling Up Every Few Hours

Scenario: "The /tmp directory keeps filling up every 3-4 hours, causing application failures. You clean it manually but it comes back. How do you find the culprit?"

Your Approach:

I'd set up monitoring to catch whatever is writing to /tmp in real-time, then trace it back to the responsible process.

Commands Used:

# See what's currently eating space in /tmp
du -sh /tmp/* | sort -rh | head -20
# Output:
# 4.2G  /tmp/core.12345
# 2.1G  /tmp/sess_abc123...
# 1.8G  /tmp/hsperfdata_app

# Watch for file creation in real-time
inotifywait -m -r /tmp -e create -e modify
# Output:
# /tmp/ CREATE core.67890
# /tmp/upload_ CREATE tmp_file_xyz

# Find which process is writing the most to /tmp
lsof +D /tmp | awk '{print $1, $2}' | sort | uniq -c | sort -rn
# Output:
# 45 java 12345
# 12 php-fpm 6789

# Check if core dumps are the issue
ulimit -c
# Output: unlimited  (that's the problem!)

# Check systemd tmp cleaner
systemctl status systemd-tmpfiles-clean.timer

Root Cause: The Java application is crashing and generating core dumps in /tmp (each one is 4GB+ for a JVM with large heap). Combined with PHP session files that aren't being cleaned up.

Fix:

# Disable or limit core dumps
echo "* hard core 0" >> /etc/security/limits.conf
# Or redirect them elsewhere:
echo "/var/crash/core.%e.%p" > /proc/sys/kernel/core_pattern

# Clean up PHP sessions (configure proper gc in php.ini)
# session.gc_maxlifetime = 1440
# session.gc_probability = 1

# Set up systemd-tmpfiles for automatic cleanup
cat > /etc/tmpfiles.d/tmp-cleanup.conf << EOF
d /tmp 1777 root root 24h
EOF

# Activate cleanup timer
systemctl enable --now systemd-tmpfiles-clean.timer

---

Category 2: CPU & Memory

Scenario 5: Load Average Is 50 on a 4-Core System

Scenario: "You get paged because a server's load average is 50. It's a 4-core machine. CPU usage in top shows only 20% used. What's causing the high load?"

Your Approach:

Load average isn't just CPU — it includes processes in uninterruptible sleep (D state), which usually means I/O wait. A load of 50 on 4 cores with low CPU usage screams I/O bottleneck.

Commands Used:

# Check load and CPU breakdown
uptime
# Output: load average: 50.23, 48.67, 45.12

top -bn1 | head -5
# Output:
# %Cpu(s):  5.2 us,  2.1 sy,  0.0 ni,  12.5 id, 79.8 wa,  0.0 hi,  0.4 si
#                                                   ^^^^^ 
#                                          79.8% I/O wait!

# Find processes in D (uninterruptible sleep) state
ps aux | awk '$8 ~ /D/ {print}'
# Output:
# app  12345 0.0  1.2 ... D  ... /usr/bin/java
# app  12346 0.0  1.2 ... D  ... /usr/bin/java
# (many processes stuck in I/O)

# Check I/O statistics
iostat -x 1 3
# Output:
# Device   r/s   w/s  await  svctm  %util
# sda    450.0  20.0  856.0   2.2  100.0

# Check which processes are doing the I/O
iotop -b -n 1

Root Cause: The disk is saturated at 100% utilization with very high await times (856ms). Dozens of Java threads are stuck waiting for I/O, each contributing to the load average. The disk (likely a single spinning HDD) can't handle the IOPS.

Fix:

# Immediate: identify the I/O-heavy process
iotop -aoP | head -10

# Check if it's a runaway backup, log rotation, or actual app traffic
# If it's a backup process:
ionice -c3 -p $(pgrep backup)  # Set to idle I/O priority
nice -n 19 ionice -c3 /usr/local/bin/backup.sh  # For future runs

# Long-term: upgrade to SSD or add more IOPS (EBS gp3 on AWS)
# Consider adding a caching layer (Redis) to reduce disk reads

---

Scenario 6: OOM Killer Keeps Killing Your Application

Scenario: "Your Java application keeps getting killed by the OOM killer. The server has 16GB RAM and you've set the JVM heap to 12GB. Why is it dying?"

Your Approach:

The OOM killer fires when the system runs out of memory. If you gave Java 12GB on a 16GB box, that leaves only 4GB for the OS, page cache, other processes, and JVM's own non-heap memory (metaspace, thread stacks, native memory).

Commands Used:

# Confirm OOM kills in logs
dmesg | grep -i "oom\|killed"
# Output:
# [34521.123] java invoked oom-killer: gfp_mask=0x24200ca(GFP_HIGHUSER_MOVABLE)
# [34521.456] Out of memory: Kill process 12345 (java) score 820

# Check what's eating memory
free -h
# Output:
#               total   used   free   shared  buff/cache  available
# Mem:          15Gi    14.8Gi  200Mi   48Mi    312Mi       400Mi

# See the full memory breakdown of your Java process
cat /proc/12345/status | grep -i vm
# VmPeak:  14523456 kB
# VmRSS:   13891234 kB  <-- actual physical memory used

# Check JVM native memory (if enabled)
jcmd 12345 VM.native_memory summary

# See who else is using memory
ps aux --sort=-%mem | head -10

# Check OOM score for your process
cat /proc/12345/oom_score_adj

Root Cause: JVM uses more than just heap. Thread stacks (1MB x 500 threads = 500MB), metaspace (~256MB), JIT code cache (~240MB), native allocations, and GC overhead push total JVM memory to ~14GB. Combined with OS needs, it exceeds 16GB.

Fix:

# Reduce JVM heap to leave room for non-heap memory
# Rule of thumb: heap should be max 60-70% of total RAM
# On 16GB: set heap to 10GB max
java -Xmx10g -Xms10g -XX:MaxMetaspaceSize=256m -Xss512k ...

# Protect critical processes from OOM killer
echo -1000 > /proc/$(pgrep java)/oom_score_adj

# Or better — add swap as a safety net (not for performance)
fallocate -l 4G /swapfile
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile
echo '/swapfile none swap sw 0 0' >> /etc/fstab

# Set vm.overcommit_memory for more predictable behavior
echo "vm.overcommit_memory=2" >> /etc/sysctl.conf
sysctl -p

---

Scenario 7: System Is Slow But CPU Shows Idle

Scenario: "Users complain the system is extremely slow. You check top and CPU is 95% idle. Memory looks fine too. What else could it be?"

Your Approach:

If CPU and memory are fine but the system feels slow, I'd look at: I/O wait (hidden in the idle breakdown), network saturation, DNS issues, or NFS/filesystem hangs.

Commands Used:

# Look at the FULL CPU breakdown, not just user/idle
top -bn1 | grep Cpu
# %Cpu(s): 2.0 us, 1.0 sy, 0.0 ni, 45.0 id, 50.0 wa, 0.0 hi, 2.0 si
#                                               ^^^^^^^^
# 50% I/O wait! That's not really "idle"

# If wa% is low too, check for NFS hangs
mount | grep nfs
# /nfs-server:/export on /data type nfs4 (rw,hard)

# Check if NFS mount is hanging
ls /data  # This command hangs — that's your answer

# Check network saturation
sar -n DEV 1 5
# Or
nload eth0

# Check for high interrupt rate or context switches
vmstat 1 5
# Output:
#  r  b   swpd   free   si   so    bi    bo   in   cs
#  1  8      0  14000    0    0     5  2000  200  50000
#                                                 ^^^^^ very high context switches

# Check disk latency
ioping /dev/sda1
# Output: avg latency 450ms (should be <10ms for SSD, <20ms for HDD)

Root Cause: An NFS mount with hard option is hanging because the NFS server is unreachable. Every process trying to access /data gets stuck, making the system feel frozen even though local CPU is idle.

Fix:

# Immediate: check NFS server connectivity
ping nfs-server
showmount -e nfs-server

# If NFS server is down, remount with soft/timeo options to prevent hangs
umount -l /data  # Lazy unmount (won't block)
mount -o soft,timeo=5,retrans=3 nfs-server:/export /data

# Or add intr option so processes can be interrupted
mount -o hard,intr nfs-server:/export /data

# Long-term: use autofs for on-demand mounting
# Or switch to a more resilient shared storage solution

---

Scenario 8: Memory Usage Keeps Growing (Memory Leak Detection)

Scenario: "Your application's memory usage grows by ~500MB per day. After a week, it gets OOM killed. How do you confirm it's a memory leak and identify the cause?"

Your Approach:

I'd first confirm the growth pattern with historical data, then narrow down whether it's the application itself or something in the system.

Commands Used:

# Track memory growth over time (run via cron every 5 min)
while true; do
  echo "$(date): $(ps -o rss= -p $(pgrep myapp)) KB" >> /tmp/mem_track.log
  sleep 300
done

# Check current memory maps for the process
pmap -x $(pgrep myapp) | tail -5
# Output:
# total kB         8234567  6123456  5987654

# Compare after an hour
pmap -x $(pgrep myapp) | tail -5
# Output:
# total kB         8734567  6623456  6487654  (+500MB!)

# For Java apps, get heap dump
jmap -dump:live,format=b,file=/tmp/heapdump.hprof $(pgrep java)

# For native/C apps, use valgrind
valgrind --leak-check=full --log-file=/tmp/valgrind.log ./myapp

# Check if it's a file descriptor leak (each fd uses kernel memory)
ls /proc/$(pgrep myapp)/fd | wc -l
# Run again in an hour and compare

# Use smaps for detailed memory breakdown
cat /proc/$(pgrep myapp)/smaps_rollup

Root Cause: The application has a connection pool that creates new connections but never releases them. Each connection holds buffers in memory. After a week, thousands of stale connections accumulate.

Fix:

# Immediate: restart the service (buys time)
systemctl restart myapp

# Set up monitoring to catch it early
# Add to crontab:
# /5    * RSS=$(ps -o rss= -p $(pgrep myapp)); [ "$RSS" -gt 8000000 ] && systemctl restart myapp

# Long-term: fix the application code
# - Set connection pool max size and idle timeout
# - Enable connection validation/eviction
# - Analyze heap dump with Eclipse MAT or VisualVM

---

Category 3: Networking

Scenario 9: Can Ping Server But Can't SSH

Scenario: "You can ping a server successfully, but SSH connections time out. Other team members can SSH to the same server. What do you check?"

Your Approach:

Ping works (ICMP) but SSH doesn't (TCP port 22) — this tells me it's not a total network outage. It's either a firewall rule, SSH daemon issue, or connection limit problem.

Commands Used:

# Test if port 22 is reachable
telnet server-ip 22
# Or better:
nc -zv server-ip 22
# Output: Connection timed out (firewall) OR Connection refused (service down)

# Check if it's just you — try from a different source
ssh -vvv user@server-ip
# The -vvv verbose output shows where it gets stuck:
# debug1: Connecting to server-ip port 22.
# debug1: Connection established.  <-- if you see this, it's auth related
# OR it hangs at "Connecting..." — network/firewall issue

# Check if there's a host-based firewall blocking you
# (If you have console/out-of-band access to the server)
iptables -L -n | grep 22
# Output:
# ACCEPT tcp -- 10.0.0.0/24  0.0.0.0/0  tcp dpt:22
# DROP   tcp -- 0.0.0.0/0    0.0.0.0/0  tcp dpt:22

# Check SSH daemon max connections
ss -tnp | grep :22 | wc -l
# Output: 50 (maybe MaxSessions or MaxStartups limit reached)

# Check if your IP is in /etc/hosts.deny or fail2ban
fail2ban-client status sshd
# Output:
# Banned IP list: 203.0.113.50 (that's your IP!)

# Check TCP wrappers
cat /etc/hosts.allow
cat /etc/hosts.deny

Root Cause: Fail2ban banned your IP address after a colleague accidentally triggered multiple failed login attempts from your shared jump host. The SSH daemon is fine — you're just firewalled.

Fix:

# Unban your IP
fail2ban-client set sshd unbanip 203.0.113.50

# Whitelist your office/VPN IP range
echo "ignoreip = 10.0.0.0/24 203.0.113.0/28" >> /etc/fail2ban/jail.local
systemctl restart fail2ban

# Alternative: if it's iptables, flush the rule
iptables -D INPUT -s 203.0.113.50 -j DROP

---

Scenario 10: DNS Resolution Intermittently Fails

Scenario: "Applications intermittently get 'Name resolution failed' errors. It works 80% of the time but fails randomly. How do you troubleshoot?"

Your Approach:

Intermittent DNS is tricky. I'd look at: multiple DNS servers with one failing, UDP packet loss, connection tracking table overflow, or systemd-resolved caching issues.

Commands Used:

# Check which DNS servers are configured
cat /etc/resolv.conf
# Output:
# nameserver 10.0.0.2
# nameserver 10.0.0.3

# Test each DNS server individually
dig @10.0.0.2 google.com +short +time=2
# Output: 142.250.80.46 (works!)

dig @10.0.0.3 google.com +short +time=2
# Output: ;; connection timed out (this one is dead!)

# Check for UDP packet loss (DNS uses UDP)
nstat -az | grep -i udp
# Look for: UdpInErrors, UdpRcvbufErrors

# Check conntrack table (very common in containerized environments)
cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max
# If count is near max, DNS packets get dropped!

# Check systemd-resolved status
resolvectl status
systemd-resolve --statistics

# Monitor DNS failures in real-time
tcpdump -i eth0 port 53 -nn
# Watch for queries with no responses

Root Cause: The secondary DNS server (10.0.0.3) is unreachable. Linux rotates between nameservers, so ~50% of queries go to the dead server and time out before falling back to the primary.

Fix:

# Immediate: remove dead DNS server
sed -i '/10.0.0.3/d' /etc/resolv.conf

# Add options to speed up failover
echo "options timeout:1 attempts:2 rotate" >> /etc/resolv.conf

# Better long-term: use a local DNS cache
apt install dnsmasq
echo "server=10.0.0.2" >> /etc/dnsmasq.conf
echo "server=8.8.8.8" >> /etc/dnsmasq.conf  # Fallback
systemctl enable --now dnsmasq

# Update resolv.conf to use local cache
echo "nameserver 127.0.0.1" > /etc/resolv.conf

# If conntrack was the issue:
echo "net.netfilter.nf_conntrack_max=262144" >> /etc/sysctl.conf
sysctl -p

---

Scenario 11: Connection Refused vs Connection Timed Out

Scenario: "An interviewer asks: 'What's the difference between connection refused and connection timed out? How would you troubleshoot each?'"

Your Approach:

These two errors tell very different stories about what's happening on the network:

• Connection refused = the packet reached the server, but nothing is listening on that port (RST packet sent back)
• Connection timed out = the packet never reached the server OR was silently dropped (no response at all)

Commands Used:

# Simulate and observe both:

# CONNECTION REFUSED — server is up, port is closed
nc -zv server-ip 8080
# Output: Connection refused
# What's happening: TCP SYN → TCP RST (immediate response)

# Troubleshoot connection refused:
# 1. Service isn't running
systemctl status myapp
# 2. Service is listening on wrong interface
ss -tlnp | grep 8080
# Output: 127.0.0.1:8080 (only localhost! not 0.0.0.0)
# 3. Service is on a different port
ss -tlnp | grep myapp

# CONNECTION TIMED OUT — packets being dropped
nc -zv server-ip 8080 -w 5
# Output: Connection timed out (after 5 seconds of waiting)
# What's happening: TCP SYN → ... (no response)

# Troubleshoot connection timed out:
# 1. Firewall dropping packets
iptables -L -n | grep 8080
# 2. Security group (cloud) blocking the traffic
# 3. Routing issue — packets going to wrong place
traceroute -p 8080 server-ip
# 4. Server is completely down
ping server-ip

Root Cause:

• Connection refused typically means: fix the service (restart it, change bind address from 127.0.0.1 to 0.0.0.0)
• Connection timed out typically means: fix the network path (open firewall rule, fix security group, repair routing)

Fix:

# For "connection refused" — service binding to localhost only:
# Edit service config to listen on all interfaces:
# Before: bind_address = 127.0.0.1
# After:  bind_address = 0.0.0.0
systemctl restart myapp

# For "connection timed out" — firewall blocking:
iptables -A INPUT -p tcp --dport 8080 -j ACCEPT
# Or on AWS: add inbound rule to security group

---

Scenario 12: High Network Latency Between Two Servers

Scenario: "Two servers in the same datacenter have 50ms latency between them. It should be <1ms. Application performance has degraded significantly. What's your approach?"

Your Approach:

50ms in the same datacenter is absurd — it should be sub-millisecond. This is either a routing issue (traffic going through an unexpected path), a saturated network link, or CPU pressure causing packet processing delays.

Commands Used:

# Confirm the latency
ping -c 10 server-b
# Output: rtt min/avg/max = 48.2/51.3/55.7 ms

# Check the route — is traffic going through an unexpected path?
traceroute server-b
# Output:
# 1  gateway (10.0.0.1)    0.5 ms
# 2  core-switch (10.0.1.1)  1.2 ms
# 3  wan-router (203.0.113.1)  25.0 ms  <-- traffic leaving the DC!
# 4  server-b (10.0.2.100)   50.1 ms

# Check for packet loss and jitter
mtr -r -c 100 server-b

# Check if the network interface has errors
ip -s link show eth0
# Look for: RX errors, TX errors, dropped, overruns

# Check for interface saturation
sar -n DEV 1 5
# Output:
# eth0  rxpck/s  txpck/s  rxkB/s  txkB/s
#       95000    92000    940000  120000
# (940MB/s on a 1Gbps link — that's saturated!)

# Check for TCP retransmissions
ss -ti | grep -i retrans
netstat -s | grep -i retrans
# Output: 45231 segments retransmitted (high!)

# Check if it's softirq/CPU related
mpstat -P ALL 1 3
# Look for high %soft on a single CPU (NIC interrupt affinity issue)

Root Cause: A routing change sent inter-server traffic through the WAN link instead of the local switch. The traffic is hairpinning out to the internet gateway and back.

Fix:

# Immediate: add a static route for the local subnet
ip route add 10.0.2.0/24 via 10.0.0.1 dev eth0

# Make it persistent
echo "10.0.2.0/24 via 10.0.0.1 dev eth0" >> /etc/sysconfig/network-scripts/route-eth0
# Or for Ubuntu:
cat >> /etc/netplan/01-config.yaml << EOF
      routes:
        - to: 10.0.2.0/24
          via: 10.0.0.1
EOF
netplan apply

# If it was NIC saturation:
# Enable TCP offloading
ethtool -K eth0 tso on gso on gro on

# If it was interrupt affinity:
# Spread NIC interrupts across multiple CPUs
echo 2 > /proc/irq/$(cat /proc/interrupts | grep eth0 | awk '{print $1}' | tr -d ':')/smp_affinity

---

Category 4: Process & Service

Scenario 13: Service Starts But Dies Immediately After

Scenario: "You start a service with systemctl start myapp and it says it started, but immediately goes to 'failed' state. No useful output on the terminal. How do you debug this?"

Your Approach:

The service is crashing right after launch. I need to check logs, the service unit file configuration, and try running the binary manually to see the error output.

Commands Used:

# Check the service status for hints
systemctl status myapp -l
# Output:
# ● myapp.service - My Application
#    Active: failed (Result: exit-code) since...
#    Process: 12345 ExecStart=/usr/local/bin/myapp (code=exited, status=1/FAILURE)
#    Main PID: 12345 (code=exited, status=1/FAILURE)

# Check journal logs for the actual error
journalctl -u myapp -n 50 --no-pager
# Output:
# myapp[12345]: Error: cannot bind to port 8080: Address already in use
# myapp[12345]: Fatal: failed to initialize. Exiting.
# systemd[1]: myapp.service: Main process exited, code=exited, status=1/FAILURE

# Try running the binary manually as the service user
sudo -u appuser /usr/local/bin/myapp --config /etc/myapp/config.yml
# This gives you the full error output without systemd masking it

# Check if it's a dependency issue (config file, directory, etc.)
ls -la /etc/myapp/config.yml
ls -la /var/lib/myapp/

# Check if required environment variables are set
systemctl show myapp | grep Environment
cat /etc/systemd/system/myapp.service | grep -A5 "\[Service\]"

# Check for library dependencies
ldd /usr/local/bin/myapp
# Output:
# libcustom.so => not found  (missing library!)

Root Cause: The application tries to bind to port 8080 which is already in use by another process. The error only shows up in journalctl, not in the terminal.

Fix:

# Find what's using port 8080
ss -tlnp | grep 8080
# Output: LISTEN  0  128  *:8080  users:(("nginx",pid=5678,fd=6))

# Either stop the conflicting service or change your app's port
systemctl stop nginx
systemctl start myapp

# Or change the port in your app config
sed -i 's/port: 8080/port: 8081/' /etc/myapp/config.yml
systemctl start myapp

# If the issue was a missing library:
ldconfig  # Refresh library cache
# Or install the missing package
apt install libcustom-dev

---

Scenario 14: Process Is Running But Not Responding

Scenario: "Your application process shows up in ps, it's using CPU, but it's not accepting connections or processing requests. Health checks are failing. What do you investigate?"

Your Approach:

A process that's alive but unresponsive is usually stuck — either in a deadlock, an infinite loop, or blocked on an external dependency. I'd attach to it and see what it's actually doing.

Commands Used:

# Confirm the process is alive and consuming resources
ps aux | grep myapp
# Output: app  12345 85.0  2.1 ... R  ... /usr/local/bin/myapp
# 85% CPU — it's doing something

# Check what the process is stuck on (system calls)
strace -p 12345 -c -t 5
# Output:
# % time     calls  syscall
# ------  --------  --------
# 99.8%      50000  futex     <-- deadlock/lock contention!

# Or see it in real-time
strace -p 12345 -e trace=network,write 2>&1 | head -20

# Get a thread dump for Java apps
kill -3 12345  # Sends SIGQUIT, generates thread dump
# Or
jstack 12345 > /tmp/thread_dump.txt

# Check if it's blocked on a network call
ss -tnp | grep 12345
# Output:
# ESTAB  0  0  10.0.0.5:45678  10.0.0.99:3306  users:(("myapp",pid=12345))
# Stuck on a MySQL connection that isn't responding!

# Check what files it has open
ls -la /proc/12345/fd | wc -l
# If this number is at the process limit, it can't accept new connections

# Check process limits
cat /proc/12345/limits | grep "open files"
# Max open files: 1024 (too low for a server!)

Root Cause: The application is stuck on a database connection. The MySQL server at 10.0.0.99 is overloaded and not responding to queries, causing all app threads to block while waiting for responses. No threads are free to accept new connections.

Fix:

# Check if the database is actually responsive
mysql -h 10.0.0.99 -u user -p -e "SELECT 1"
# If this hangs too, the DB is the problem

# Immediate: restart the app with connection timeouts
# Add to app config:
# connection_timeout: 5000  # 5 seconds
# read_timeout: 10000       # 10 seconds
# pool_size: 20
# pool_timeout: 3000

systemctl restart myapp

# Increase file descriptor limits
cat >> /etc/systemd/system/myapp.service.d/limits.conf << EOF
[Service]
LimitNOFILE=65536
EOF
systemctl daemon-reload
systemctl restart myapp

---

Scenario 15: Zombie Processes Accumulating

Scenario: "You notice hundreds of zombie (defunct) processes on a server. The system isn't running out of PIDs yet, but it's concerning. What causes this and how do you fix it?"

Your Approach:

Zombies are child processes that have exited but their parent hasn't called wait() to collect their exit status. They don't use CPU or memory, but they do consume PID entries.

Commands Used:

# Count zombie processes
ps aux | awk '$8=="Z" {count++} END {print count}'
# Output: 347

# Find the zombies and their parent processes
ps -eo pid,ppid,stat,cmd | grep "Z"
# Output:
# 23456 12345 Z  [worker] <defunct>
# 23457 12345 Z  [worker] <defunct>
# 23458 12345 Z  [worker] <defunct>
# ... all have the same parent PID 12345!

# Identify the parent process
ps -p 12345 -o pid,cmd
# Output: 12345 /usr/local/bin/job-scheduler

# Check if the parent is handling SIGCHLD properly
strace -p 12345 -e signal -t 10
# Look for: rt_sigaction(SIGCHLD, ...) or wait4() calls

# See the rate of zombie creation
watch -n 1 'ps aux | grep -c Z'

# Check PID limits
cat /proc/sys/kernel/pid_max
# Output: 32768
# With 347 zombies and growing, we'll hit the limit eventually

Root Cause: The job-scheduler process spawns worker processes but has a bug where it doesn't wait() for them after they finish. It's ignoring the SIGCHLD signal instead of handling it properly.

Fix:

# Option 1: Restart the parent process (clears all its zombies)
systemctl restart job-scheduler

# Option 2: Send SIGCHLD to the parent to trigger reaping (if it handles it)
kill -SIGCHLD 12345

# Option 3: Kill the parent — init/systemd adopts zombies and reaps them
kill 12345
# systemd (PID 1) will automatically wait() on orphaned zombies

# Option 4: Increase PID limit as a temporary buffer
echo 4194304 > /proc/sys/kernel/pid_max

# Long-term fix in application code:
# - Add signal(SIGCHLD, SIG_IGN) to auto-reap children
# - Or implement proper wait()/waitpid() in parent process
# - Or use double-fork technique to daemonize children

---

Scenario 16: Service Won't Start, Port Already in Use

Scenario: "You try to restart Nginx after a config change and it fails with 'Address already in use' for port 80. The old Nginx process isn't showing in systemctl status. How do you recover?"

Your Approach:

Something is still holding port 80. It could be an orphaned Nginx worker, another web server, or even a leftover process from a failed restart.

Commands Used:

# Find what's holding port 80
ss -tlnp | grep :80
# Output:
# LISTEN  0  511  0.0.0.0:80  users:(("nginx",pid=9876,fd=6))

# So there IS an Nginx process, but systemd doesn't know about it
# Check if it's a different Nginx instance
ps aux | grep nginx
# Output:
# root  9875  ... nginx: master process /usr/sbin/nginx (this is the old one)
# www   9876  ... nginx: worker process
# www   9877  ... nginx: worker process

# Why doesn't systemd know? Check the PID file
cat /run/nginx.pid
# Output: 9875 (matches the running process)

# But systemd's tracking is out of sync
systemctl status nginx
# Output: inactive (dead)

# This happens when someone started nginx manually outside systemd
# Or systemd lost track after a failed reload

# Check if there's a different nginx binary
which nginx
ls -la /usr/sbin/nginx /usr/local/nginx/sbin/nginx 2>/dev/null

Root Cause: Someone started Nginx manually with /usr/sbin/nginx (bypassing systemd) during troubleshooting and forgot about it. Systemd doesn't track manually-started processes, so systemctl start nginx fails because the port is occupied.

Fix:

# Kill the orphaned Nginx processes
nginx -s stop
# Or if that doesn't work:
kill 9875
# Verify it's gone
ss -tlnp | grep :80

# Now start properly through systemd
systemctl start nginx
systemctl status nginx
# Output: active (running)

# If you need to force-kill (last resort):
fuser -k 80/tcp
# This kills ANY process using port 80

# Prevent this in the future — always use systemctl
# Add to team runbook: NEVER start services manually

---

Category 5: Security & Permissions

Scenario 17: User Can't Execute a Script They Own

Scenario: "A developer says they can't run their own shell script. They own the file, they can cat it, but ./script.sh gives 'Permission denied'. What's wrong?"

Your Approach:

If they can read it but not execute it, the first thing to check is the execute permission bit. But there are several other sneaky causes too.

Commands Used:

# Check the file permissions
ls -la script.sh
# Output: -rw-r--r-- 1 developer developer 1234 Jan 5 10:00 script.sh
# No execute bit! That's the obvious answer. But let's go deeper...

# Fix the obvious case
chmod +x script.sh
./script.sh
# Still fails? Check the filesystem mount options

# Check if the filesystem is mounted with noexec
mount | grep $(df script.sh | tail -1 | awk '{print $1}')
# Output: /dev/sdb1 on /home type ext4 (rw,noexec,nosuid,relatime)
#                                            ^^^^^^ there it is!

# Check for ACLs overriding standard permissions
getfacl script.sh
# Output:
# user::rw-
# user:developer:rw-
# group::r--
# mask::rw-       <-- mask is blocking execute even if you add it!

# Check if SELinux is blocking execution
ls -Z script.sh
# Output: unconfined_u:object_r:user_home_t:s0 script.sh
getenforce
# Output: Enforcing

ausearch -m avc -ts recent | grep script.sh

Root Cause: The /home partition is mounted with noexec option. This is a security hardening measure that prevents execution of any binary or script on that filesystem, regardless of file permissions.

Fix:

# Option 1: Run the script through the interpreter explicitly
bash script.sh
# This works because you're executing bash (from /usr/bin), not the script itself

# Option 2: Remount without noexec (if policy allows)
mount -o remount,exec /home

# Option 3: Copy to a filesystem that allows execution
cp script.sh /tmp/
chmod +x /tmp/script.sh
/tmp/script.sh

# Option 4: If it's an ACL issue, fix the mask
setfacl -m m::rwx script.sh

# Option 5: If SELinux is blocking
chcon -t bin_t script.sh
# Or create a proper policy module

---

Scenario 18: SSH Key Authentication Not Working

Scenario: "You've added your public key to ~/.ssh/authorized_keys on a server, but it still asks for a password. What do you check?"

Your Approach:

SSH key auth is extremely picky about file permissions. If anything in the chain (home dir, .ssh dir, authorized_keys file) has wrong permissions, SSH silently falls back to password auth.

Commands Used:

# First, try with verbose output to see why the key is rejected
ssh -vvv user@server 2>&1 | grep -A2 "Offering\|Trying\|Authentication"
# Output:
# debug1: Offering public key: /home/user/.ssh/id_rsa
# debug1: Server accepts key: /home/user/.ssh/id_rsa
# debug1: Authentication that can continue: publickey,password
# debug3: sign_and_send_pubkey: signing failed: agent refused operation

# On the server, check permissions (the #1 cause)
ls -la ~/ | grep -E "^\.|total"
# Output: drwxrwxrwx 5 user user 4096 ... .   <-- home dir is world-writable!

ls -la ~/.ssh/
# Output:
# drw------- 2 user user 4096 ... .
# -rw------- 1 user user  400 ... authorized_keys  (looks correct)

# Check SSH daemon logs for the actual rejection reason
tail -50 /var/log/auth.log | grep sshd
# Output:
# sshd[12345]: Authentication refused: bad ownership or modes for directory /home/user

# Check SSH config on the server
grep -i "StrictModes\|AuthorizedKeysFile\|PubkeyAuthentication" /etc/ssh/sshd_config
# Output:
# StrictModes yes  (this enforces permission checks!)
# PubkeyAuthentication yes
# AuthorizedKeysFile .ssh/authorized_keys

# Verify the key format is correct
ssh-keygen -l -f ~/.ssh/authorized_keys
# If this errors, the key file is malformed (line breaks, extra spaces)

Root Cause: The user's home directory (/home/user) is world-writable (permissions 777). With StrictModes yes, SSH refuses to trust authorized_keys if any directory in the path is writable by others.

Fix:

# Fix permissions on the entire chain
chmod 755 /home/user        # or 700
chmod 700 /home/user/.ssh
chmod 600 /home/user/.ssh/authorized_keys

# Ensure correct ownership
chown -R user:user /home/user/.ssh

# If the key format is wrong (copied with line breaks):
# Remove and re-add the key
ssh-copy-id user@server
# Or manually paste the key on ONE line in authorized_keys

# If SELinux is relabeling:
restorecon -Rv /home/user/.ssh

# Restart SSH if you changed sshd_config
systemctl restart sshd

---

Scenario 19: Sudo Not Working for a User

Scenario: "A user reports that sudo gives them 'user is not in the sudoers file. This incident will be reported.' You're sure you added them to the right group. What went wrong?"

Your Approach:

I'd verify the user's group membership, check the sudoers configuration, and look for syntax errors that might have broken the entire sudo config.

Commands Used:

# Check what groups the user is actually in
id username
# Output: uid=1001(username) gid=1001(username) groups=1001(username)
# Notice: they're NOT in the sudo/wheel group!

# Wait, did we add them correctly?
grep username /etc/group
# Output: sudo:x:27:username
# They ARE in the group in /etc/group... so why doesn't id show it?

# The user hasn't logged out and back in since being added!
# Current session doesn't pick up new group memberships

# Alternative: check if there's a sudoers syntax error
visudo -c
# Output:
# /etc/sudoers.d/custom: syntax error near line 3
# parse error in /etc/sudoers.d/custom near line 3

# The broken file breaks ALL sudo, not just for this user!
cat /etc/sudoers.d/custom
# Output:
# username ALL=(ALL) ALL
# %developers ALL=(ALL) NOPASSWD ALL  <-- missing colon before ALL!
# admin ALL=(ALL:ALL) ALL

# Check if there's a requiretty setting blocking
grep requiretty /etc/sudoers

Root Cause: Two issues: (1) The user hasn't started a new login session since being added to the group, and (2) a syntax error in /etc/sudoers.d/custom broke sudo for everyone.

Fix:

# Fix the sudoers syntax error (use visudo to validate!)
# NEVER edit sudoers files directly — always use visudo
visudo -f /etc/sudoers.d/custom
# Fix the line:
# %developers ALL=(ALL:ALL) NOPASSWD: ALL

# Validate the fix
visudo -c
# Output: /etc/sudoers: parsed OK

# For the group membership issue — user needs to log out and back in
# Or use newgrp as a workaround:
newgrp sudo

# If you're completely locked out of sudo:
# Boot into single-user mode or use root console
# Or if pkexec is available:
pkexec visudo -f /etc/sudoers.d/custom

# Prevention: always validate before saving
echo "username ALL=(ALL:ALL) ALL" | visudo -c -f /dev/stdin

---

Scenario 20: Mysterious File Permission Changes After Reboot

Scenario: "Every time the server reboots, certain application files in /opt/myapp lose their custom permissions. You set them to 775 with specific ownership, but after reboot they're back to 755 owned by root. What's resetting them?"

Your Approach:

Something is actively resetting permissions on boot. The usual suspects: systemd-tmpfiles, a startup script, configuration management (Ansible/Puppet), or the package manager's config.

Commands Used:

# Check if systemd-tmpfiles is managing this path
grep -r "/opt/myapp" /etc/tmpfiles.d/ /usr/lib/tmpfiles.d/
# Output:
# /usr/lib/tmpfiles.d/myapp.conf:d /opt/myapp 0755 root root -

# Check if there's a systemd service that runs on boot
grep -r "/opt/myapp" /etc/systemd/system/ /usr/lib/systemd/system/
# Output:
# /etc/systemd/system/myapp.service:ExecStartPre=/usr/bin/install -d -m 755 -o root /opt/myapp

# Check for init scripts or cron jobs at boot
grep -r "chmod\|chown" /etc/rc.local /etc/init.d/ /etc/cron.d/ 2>/dev/null
# Output:
# /etc/rc.local: chown -R root:root /opt/myapp && chmod -R 755 /opt/myapp

# Check if a configuration management tool runs on boot
systemctl list-timers | grep -i "puppet\|ansible\|chef"
crontab -l | grep -i "puppet\|ansible"

# Check rpm/dpkg for config file protection
rpm -V myapp-package 2>/dev/null
dpkg --verify myapp 2>/dev/null
# Output: .M....... /opt/myapp/config (Mode differs — package wants to "fix" it)

Root Cause: Three culprits working together: (1) A systemd-tmpfiles rule resets permissions at boot, (2) the ExecStartPre directive in the service file recreates the directory with root ownership, and (3) an old entry in /etc/rc.local from a previous admin.

Fix:

# Fix 1: Override the tmpfiles rule
cat > /etc/tmpfiles.d/myapp-override.conf << EOF
d /opt/myapp 0775 appuser appgroup -
EOF
# This overrides /usr/lib/tmpfiles.d/myapp.conf

# Fix 2: Modify the service file ExecStartPre
systemctl edit myapp
# Add:
# [Service]
# ExecStartPre=
# ExecStartPre=/usr/bin/install -d -m 775 -o appuser -g appgroup /opt/myapp

# Fix 3: Remove the rc.local entry
sed -i '/\/opt\/myapp/d' /etc/rc.local

# Fix 4: If it's the package manager, tell it to ignore this path
# For RPM: add %config(noreplace) in spec
# For dpkg: dpkg-statoverride --add appuser appgroup 0775 /opt/myapp

# Verify: apply tmpfiles rule now without rebooting
systemd-tmpfiles --create /etc/tmpfiles.d/myapp-override.conf
ls -la /opt/myapp

---

Interview Tips: How to Stand Out

Now that you've seen all 20 scenarios, here are some meta-tips for handling these in actual interviews:

1. Narrate your thought process. Don't just jump to the answer. Say "First I'd check X because..." Interviewers want to see HOW you think, not just that you know the command.

2. Start broad, then narrow down. Always begin with the least invasive commands (ps, top, df, ss) before reaching for more intensive tools (strace, tcpdump, perf).

3. Mention the quick wins. Before diving into complex debugging, mention you'd check logs first: journalctl -u service -n 100, dmesg | tail, /var/log/syslog.

4. Talk about prevention. After solving the problem, briefly mention how you'd prevent it: monitoring alerts, log rotation, capacity planning, automated remediation.

5. Know when to escalate. If hardware is failing (bad SMART data, ECC errors), say you'd replace the hardware rather than spending hours on software workarounds.

6. Practice in a lab. Spin up VMs and deliberately break things. Create the scenarios yourself — fill a disk, exhaust file descriptors, create zombie processes. Muscle memory matters.

---

Quick Reference: Essential Troubleshooting Commands

# System overview
uptime && free -h && df -h && iostat -x 1 1

# Find resource hogs
top -bn1 | head -20
ps aux --sort=-%cpu | head -10
ps aux --sort=-%mem | head -10

# Disk investigation
lsof +L1              # Deleted files still open
du -sh /* | sort -rh  # Where's space being used
find / -xdev -size +100M  # Large files

# Network debugging
ss -tlnp              # Listening ports
ss -tn | awk '{print $4}' | sort | uniq -c | sort -rn  # Connection counts
tcpdump -i any port 80 -nn -c 50  # Packet capture

# Process debugging
strace -p PID -c      # System call summary
lsof -p PID           # Open files/connections
cat /proc/PID/status  # Full process info

These scenarios represent the bread and butter of Linux troubleshooting in DevOps. Master them, and you'll handle 90% of what interviewers throw at you. The remaining 10%? That's where your genuine curiosity and willingness to say "I'd look at the documentation for that specific case" comes in. Nobody knows everything — but knowing where to look is what separates good engineers from great ones.

---

Frequently Asked Questions

What Linux topics are most commonly asked in DevOps interviews?

The most common topics are: process management (ps, top, kill, systemctl), filesystem and permissions (chmod, chown, find), networking (netstat, ss, iptables, DNS troubleshooting), disk management (df, du, mount, LVM), and scripting (bash loops, conditionals, text processing with awk/sed/grep). Expect hands-on scenario questions rather than pure theory.

How do I troubleshoot a Linux server that is running slow?

Follow a systematic approach: check CPU usage with top or htop, memory with free -m, disk I/O with iostat or iotop, and network with iftop. Identify the top resource-consuming processes, check for runaway processes or memory leaks, verify disk space isn't full with df -h, and review system logs in /var/log/syslog for errors.

What are the most important Linux commands for DevOps?

Essential commands include: systemctl (service management), journalctl (logs), top/htop (process monitoring), df/du (disk), netstat/ss (networking), grep/awk/sed (text processing), find (file search), chmod/chown (permissions), tar/gzip (archiving), and curl/wget (HTTP testing). Master these and you can troubleshoot most production issues.

How do I answer scenario-based Linux interview questions?

Structure your answer with: identify the problem category (CPU, memory, disk, network), explain what commands you'd run and why, describe how you'd interpret the output, and state what action you'd take to resolve it. Demonstrate systematic troubleshooting rather than jumping to conclusions. Always mention checking logs as part of your investigation.