Linux Performance Troubleshooting: A Senior Engineer's Toolkit

The 60-Second Analysis

When you SSH into a box that's on fire, you don't have time to think. You need a systematic approach that narrows down the problem fast. Here's the sequence I run every time:

# The first 60 seconds
uptime              # Load averages - is the system overloaded?
dmesg | tail        # Kernel messages - OOM kills, hardware errors?
vmstat 1 5          # CPU, memory, I/O at a glance
iostat -xz 1 5     # Disk I/O - is storage the bottleneck?
free -h             # Memory usage overview
top -bn1 | head -20 # Top processes right now
ss -tlnp            # What's listening, what's connected?

This tells you within a minute whether you're dealing with CPU saturation, memory pressure, disk I/O bottleneck, or network issues. Let's go deeper on each.

CPU Troubleshooting

Reading <code class="inline-code">top</code> Like a Pro

top -bn1

top - 14:32:01 up 45 days,  3:21,  2 users,  load average: 12.34, 8.21, 4.56
Tasks: 312 total,   4 running, 308 sleeping,   0 stopped,   0 zombie
%Cpu(s): 78.2 us, 12.1 sy,  0.0 ni,  5.4 id,  2.1 wa,  0.0 hi,  2.2 si,  0.0 st
MiB Mem :  15906.2 total,    234.1 free,  14201.3 used,   1470.8 buff/cache
MiB Swap:   4096.0 total,   2048.0 free,   2048.0 used.    892.4 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 8432 app       20   0   4.2g   3.1g   12m  S 312.0 20.1  1024:32 java
 9021 app       20   0   812m   402m    8m  S  89.0  2.5    512:11 python3

What this tells us:

• Load average 12.34 on, say, a 4-core box means 3x overloaded. Load should be ≤ number of cores.
• 78.2% us (user): CPU is spending most time in application code. Not a kernel issue.
• 12.1% sy (system): Elevated syscall overhead. Could be excessive I/O or context switching.
• 2.1% wa (iowait): Some processes are waiting on disk.
• 0.0% st (steal): No hypervisor stealing CPU (good — rules out noisy neighbor on shared hosts).

Finding CPU-Hungry Threads

# Show individual threads for a process
top -H -p 8432

# Get a thread dump for Java
jstack 8432 > /tmp/thread_dump.txt

# Correlate thread ID (convert PID to hex for Java)
printf "%x\n" 8445
# Output: 20fd — search for "nid=0x20fd" in the thread dump

CPU Scheduling Issues

# Check context switches — high values mean too many threads fighting for CPU
vmstat 1 5

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
14  2 2048000 234100  12040 1458800    0    0     4  1200 8200 52000 78 12  5  2  0
12  1 2048000 231200  12040 1458800    0    0     0   800 7800 48000 75 14  8  2  0

• r=14: 14 processes waiting for CPU time. We're heavily oversubscribed.
• cs=52000: 52k context switches per second. Could indicate too many threads or contention.
• b=2: 2 processes in uninterruptible sleep (usually waiting on disk).

Memory Troubleshooting

Understanding <code class="inline-code">free</code> Output

free -h

total        used        free      shared  buff/cache   available
Mem:           15Gi       13Gi       234Mi       128Mi       1.5Gi       892Mi
Swap:         4.0Gi       2.0Gi       2.0Gi

Critical insight: available is what matters, not free. Linux uses "free" memory for buffer/cache, which is instantly reclaimable. Available = free + reclaimable cache.

Here, available is only 892MB on a 15GB box with 2GB swap used. This system is under memory pressure.

Checking for OOM Kills

# Recent OOM kills
dmesg | grep -i "oom\|killed process"

# Or in journald
journalctl -k | grep -i oom

# Which process was killed and why
dmesg | grep -A 5 "Out of memory"

[14234.567890] Out of memory: Killed process 8432 (java) total-vm:4398080kB, anon-rss:3254780kB

Finding Memory Leaks

# Sort processes by resident memory
ps aux --sort=-%mem | head -20

# Watch memory growth over time (run every 10 seconds)
watch -n 10 'ps -eo pid,ppid,rss,vsz,comm --sort=-rss | head -15'

# Detailed memory map for a specific process
pmap -x 8432 | tail -5

# Check for memory fragmentation
cat /proc/buddyinfo

Swap Behavior

# Who's using swap?
for proc in /proc/*/status; do
  awk '/VmSwap|Name/{printf $2 " "}END{print ""}' "$proc" 2>/dev/null
done | sort -k2 -n | tail -10

# Current swappiness (0-100, lower = prefer killing vs swapping)
cat /proc/sys/vm/swappiness

# Temporarily reduce swappiness for memory-sensitive apps
echo 10 | sudo tee /proc/sys/vm/swappiness

Disk I/O Troubleshooting

Reading <code class="inline-code">iostat</code> Output

iostat -xz 1 5

Device  r/s     w/s    rkB/s    wkB/s  rrqm/s  wrqm/s  %rrqm  %wrqm  r_await  w_await  aqu-sz  rareq-sz  wareq-sz  svctm  %util
nvme0n1 125.00  890.00 2000.00  45600.00  0.00   120.00   0.00   11.88    0.80    12.40    8.92    16.00     51.24   0.98   99.10

Key indicators:

• %util=99.10%: This disk is completely saturated. It's the bottleneck.
• w_await=12.40ms: Write latency is elevated. Normal for NVMe is <1ms.
• aqu-sz=8.92: Average queue size. >1 means requests are waiting.
• r/s + w/s: Total IOPS. Compare to your disk's rated capacity.

Finding I/O-Heavy Processes

# Real-time I/O per process
iotop -oP

# Or without iotop installed
pidstat -d 1 5

Average:      UID       PID   kB_rd/s   kB_wr/s kB_ccwr/s iodelay  Command
Average:     1000      8432      0.00  42000.00      0.00      22  java
Average:      999      1234    800.00    200.00      0.00       3  postgres

The Java process is writing 42MB/s and experiencing I/O delays.

Checking for Filesystem Issues

# Filesystem usage
df -h

# Inode usage (you can run out of inodes before running out of space)
df -i

# Find large files
find / -type f -size +1G -exec ls -lh {} \; 2>/dev/null

# Find directories with many files (inode hogs)
find /var -xdev -type d -exec sh -c 'echo "$(find "$1" -maxdepth 1 | wc -l) $1"' _ {} \; | sort -n | tail -20

Network Troubleshooting

Connection State Analysis

# Summary of all connection states
ss -s

Total: 1842
TCP:   1523 (estab 1200, closed 89, orphaned 12, timewait 134)

# Connections by state
ss -tan | awk '{print $1}' | sort | uniq -c | sort -rn

1200 ESTAB
    134 TIME-WAIT
     89 CLOSE-WAIT
     67 SYN-SENT
     33 LISTEN

Red flags:

• High CLOSE-WAIT: Your application isn't closing connections properly. It's a bug in your code.
• High TIME-WAIT: Many short-lived connections. Consider connection pooling.
• High SYN-SENT: Can't reach downstream services. DNS or connectivity issue.

Finding Connection Leaks

# Connections per remote host
ss -tn | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn | head -10

# Connections per process
ss -tlnp | grep LISTEN

# Watch for connection growth over time
watch -n 5 'ss -s'

Bandwidth and Packet Analysis

# Real-time bandwidth per interface
sar -n DEV 1 5

# Packet drops and errors
ip -s link show eth0

# TCP retransmits (indicator of network congestion)
netstat -s | grep -i retrans
# Or
ss -ti | grep -c retrans

# Quick packet capture for debugging
tcpdump -i eth0 -nn port 5432 -c 100 -w /tmp/db_traffic.pcap

# DNS resolution timing
dig +stats api.internal.service

Putting It All Together: Real Scenario

Symptom: API response times jumped from 50ms to 2000ms at 2pm.

# Step 1: Load average
$ uptime
14:05:32 up 45 days, load average: 2.10, 1.98, 1.45
# Load is fine for an 8-core box

# Step 2: CPU
$ vmstat 1 3
 r  b   swpd   free  si  so  bi    bo    cs   us sy id wa
 2  4      0  892000  0   0  45000 1200  3200  15 8  22 55
# wa=55%! Processes are waiting on disk I/O

# Step 3: Disk
$ iostat -xz 1 3
Device   %util  r_await  w_await  aqu-sz
sda      100.0    45.2     89.3     12.4
# Disk is 100% utilized with high latency

# Step 4: What's doing the I/O?
$ pidstat -d 1 3
PID   kB_rd/s   kB_wr/s  Command
5432  180000.0   12000.0  postgres
# PostgreSQL is reading 180MB/s

# Step 5: What query?
$ sudo -u postgres psql -c "SELECT pid, query, state FROM pg_stat_activity WHERE state = 'active';"
# Found: Full table scan on a 50GB table — missing index

Root cause: A new feature deployed without an index on a frequently-queried column. The fix was a single CREATE INDEX CONCURRENTLY statement.

Quick Reference: Command Cheat Sheet

Final Thought

Performance troubleshooting is pattern recognition. The more you practice this sequence, the faster you converge on root causes. Keep these commands in muscle memory. When your pager fires at 3am, you don't want to be reading man pages — you want to be fixing the issue.

The 60-second analysis flow rarely fails: uptime → vmstat → iostat → ss. Four commands, and you know which subsystem is the bottleneck. From there, you zoom in.

---

Frequently Asked Questions

How do I identify what is causing high CPU usage in Linux?

Run top and press P to sort by CPU, or use ps aux --sort=-%cpu | head -20 for a snapshot. For more detail, use pidstat 1 to see per-process CPU usage over time. If the high CPU is in system time (sy), use strace -p <pid> to see system calls. For persistent investigation, use perf top or perf record to profile at the function level.

What tools should I use for Linux memory analysis?

Start with free -m for overall memory status, then top sorted by memory (press M). Use vmstat 1 to watch for swapping (si/so columns). For per-process detail, check /proc/<pid>/status for VmRSS (actual RAM used). Use smem for proportional memory accounting and valgrind or /proc/<pid>/smaps for memory leak investigation.

How do I troubleshoot high disk I/O on Linux?

Use iostat -x 1 to see per-device I/O utilization and queue depth, then iotop to identify which processes are generating the I/O. High await times indicate slow storage. Use lsof +D /path to see which files processes have open, and blktrace for deep block-level analysis. Common fixes include adding more RAM (reduces page cache pressure) or moving to SSDs.

What does high load average mean and how do I diagnose it?

Load average represents the average number of processes waiting for CPU or I/O. A load above your CPU core count indicates saturation. If CPU is low but load is high, processes are waiting on I/O — check with vmstat (wa column) and iostat. If both CPU and load are high, you have CPU saturation. Use ps aux to find the processes in D (uninterruptible sleep) state for I/O-bound issues.

How do I check network performance issues on Linux?

Use ss -s for connection statistics, ss -tnp for established connections, sar -n DEV 1 for interface throughput, and netstat -i for error/drop counters. Test bandwidth with iperf3, latency with ping and mtr, and DNS resolution with dig. Check for packet loss, retransmissions (in netstat -s), and connection timeouts indicating network congestion or misconfiguration.

---

Related Resources

• Linux Commands Reference — 50+ Linux commands with production examples
• DevOps Interview Academy — Linux interview questions
• Linux Process Management Guide — Managing processes affecting performance
• How to Check Disk Space in Linux — Diagnosing disk-related performance issues
• Monitoring and Alerting in Production — Setting up performance monitoring