grep, find & awk — Searching 50GB of Logs Without Losing Your Mind

The Problem

It is 3 AM and your application is throwing 500 errors. You have 50GB of logs spread across multiple files. You need to find the specific error, correlate it with a timestamp, count occurrences, and identify the pattern — all from the command line.

grep, find, and awk are your log analysis toolkit. Master their combination and you can answer any question in seconds.

grep — Search Text Content

Basic matching

grep "ERROR" /var/log/app/application.log
grep -i "connection refused" /var/log/app/application.log
grep -r "OutOfMemoryError" /var/log/
grep -n "FATAL" /var/log/app/application.log
grep -B 3 -A 2 "NullPointerException" /var/log/app/application.log
grep -c "ERROR" /var/log/app/application.log
grep -rl "database timeout" /var/log/

Regular expressions

grep -E "HTTP/(4|5)[0-9]{2}" access.log
grep -E "\b[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\b" access.log
grep -E "2024-01-1[5-9]T(0[0-9]|1[0-2]):" application.log
grep -E "ERROR|FATAL|CRITICAL" application.log
grep -v "DEBUG" application.log
grep -v -E "health_check|readiness" access.log
grep -w "error" application.log

Production patterns

# Unique error types in the last hour
grep "ERROR" application.log | grep "$(date +%Y-%m-%dT%H)" | \
  sed 's/.*ERROR //' | sort | uniq -c | sort -rn

# Failed HTTP requests
grep -E "HTTP/1\.[01]\" [45][0-9]{2}" access.log

# Slow requests (>5000ms)
grep -E "response_time=[5-9][0-9]{3}|response_time=[0-9]{5,}" access.log

# Track request ID across files
grep -r "req-abc123-def456" /var/log/app/

# Search compressed logs
zgrep "ERROR" /var/log/app/application.log.*.gz

find — Locate Files

By name and path

find /var/log -name "*.log"
find /var/log -iname "*.LOG"
find /opt/app -name "config"

By time

find /var/log -name "*.log" -mtime -1      # Modified last 24h
find /var/log -name "*.log" -mtime +7      # Older than 7 days
find /var/log -name "*.log" -mmin -30      # Modified last 30 min

By size

find /var/log -type f -size +100M
find /var/log -type f -size +10M -size -100M
find /var/log -type f -empty

With actions

find /var/log/app -name "*.log" -mtime +30 -delete
find /var/log/app -name "*.log" -mtime +1 -exec gzip {} \;
find / -type f -size +500M -exec ls -lh {} \; 2>/dev/null
find /var/log -name "*.log" -mtime -1 -exec grep -l "ERROR" {} \;
find /var/log -name "*.log" -mtime +30 -print0 | xargs -0 rm -f

awk — Structured Text Processing

Column extraction

awk '{print $1, $4}' access.log
awk -F',' '{print $2, $5}' data.csv
awk '{print $NF}' access.log        # Last column
awk '{print $(NF-1)}' access.log    # Second-to-last

Filtering

awk '$9 == 500' access.log
awk '$11 > 5000' access.log
awk '$7 ~ /api\/v1\/users/' access.log
awk '$9 >= 500 && $11 > 1000' access.log

Aggregation and statistics

# Count per status code
awk '{count[$9]++} END {for (code in count) print code, count[code]}' access.log | sort -k2 -rn

# Sum of response sizes
awk '{sum += $10} END {print "Total bytes:", sum}' access.log

# Average response time
awk '{sum += $11; count++} END {print "Average:", sum/count, "ms"}' access.log

# Top 10 URLs
awk '{count[$7]++} END {for (url in count) print count[url], url}' access.log | sort -rn | head -10

# Top 10 IPs
awk '{count[$1]++} END {for (ip in count) print count[ip], ip}' access.log | sort -rn | head -10

# Unique IPs
awk '{ips[$1]++} END {print "Unique IPs:", length(ips)}' access.log

Multi-line processing

# Extract stack traces (between pattern and blank line)
awk '/Exception/,/^$/' application.log

# Process blocks separated by blank lines
awk 'BEGIN{RS=""} /ERROR/' application.log

Combining All Three

Incident investigation

# Which log files had errors recently, ranked by count
find /var/log/app -name "*.log" -mmin -60 -exec sh -c \
  'count=$(grep -c "ERROR" "$1" 2>/dev/null); [ "$count" -gt 0 ] && echo "$count $1"' _ {} \; | sort -rn

# Error rates per minute
grep "ERROR" application.log | grep "$(date +%Y-%m-%dT%H)" | \
  awk -F'T' '{split($2,a,":"); print $1"T"a[1]":"a[2]}' | sort | uniq -c

# Most common error messages
grep "ERROR" application.log | \
  awk -F'ERROR' '{print $2}' | \
  sed 's/^ *//' | sort | uniq -c | sort -rn | head -20

# High response time requests with details
awk '$9 >= 500 {print $4, $7, $9, $11"ms"}' access.log | sort -t' ' -k4 -rn | head -20

Disk investigation

# Directories with most recently modified large files
find /var -type f -mtime -7 -printf '%s %h\n' | \
  awk '{sum[$2]+=$1} END {for (d in sum) print sum[d], d}' | sort -rn | head -20

# Large log files modified today
find /var/log -name "*.log" -mtime -1 -exec ls -la {} \; | \
  awk '$5 > 100000000 {print $5/1048576"MB", $9}'

Common Mistakes

Using grep without -r for directory search — grep does not recurse by default. Use grep -r or grep -R.

Forgetting to escape regex metacharacters — ., *, [, ( have special meaning. Use grep -F for literal strings.

Not using -print0 and xargs -0 — Filenames with spaces break without null-terminated handling.

awk field separator confusion — Default is whitespace. For CSVs: awk -F','. For colons: awk -F':'.

Ignoring compressed logs — Rotated logs are gzipped. Use zgrep and zcat.

Running find without -type f — Returns both files and directories, producing confusing results with -exec.

Quick Reference

Summary

grep finds text, find locates files, and awk processes structured data. Together they answer any log analysis question without specialized tools. The patterns here cover 90% of production scenarios — error rates, top offenders, correlation analysis, and disk investigation.

---

Frequently Asked Questions

What is the difference between grep, find, and awk?

grep searches file contents for patterns and prints matching lines. find searches the filesystem for files/directories by name, type, size, or time. awk is a programming language for processing structured text, especially columnar data. Use grep to find text in files, find to locate files, and awk to extract and transform fields.

How do I use grep to search recursively in all files?

Use grep -r "pattern" /path/to/directory for recursive search. Add -l to show only filenames, -n for line numbers, -i for case-insensitive matching. Modern alternative: grep -rn --include="*.py" "import os" to search only Python files. For better performance on large codebases, use ripgrep (rg) which respects .gitignore by default.

How do I find and delete files older than 30 days?

Use find /path -type f -mtime +30 -delete to delete files modified more than 30 days ago. Always test first without -delete by using -print or -ls to see what would be removed. Add -name "*.log" to target specific file types. For safer deletion, use -exec rm -i {} \; for interactive confirmation.

What is awk and when should I use it instead of cut?

Use awk when you need conditional logic, field manipulation, or calculations on columnar data. While cut splits on simple delimiters, awk handles multiple delimiters, variable-width fields, and can perform math. Example: awk '$3 > 80 {print $1, $3}' data.txt prints username and score only when score exceeds 80.

---

Related Resources

• Linux Commands Reference — 50+ Linux commands with production examples
• grep Command Reference — Detailed grep command usage and examples
• DevOps Interview Academy — Linux interview questions
• Linux Performance Troubleshooting — Using text tools to analyze performance logs
• Docker Container Logs Debugging — Applying grep and awk to container logs