TL;DR Quick Fix
If you are seeing intermittent timeouts without CPU/memory spikes, check for packet loss:
# Quick check: TCP retransmissions on a Linux host
ss -ti | grep -E "retrans|rto"
# Check conntrack table overflow (common cause of silent drops)
cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max
# If count is near max, you are dropping packets silently
# Immediate fix: increase conntrack table size
sudo sysctl -w net.netfilter.nf_conntrack_max=524288
sudo sysctl -w net.netfilter.nf_conntrack_buckets=131072
For VPC-level diagnosis, enable Flow Logs and query with Athena:
SELECT srcaddr, dstaddr, srcport, dstport, protocol, action, packets
FROM vpc_flow_logs
WHERE action = 'REJECT'
AND date = current_date - interval '1' day
ORDER BY packets DESC
LIMIT 50;
---
Architecture Overview
---
Symptoms of Silent Packet Loss
Silent packet loss is tricky because monitoring looks normal:
- CPU and memory are fine
- No error logs from the application
- But requests timeout intermittently (0.1-2% of traffic)
- Latency spikes appear in P99 but not P50
- Health checks pass but users report failures
---
VPC Flow Logs Analysis with Athena
Enable Flow Logs
# terraform/flow-logs.tf
resource "aws_flow_log" "vpc" {
vpc_id = aws_vpc.main.id
traffic_type = "ALL"
log_destination_type = "s3"
log_destination = aws_s3_bucket.flow_logs.arn
# Enhanced fields for better debugging
log_format = "$${version} $${account-id} $${interface-id} $${srcaddr} $${dstaddr} $${srcport} $${dstport} $${protocol} $${packets} $${bytes} $${start} $${end} $${action} $${log-status} $${tcp-flags} $${flow-direction}"
}
resource "aws_athena_database" "flow_logs" {
name = "vpc_flow_logs"
bucket = aws_s3_bucket.athena_results.id
}
Athena Queries for Packet Loss Detection
-- Find rejected traffic patterns
SELECT
srcaddr, dstaddr, dstport, protocol,
SUM(packets) as total_packets,
COUNT(*) as flow_count
FROM vpc_flow_logs
WHERE action = 'REJECT'
AND date_partition >= date_format(current_date - interval '1' day, '%Y/%m/%d')
GROUP BY srcaddr, dstaddr, dstport, protocol
ORDER BY total_packets DESC
LIMIT 20;
-- Detect asymmetric routing (traffic goes out but no response)
SELECT
a.srcaddr, a.dstaddr, a.dstport,
a.packets as outbound_packets,
COALESCE(b.packets, 0) as return_packets
FROM
(SELECT srcaddr, dstaddr, dstport, SUM(packets) as packets
FROM vpc_flow_logs WHERE flow_direction = 'egress' AND action = 'ACCEPT'
GROUP BY srcaddr, dstaddr, dstport) a
LEFT JOIN
(SELECT srcaddr, dstaddr, srcport, SUM(packets) as packets
FROM vpc_flow_logs WHERE flow_direction = 'ingress' AND action = 'ACCEPT'
GROUP BY srcaddr, dstaddr, srcport) b
ON a.dstaddr = b.srcaddr AND a.dstport = b.srcport
WHERE COALESCE(b.packets, 0) < a.packets * 0.9 -- More than 10% loss
ORDER BY a.packets DESC;
-- Identify potential conntrack overflow periods
SELECT
date_trunc('minute', from_unixtime(start)) as time_window,
COUNT(*) as total_flows,
SUM(CASE WHEN action = 'REJECT' THEN 1 ELSE 0 END) as rejected_flows,
CAST(SUM(CASE WHEN action = 'REJECT' THEN 1 ELSE 0 END) AS DOUBLE) / COUNT(*) as reject_ratio
FROM vpc_flow_logs
WHERE date_partition >= date_format(current_date - interval '1' day, '%Y/%m/%d')
GROUP BY date_trunc('minute', from_unixtime(start))
HAVING SUM(CASE WHEN action = 'REJECT' THEN 1 ELSE 0 END) > 100
ORDER BY reject_ratio DESC;
---
Security Group and NACL Audit
#!/bin/bash
# sg-nacl-audit.sh - Find potential packet-dropping rules
set -euo pipefail
VPC_ID=$1
echo "=== Security Group Audit for $VPC_ID ==="
# Find security groups with no inbound rules (may cause return traffic drops)
aws ec2 describe-security-groups \
--filters "Name=vpc-id,Values=$VPC_ID" \
--query 'SecurityGroups[?length(IpPermissions)==0].{ID:GroupId,Name:GroupName}' \
--output table
echo ""
echo "=== NACL Rules (check for asymmetric denies) ==="
# List NACLs with explicit deny rules
aws ec2 describe-network-acls \
--filters "Name=vpc-id,Values=$VPC_ID" \
--query 'NetworkAcls[].{ID:NetworkAclId,Entries:Entries[?RuleAction==deny]}' \
--output json | jq '.[] | select(.Entries | length > 0)'
echo ""
echo "=== Checking for ephemeral port NACL issues ==="
# NACLs are stateless - return traffic on ephemeral ports must be explicitly allowed
aws ec2 describe-network-acls \
--filters "Name=vpc-id,Values=$VPC_ID" \
--query 'NetworkAcls[].Entries[?PortRange.From>=1024 && PortRange.To<=65535]'
---
MTU Issues and Path MTU Discovery
# Test for MTU black holes (common with VPN/transit gateway)
# If packets larger than path MTU are dropped silently:
# Check current MTU
ip link show eth0 | grep mtu
# Test with different packet sizes (1500 is standard, 9001 for jumbo)
ping -M do -s 1472 target-host # 1472 + 28 bytes header = 1500
ping -M do -s 8972 target-host # For jumbo frame testing
# Fix: Set MTU to match the path
sudo ip link set dev eth0 mtu 1500
# For Kubernetes pods behind a VPN/Transit Gateway:
# Set pod MTU in CNI configuration
# Cilium MTU configuration
apiVersion: v1
kind: ConfigMap
metadata:
name: cilium-config
namespace: kube-system
data:
mtu: "1400" # Reduce for encapsulation overhead (VXLAN: -50, WireGuard: -60)
enable-pmtu-discovery: "true"
---
Conntrack Table Overflow Detection
#!/bin/bash
# conntrack-monitor.sh - Detect and alert on conntrack table pressure
set -euo pipefail
CURRENT=$(cat /proc/sys/net/netfilter/nf_conntrack_count)
MAX=$(cat /proc/sys/net/netfilter/nf_conntrack_max)
USAGE_PCT=$((CURRENT * 100 / MAX))
echo "Conntrack usage: $CURRENT / $MAX ($USAGE_PCT%)"
if [ $USAGE_PCT -gt 80 ]; then
echo "WARNING: Conntrack table at ${USAGE_PCT}% capacity"
echo "Dropped packets (conntrack full):"
dmesg | grep "nf_conntrack: table full" | tail -5
# Show top consumers
echo ""
echo "Top 10 conntrack consumers by destination:"
conntrack -L 2>/dev/null | \
awk '{for(i=1;i<=NF;i++) if($i ~ /dst=/) print $i}' | \
sort | uniq -c | sort -rn | head -10
fi
# Prometheus metric export
echo "# HELP conntrack_usage_ratio Current conntrack table usage"
echo "# TYPE conntrack_usage_ratio gauge"
echo "conntrack_usage_ratio $USAGE_PCT"
# Kubernetes DaemonSet for conntrack monitoring
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: conntrack-monitor
namespace: monitoring
spec:
selector:
matchLabels:
app: conntrack-monitor
template:
metadata:
labels:
app: conntrack-monitor
spec:
hostNetwork: true
containers:
- name: monitor
image: alpine:3.19
command: ["/bin/sh", "-c"]
args:
- |
while true; do
CURRENT=$(cat /proc/sys/net/netfilter/nf_conntrack_count)
MAX=$(cat /proc/sys/net/netfilter/nf_conntrack_max)
echo "conntrack_entries $CURRENT"
echo "conntrack_max $MAX"
sleep 15
done
securityContext:
privileged: true
volumeMounts:
- name: proc
mountPath: /proc
readOnly: true
volumes:
- name: proc
hostPath:
path: /proc
---
eBPF Packet Tracing with Cilium Hubble
# Install Hubble CLI
export HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)
curl -L --remote-name-all https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-amd64.tar.gz
tar xzvf hubble-linux-amd64.tar.gz
sudo mv hubble /usr/local/bin/
# Enable Hubble in Cilium
cilium hubble enable --ui
# Observe dropped packets in real-time
hubble observe --verdict DROPPED --namespace production
# Filter by specific pod or service
hubble observe --to-pod production/api-server --verdict DROPPED
# Trace a specific connection
hubble observe \
--from-pod production/frontend \
--to-pod production/api-server \
--protocol TCP \
--port 8080
# Export flow data for analysis
hubble observe --verdict DROPPED --output json > dropped-packets.json
---
TCP Retransmission Monitoring
# Real-time retransmission monitoring
ss -ti | awk '/retrans/{print}'
# Watch for retransmission spikes
watch -n 1 'netstat -s | grep -E "retrans|timeout"'
# Prometheus node_exporter metrics to monitor:
# node_netstat_Tcp_RetransSegs - total retransmissions
# node_netstat_Tcp_InErrs - incoming errors
# node_netstat_TcpExt_TCPTimeouts - TCP timeouts
# Prometheus alerting rule for packet loss
groups:
- name: network-packet-loss
rules:
- alert: HighTCPRetransmissions
expr: rate(node_netstat_Tcp_RetransSegs[5m]) > 100
for: 5m
labels:
severity: warning
annotations:
summary: "High TCP retransmission rate on {{ $labels.instance }}"
description: "{{ $value }} retransmissions/sec detected"
- alert: ConntrackTableNearFull
expr: node_nf_conntrack_entries / node_nf_conntrack_entries_limit > 0.8
for: 2m
labels:
severity: critical
annotations:
summary: "Conntrack table at {{ $value | humanizePercentage }} on {{ $labels.instance }}"
---
FAQ
How do I distinguish between network issues and application issues?
If TCP retransmissions are high but the application shows no errors, it is a network issue. Use ss -ti to check per-connection retransmission stats. If specific destination IPs show high retrans but others are fine, the problem is in the path to those IPs (security groups, NACLs, or intermediate network devices).
Can VPC Flow Logs show individual dropped packets?
No. Flow Logs aggregate flows, not individual packets. You see that traffic was REJECTED but not which specific packets were dropped within an ACCEPTED flow. For packet-level visibility, you need eBPF tools (Cilium Hubble) or tcpdump on the host.
Why do I see packet loss only during peak hours?
Likely conntrack table overflow or bandwidth limits. The conntrack table has a fixed size and fills up during traffic spikes. Also check if you are hitting instance network bandwidth limits (each EC2 instance type has a maximum network throughput). CloudWatch NetworkPacketsIn/Out metrics can confirm this.
How do I fix MTU-related packet loss with Kubernetes?
Set the pod network MTU lower than the host MTU to account for encapsulation overhead. For VXLAN overlay networks subtract 50 bytes, for WireGuard subtract 60 bytes. In Cilium config set mtu: "1400". For AWS EKS with VPC CNI, the MTU is usually handled automatically but cross-VPC traffic via Transit Gateway may need adjustment.
---