TL;DR Quick Fix
Deploy a minimal air-gapped logging stack with Vector (collector) + Loki (storage) + Grafana (query):
# Deploy the logging stack via Helm (pre-downloaded charts)
helm install loki ./charts/loki-stack \
--namespace logging --create-namespace \
--set loki.storage.type=s3 \
--set loki.storage.s3.endpoint=http://minio.storage:9000 \
--set loki.storage.s3.bucketnames=loki-logs \
--set loki.storage.s3.access_key_id=minioadmin \
--set loki.storage.s3.secret_access_key=minioadmin
# Deploy Vector as a DaemonSet for log collection
helm install vector ./charts/vector \
--namespace logging \
--set role=Agent \
--set sinks.loki.type=loki \
--set sinks.loki.endpoint=http://loki.logging:3100
For immediate log verification:
# Query recent logs via Grafana's Loki API
curl -s "http://grafana.logging:3000/loki/api/v1/query_range" \
--data-urlencode 'query={namespace="production"}' \
--data-urlencode "start=$(date -d '1 hour ago' +%s)000000000" \
--data-urlencode "end=$(date +%s)000000000" | jq '.data.result[0].values[:5]'
---
Architecture Overview
---
Why Air-Gapped Environments Need Special Treatment
Air-gapped environments cannot use:
- Datadog (requires internet for agent communication)
- Splunk Cloud (SaaS only, no on-prem logs ship)
- CloudWatch (AWS-only, needs internet egress)
- Elastic Cloud (SaaS variant needs connectivity)
Common in: defense/government, healthcare (HIPAA), financial services, critical infrastructure, and classified environments.
---
Vector Configuration for Log Collection
# vector-agent-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: vector-config
namespace: logging
data:
vector.yaml: |
sources:
kubernetes_logs:
type: kubernetes_logs
auto_partial_merge: true
pod_annotation_fields:
pod_labels: "pod_labels"
pod_namespace: "namespace"
pod_name: "pod"
journal_logs:
type: journald
include_units:
- kubelet
- containerd
- docker
transforms:
parse_json:
type: remap
inputs: ["kubernetes_logs"]
source: |
. = parse_json!(.message) ?? .
.timestamp = now()
.cluster = "airgapped-prod-01"
filter_noise:
type: filter
inputs: ["parse_json"]
condition:
type: vrl
source: |
!includes(["kube-probe/", "healthz"], to_string(.path) ?? "")
add_metadata:
type: remap
inputs: ["filter_noise"]
source: |
.environment = "production"
.retention_class = if includes(["audit", "security"], to_string(.namespace) ?? "") {
"long"
} else {
"standard"
}
sinks:
loki:
type: loki
inputs: ["add_metadata"]
endpoint: http://loki-gateway.logging.svc:3100
encoding:
codec: json
labels:
namespace: "{{ namespace }}"
pod: "{{ pod }}"
cluster: "{{ cluster }}"
retention_class: "{{ retention_class }}"
buffer:
type: disk
max_size: 5368709120 # 5GB disk buffer for resilience
when_full: block
batch:
max_bytes: 2097152
timeout_secs: 5
---
Loki Deployment for Air-Gapped Storage
# loki-values.yaml (Helm)
loki:
auth_enabled: true
schemaConfig:
configs:
- from: "2024-01-01"
store: tsdb
object_store: s3
schema: v13
index:
prefix: loki_index_
period: 24h
storage:
type: s3
s3:
endpoint: http://minio.storage.svc:9000
bucketnames: loki-chunks
access_key_id: ${MINIO_ACCESS_KEY}
secret_access_key: ${MINIO_SECRET_KEY}
insecure: true
s3ForcePathStyle: true
limits_config:
retention_period: 90d
ingestion_rate_mb: 20
ingestion_burst_size_mb: 30
max_query_parallelism: 32
max_query_series: 5000
compactor:
retention_enabled: true
delete_request_store: s3
working_directory: /loki/compactor
# Deploy in microservices mode for scale
deploymentMode: SimpleScalable
read:
replicas: 3
write:
replicas: 3
backend:
replicas: 2
---
MinIO for S3-Compatible Object Storage
# minio-values.yaml
mode: distributed
replicas: 4
persistence:
enabled: true
size: 500Gi
storageClass: local-ssd
resources:
requests:
memory: 4Gi
cpu: 2
limits:
memory: 8Gi
cpu: 4
buckets:
- name: loki-chunks
policy: none
purge: false
- name: loki-ruler
policy: none
purge: false
environment:
MINIO_STORAGE_CLASS_STANDARD: EC:2
MINIO_COMPRESSION_ENABLE: "on"
MINIO_COMPRESSION_EXTENSIONS: ".txt,.log,.json"
# Lifecycle rules for storage management
lifecycle:
rules:
- id: expire-old-chunks
prefix: "loki-chunks/"
expiration:
days: 95 # Slightly beyond Loki retention for safety
status: Enabled
# Initialize MinIO buckets (run once during setup)
mc alias set airgapped http://minio.storage:9000 minioadmin minioadmin
mc mb airgapped/loki-chunks
mc mb airgapped/loki-ruler
mc ilm set --expire-days 95 airgapped/loki-chunks
---
Grafana Log Querying
# grafana-datasource.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: grafana-datasources
namespace: logging
data:
loki.yaml: |
apiVersion: 1
datasources:
- name: Loki
type: loki
url: http://loki-gateway.logging.svc:3100
access: proxy
isDefault: true
jsonData:
maxLines: 5000
timeout: 60
Common LogQL Queries
# Find errors across all production pods
{namespace="production"} |= "error" | json | level="error"
# Count errors by service over time
sum by (pod) (count_over_time({namespace="production"} |= "error" [5m]))
# Search for specific request IDs across services
{cluster="airgapped-prod-01"} |= "req-id-abc123"
# Parse and filter structured logs
{namespace="production"} | json | status >= 500 | line_format "{{.method}} {{.path}} {{.status}}"
# Detect log volume anomalies (sudden spike or drop)
sum(rate({namespace="production"}[5m])) by (pod)
---
Log Shipping Without Internet
#!/bin/bash
# offline-log-export.sh - Export logs for external analysis
set -euo pipefail
EXPORT_DATE=${1:-$(date -d "yesterday" +%Y-%m-%d)}
EXPORT_DIR="/mnt/export/logs/$EXPORT_DATE"
mkdir -p "$EXPORT_DIR"
echo "Exporting logs for $EXPORT_DATE..."
# Export via Loki API
curl -s "http://loki.logging:3100/loki/api/v1/query_range" \
--data-urlencode "query={namespace=~\".+\"}" \
--data-urlencode "start=$(date -d "$EXPORT_DATE 00:00:00" +%s)000000000" \
--data-urlencode "end=$(date -d "$EXPORT_DATE 23:59:59" +%s)000000000" \
--data-urlencode "limit=100000" | \
gzip > "$EXPORT_DIR/all-logs.json.gz"
# Create checksums for integrity verification
sha256sum "$EXPORT_DIR"/* > "$EXPORT_DIR/checksums.sha256"
echo "Export complete: $EXPORT_DIR"
echo "Size: $(du -sh "$EXPORT_DIR" | awk '{print $1}')"
# Transfer to removable media for cross-domain analysis
# (manual step for air-gapped environments)
---
Security Hardening for the Logging Stack
# network-policy.yaml - Restrict logging component communication
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: loki-ingress
namespace: logging
spec:
podSelector:
matchLabels:
app: loki
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
app: vector
- podSelector:
matchLabels:
app: grafana
ports:
- port: 3100
protocol: TCP
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: minio-ingress
namespace: storage
spec:
podSelector:
matchLabels:
app: minio
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
name: logging
ports:
- port: 9000
protocol: TCP
# Enable mTLS between Vector and Loki
# Generate certificates (using cert-manager or manual CA)
openssl req -x509 -newkey rsa:4096 -keyout loki-tls.key -out loki-tls.crt \
-days 365 -nodes -subj "/CN=loki.logging.svc"
kubectl create secret tls loki-tls \
--cert=loki-tls.crt --key=loki-tls.key \
-n logging
---
FAQ
How much storage do I need for the logging stack?
Estimate based on log volume: if your cluster produces 10GB/day of raw logs, Loki compresses this to roughly 1-2GB/day stored in MinIO. For 90-day retention, plan for 90-180GB of object storage per 10GB/day of ingestion. Add 30% overhead for indices and temporary files.
Can I use Elasticsearch instead of Loki in air-gapped environments?
Yes, but Loki is significantly more resource-efficient for log storage. Elasticsearch requires more memory (JVM heap), more storage (full-text indexing), and more operational overhead. Loki only indexes labels, not log content, which reduces storage by 10-50x for typical workloads.
How do I handle log rotation and prevent disk exhaustion?
Vector's disk buffer has a configurable max size and will block ingestion when full (preventing OOM). Loki's compactor handles chunk expiration based on retention rules. MinIO lifecycle policies provide a final safety net. Monitor disk usage with Prometheus alerts at 70% and 85% thresholds.
What about audit log integrity for compliance?
Use immutable storage in MinIO (object locking / WORM mode) for audit logs. Configure a separate Loki tenant for audit logs with longer retention. Generate daily checksums and store them separately. For FIPS compliance, configure MinIO with FIPS-validated encryption.
How do I upgrade components in an air-gapped environment?
Pre-download Helm charts and container images on a connected machine, transfer them to the air-gapped registry via removable media or a data diode. Use a private container registry (Harbor) inside the air-gapped network. Version-pin everything and test upgrades in a staging air-gap first.
---