TL;DR — Quick Fix
Replace permanent SSH keys with time-bounded Teleport access:
# Request temporary production access (requires approval)
tsh request create --roles=prod-admin --reason="Investigating P1 incident INC-4521"
# Once approved, connect with full audit trail
tsh ssh --cluster=production user@web-server-01
# For Kubernetes access
tsh kube login production-cluster
kubectl get pods -n critical-service
# Access expires after 1 hour automatically
tsh request ls # See active requests and TTL
---
Risks of Permanent Admin Access
Standing privileges are the top source of insider threats and credential compromise.
---
Teleport Setup for SSH, K8s, and Databases
# teleport-values.yaml — Helm configuration
teleport:
auth:
type: kubernetes
proxy:
publicAddr: ["teleport.example.com:443"]
roles:
- metadata:
name: prod-readonly
spec:
allow:
logins: ["readonly"]
kubernetes_groups: ["view"]
node_labels:
env: production
options:
max_session_ttl: 1h
request_access: always
- metadata:
name: prod-admin
spec:
allow:
logins: ["admin", "root"]
kubernetes_groups: ["system:masters"]
node_labels:
env: production
options:
max_session_ttl: 4h
request_access: always
require_session_mfa: true
# Deploy Teleport to Kubernetes
helm repo add teleport https://charts.releases.teleport.dev
helm install teleport-cluster teleport/teleport-cluster \
--namespace teleport --create-namespace \
-f teleport-values.yaml
# Register a Kubernetes cluster
tctl create -f << 'EOF'
kind: kube_cluster
version: v3
metadata:
name: production-cluster
spec:
aws:
region: us-east-1
account_id: "123456789012"
name: production-eks
EOF
# Register database access
tctl create -f << 'EOF'
kind: db
version: v3
metadata:
name: production-postgres
spec:
protocol: postgres
uri: prod-db.cluster-xxx.us-east-1.rds.amazonaws.com:5432
EOF
---
AWS IAM Identity Center with Approval Workflows
# iam-identity-center.tf
resource "aws_ssoadmin_permission_set" "prod_admin" {
name = "ProductionAdmin"
instance_arn = data.aws_ssoadmin_instances.main.arns[0]
session_duration = "PT1H" # 1 hour max
tags = {
access_type = "jit"
requires_approval = "true"
}
}
resource "aws_ssoadmin_permission_set_inline_policy" "prod_scoped" {
instance_arn = data.aws_ssoadmin_instances.main.arns[0]
permission_set_arn = aws_ssoadmin_permission_set.prod_admin.arn
inline_policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Effect = "Allow"
Action = ["ecs:", "eks:", "logs:", "cloudwatch:"]
Resource = "*"
Condition = {
StringEquals = {
"aws:RequestedRegion" = ["us-east-1", "us-west-2"]
}
}
}
]
})
}
---
Break-Glass Procedures
# break-glass-role.yaml — Emergency access without approval
kind: role
version: v7
metadata:
name: break-glass
spec:
allow:
logins: ["root", "admin"]
kubernetes_groups: ["system:masters"]
node_labels:
env: ["production", "staging"]
options:
max_session_ttl: 2h
request_access: optional
require_session_mfa: true
deny:
node_labels:
role: vault
#!/bin/bash
# scripts/break-glass.sh — Emergency access activation
set -euo pipefail
ENGINEER=$1
INCIDENT=$2
echo "BREAK-GLASS ACTIVATED by $ENGINEER for $INCIDENT at $(date -u)"
# Grant temporary break-glass role
tctl users update "$ENGINEER" --set-roles=break-glass,engineer
# Alert security team
curl -X POST "$SLACK_SECURITY_WEBHOOK" \
-H 'Content-Type: application/json' \
-d "{\"text\": \"BREAK-GLASS: $ENGINEER activated for $INCIDENT\"}"
echo "Access granted for 2 hours. Session will be recorded."
echo "File post-incident review within 24h."
---
Access Reviews and Audit
# Generate weekly access review report
tsh request ls --format=json | jq '[
.[] | {
user: .user,
roles: .roles,
reason: .request_reason,
created: .created,
state: .state
}
]' > weekly-access-report.json
# Check for anomalous access patterns
tsh events ls --since="7d" --type="session.start" | \
jq 'group_by(.user) | map({user: .[0].user, sessions: length})' | \
jq '.[] | select(.sessions > 20)'
# Kubernetes RBAC for incident responders
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: incident-responder
rules:
- apiGroups: [""]
resources: ["pods", "pods/log", "services"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create"]
- apiGroups: ["apps"]
resources: ["deployments"]
verbs: ["get", "list", "watch", "patch"]
- apiGroups: ["apps"]
resources: ["deployments/rollback"]
verbs: ["create"]
---
FAQ
Q: What if the approver is unavailable during an incident?
A: Implement tiered approval. Try peer approval first. After 5 minutes, escalate to on-call manager. After 10 minutes, allow break-glass self-approval with mandatory post-incident review.
Q: How do I handle service accounts that need permanent access?
A: Use workload identity (IRSA, GKE Workload Identity) for automated systems. These have fixed, auditable scope and don't need JIT. Review permissions quarterly.
Q: Won't JIT access slow down incident response?
A: Typical Slack-based approval takes 30-90 seconds. Break-glass bypasses approval for P1s. The delay is negligible compared to MTTR improvement from better audit trails.
Q: How do I comply with SOC 2 using JIT access?
A: JIT makes SOC 2 easier. Prove: no standing access (CC6.1), time-bounded sessions (CC6.2), full audit trails (CC7.2), approval workflows (CC6.3).
Q: What about database access specifically?
A: Teleport supports native database protocols. Engineers get short-lived credentials via tsh db connect. All queries logged. No shared passwords.
---