TL;DR Quick Fix
Stop paying for dev environments that run 24/7 when nobody uses them on nights and weekends:
# Install kube-downscaler for Kubernetes environments
helm install kube-downscaler \
oci://ghcr.io/caas-team/charts/kube-downscaler \
--namespace kube-system \
--set schedule="Mon-Fri 08:00-18:00 US/Eastern"
# Annotate namespaces for automatic shutdown
kubectl annotate namespace dev \
downscaler/downtime="Mon-Fri 18:00-08:00 US/Eastern,Sat-Sun 00:00-24:00 US/Eastern"
Most teams waste 65-70% of their dev/staging budget on environments running during off-hours. This guide shows you how to automate shutdowns without disrupting developer workflows.
---
Architecture Overview
---
The Cost of Idle Environments
# Calculate your idle environment waste
# Typical dev environment running 24/7:
# EC2 (3x m5.xlarge): $0.192/hr 3 730hr = $420/mo
# RDS (db.r5.large): $0.24/hr * 730hr = $175/mo
# EKS node group: $0.10/hr 3 730hr = $219/mo
# Total: $814/mo per environment
#
# With scheduling (10hr/day * 22 workdays = 220hr):
# Total: $245/mo (70% savings!)
# Quick check: what are you actually spending on non-prod?
aws ce get-cost-and-usage \
--time-period Start=2024-01-01,End=2024-02-01 \
--granularity MONTHLY \
--filter '{
"Tags": {
"Key": "Environment",
"Values": ["dev", "staging", "qa"]
}
}' \
--metrics "UnblendedCost" \
--group-by Type=TAG,Key=Environment
---
kube-downscaler for Kubernetes
Installation and Configuration
# kube-downscaler-values.yaml
image:
repository: ghcr.io/caas-team/kube-downscaler
tag: "23.2.0"
schedule:
default: "Mon-Fri 08:00-18:00 US/Eastern"
resources:
requests:
cpu: 50m
memory: 64Mi
limits:
cpu: 200m
memory: 128Mi
# Namespaces to exclude from downscaling
excludedNamespaces:
- kube-system
- monitoring
- istio-system
# Install with Helm
helm install kube-downscaler \
oci://ghcr.io/caas-team/charts/kube-downscaler \
--namespace kube-system \
-f kube-downscaler-values.yaml
# Annotate namespaces for custom schedules
kubectl annotate namespace dev \
downscaler/downtime="Mon-Fri 18:00-08:00 US/Eastern,Sat-Sun 00:00-24:00 US/Eastern"
# Force a namespace to stay up (for active testing)
kubectl annotate namespace staging \
downscaler/exclude="true"
# Set specific deployments to stay running
kubectl annotate deployment critical-service -n dev \
downscaler/exclude-until="2024-03-15"
---
AWS Instance Scheduler with Lambda
Lambda Function for EC2/RDS Scheduling
#!/usr/bin/env python3
# env_scheduler.py - Start/Stop non-production environments
import boto3
import json
def lambda_handler(event, context):
action = event.get('action', 'stop') # 'start' or 'stop'
tag_key = 'Schedule'
tag_value = 'office-hours'
ec2 = boto3.client('ec2')
rds = boto3.client('rds')
# Handle EC2 instances
filters = [{'Name': f'tag:{tag_key}', 'Values': [tag_value]}]
if action == 'stop':
instances = ec2.describe_instances(
Filters=filters + [{'Name': 'instance-state-name', 'Values': ['running']}]
)
instance_ids = [
i['InstanceId']
for r in instances['Reservations']
for i in r['Instances']
]
if instance_ids:
ec2.stop_instances(InstanceIds=instance_ids)
print(f"Stopped EC2: {instance_ids}")
# Stop RDS instances
rds_instances = rds.describe_db_instances()
for db in rds_instances['DBInstances']:
tags = rds.list_tags_for_resource(
ResourceName=db['DBInstanceArn']
)['TagList']
if any(t['Key'] == tag_key and t['Value'] == tag_value for t in tags):
if db['DBInstanceStatus'] == 'available':
rds.stop_db_instance(DBInstanceIdentifier=db['DBInstanceIdentifier'])
print(f"Stopped RDS: {db['DBInstanceIdentifier']}")
elif action == 'start':
instances = ec2.describe_instances(
Filters=filters + [{'Name': 'instance-state-name', 'Values': ['stopped']}]
)
instance_ids = [
i['InstanceId']
for r in instances['Reservations']
for i in r['Instances']
]
if instance_ids:
ec2.start_instances(InstanceIds=instance_ids)
print(f"Started EC2: {instance_ids}")
return {'statusCode': 200, 'body': json.dumps(f'{action} completed')}
EventBridge Scheduler Rules
# terraform/scheduler.tf
resource "aws_scheduler_schedule" "start_dev" {
name = "start-dev-environments"
group_name = "default"
flexible_time_window {
mode = "OFF"
}
schedule_expression = "cron(0 8 ? MON-FRI )"
schedule_expression_timezone = "US/Eastern"
target {
arn = aws_lambda_function.env_scheduler.arn
role_arn = aws_iam_role.scheduler_role.arn
input = jsonencode({
action = "start"
})
}
}
resource "aws_scheduler_schedule" "stop_dev" {
name = "stop-dev-environments"
group_name = "default"
flexible_time_window {
mode = "OFF"
}
schedule_expression = "cron(0 18 ? MON-FRI )"
schedule_expression_timezone = "US/Eastern"
target {
arn = aws_lambda_function.env_scheduler.arn
role_arn = aws_iam_role.scheduler_role.arn
input = jsonencode({
action = "stop"
})
}
}
---
EKS Node Group Scaling
#!/bin/bash
# scale-eks-nodegroup.sh - Scale EKS node groups to zero
set -euo pipefail
CLUSTER_NAME="dev-cluster"
ACTION=${1:-"down"} # "up" or "down"
if [ "$ACTION" == "down" ]; then
echo "Scaling down dev EKS node groups..."
aws eks update-nodegroup-config \
--cluster-name $CLUSTER_NAME \
--nodegroup-name workers-dev \
--scaling-config minSize=0,maxSize=3,desiredSize=0
elif [ "$ACTION" == "up" ]; then
echo "Scaling up dev EKS node groups..."
aws eks update-nodegroup-config \
--cluster-name $CLUSTER_NAME \
--nodegroup-name workers-dev \
--scaling-config minSize=2,maxSize=6,desiredSize=3
fi
---
Terraform Workspace Destroy Schedules
# terraform/ephemeral-env.tf
resource "null_resource" "auto_destroy_timer" {
triggers = {
destroy_after = timeadd(timestamp(), "8h")
}
provisioner "local-exec" {
command = <<-EOT
echo "Environment will auto-destroy at ${self.triggers.destroy_after}"
# Schedule destruction via CI pipeline
gh workflow run destroy-env.yml \
--field environment=pr-${var.pr_number} \
--field scheduled_at=${self.triggers.destroy_after}
EOT
}
}
# .github/workflows/destroy-env.yml
name: Destroy Ephemeral Environment
on:
workflow_dispatch:
inputs:
environment:
required: true
scheduled_at:
required: true
schedule:
- cron: '0 2 *' # Check nightly for expired envs
jobs:
cleanup:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Destroy expired environments
run: |
cd terraform/environments/${{ inputs.environment }}
terraform destroy -auto-approve
---
Slack Bot for On-Demand Spin-Up
#!/usr/bin/env python3
# slack_env_bot.py - On-demand environment control via Slack
from slack_bolt import App
import boto3
import subprocess
app = App(token="xoxb-your-token", signing_secret="your-secret")
@app.command("/env-start")
def start_environment(ack, respond, command):
ack()
env_name = command['text'] or 'dev'
user = command['user_name']
respond(f"Starting {env_name} environment for @{user}...")
# Start the environment
subprocess.run([
'aws', 'lambda', 'invoke',
'--function-name', 'env-scheduler',
'--payload', f'{{"action": "start", "env": "{env_name}"}}',
'/tmp/response.json'
])
respond(f"{env_name} environment is starting up. ETA: 3-5 minutes.")
@app.command("/env-stop")
def stop_environment(ack, respond, command):
ack()
env_name = command['text'] or 'dev'
respond(f"Shutting down {env_name} environment...")
subprocess.run([
'aws', 'lambda', 'invoke',
'--function-name', 'env-scheduler',
'--payload', f'{{"action": "stop", "env": "{env_name}"}}',
'/tmp/response.json'
])
respond(f"{env_name} environment shut down successfully.")
if __name__ == "__main__":
app.start(port=3000)
---
FAQ
How much can I realistically save with environment scheduling?
If your dev/staging environments run 24/7 but are only used during business hours (10hr/day, 5 days/week), you can save approximately 70% by shutting them down during off-hours. For a team with 3 environments costing $800/mo each, that is roughly $1,680/mo in savings.
What about database state when environments shut down?
RDS instances retain all data when stopped. For Kubernetes workloads using EBS-backed PVCs, the data persists even when pods scale to zero. The only risk is with in-memory caches (Redis, Memcached) which will be cold on restart. Plan for cache warming in your startup scripts.
How do I handle environments that need to stay up for overnight tests?
Use annotations or tags to temporarily exclude environments from shutdown. With kube-downscaler, annotate with downscaler/exclude-until="2024-03-15T08:00:00Z". For AWS, add a Schedule=always-on tag temporarily and remove it when tests complete.
What if a developer needs the environment outside business hours?
Implement an on-demand spin-up mechanism via Slack bot, CLI tool, or simple web UI. The startup time (3-5 minutes for most environments) is acceptable for occasional off-hours work. Some teams also allow developers to extend the shutdown window via a Slack command.
---