Fixing Flaky Integration Tests in CI/CD Once and For All

TL;DR — Quick Fix

# GitHub Actions — quarantine flaky tests and retry
name: Run tests with retry
  uses: nick-fields/retry@v3

  with:
    max_attempts: 3
    timeout_minutes: 10
    command: npm run test:integration -- --bail --forceExit

But retries are a band-aid. This guide shows you how to fix the root causes.

---

Why Tests Are Flaky (The Real Reasons)

A test is "flaky" when it passes sometimes and fails other times with no code changes. This destroys developer trust in CI/CD — teams start ignoring red builds because "it's probably just flaky."

Step 1: Identify and Track Flaky Tests

Detect Flakiness Automatically

# GitHub Actions — weekly flake detection job
name: Flake Detection
on:
  schedule:
    - cron: '0 3   0'

jobs:
  detect-flakes:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - name: Run tests 5 times
        run: |
          for i in {1..5}; do
            echo "=== Run $i ==="
            npm run test:integration 2>&1 | tee -a results.txt || true
          done
      - name: Analyze results
        run: |
          grep -E "(PASS|FAIL)" results.txt | sort | uniq -c | sort -rn

Step 2: Fix Shared State (40% of Flakes)

The most common cause: Test A modifies a database/cache, and Test B reads stale data.

Database Isolation

// Each test gets a clean database state
beforeEach(async () => {
  // Option 1: Truncate all tables
  await db.query('TRUNCATE TABLE users, orders, sessions CASCADE');
  
  // Option 2: Use transactions that rollback
  transaction = await db.beginTransaction();
});

afterEach(async () => {
  await transaction.rollback();
});

Docker Compose for Test Isolation

# docker-compose.test.yml
version: '3.8'
services:
  postgres:
    image: postgres:16-alpine
    environment:
      POSTGRES_DB: test_db
      POSTGRES_USER: test
      POSTGRES_PASSWORD: test
    tmpfs:
      - /var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U test"]
      interval: 1s
      timeout: 5s
      retries: 10

  redis:
    image: redis:7-alpine
    tmpfs:
      - /data

  tests:
    build:
      context: .
      dockerfile: Dockerfile.test
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_started
    environment:
      DATABASE_URL: postgres://test:test@postgres:5432/test_db
      REDIS_URL: redis://redis:6379

Step 3: Fix Timing Issues (25% of Flakes)

Replace Sleep with Polling

// Before: Fragile
await new Promise(resolve => setTimeout(resolve, 2000));
expect(await getStatus()).toBe('ready');

// After: Deterministic — waits until condition is met
async function waitFor(
  condition: () => Promise<boolean>,
  timeoutMs = 10000,
  intervalMs = 100
): Promise<void> {
  const start = Date.now();
  while (Date.now() - start < timeoutMs) {
    if (await condition()) return;
    await new Promise(r => setTimeout(r, intervalMs));
  }
  throw new Error(Condition not met within ${timeoutMs}ms);
}

await waitFor(async () => (await getStatus()) === 'ready');

Fix Async Event Ordering

// Before: Race condition
service.start();
const result = await waitForEvent(service, 'ready');

// After: Attach listener first, then trigger
const readyPromise = waitForEvent(service, 'ready');
service.start();
const result = await readyPromise;

Use Deterministic Timestamps

beforeEach(() => {
  jest.useFakeTimers();
  jest.setSystemTime(new Date('2026-06-15T12:00:00Z'));
});

afterEach(() => {
  jest.useRealTimers();
});

Step 4: Mock External Services (15% of Flakes)

Use MSW for HTTP Dependencies

import { setupServer } from 'msw/node';
import { http, HttpResponse } from 'msw';

const server = setupServer(
  http.get('https://api.stripe.com/v1/charges', () => {
    return HttpResponse.json({
      data: [{ id: 'ch_123', amount: 2000, status: 'succeeded' }]
    });
  }),

  http.post('https://api.sendgrid.com/v3/mail/send', () => {
    return HttpResponse.json({ message: 'success' }, { status: 202 });
  })
);

beforeAll(() => server.listen());
afterEach(() => server.resetHandlers());
afterAll(() => server.close());

Step 5: Quarantine and Retry Strategy

Quarantine Workflow

name: CI
on: [pull_request]

jobs:
  reliable-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - name: Run reliable test suite
        run: npm run test -- --testPathIgnorePatterns='quarantine'

  quarantined-tests:
    runs-on: ubuntu-latest
    continue-on-error: true
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - name: Run quarantined tests
        run: npm run test -- --testPathPattern='quarantine'

Smart Retry (Not Blind Retry)

- name: Run integration tests
  id: tests
  run: npm run test:integration 2>&1 | tee test-output.log
  continue-on-error: true

name: Retry on known flake patterns
  if: steps.tests.outcome == 'failure'

  run: |
    if grep -q "ECONNRESET\|ETIMEDOUT\|socket hang up" test-output.log; then
      echo "Known transient error — retrying"
      npm run test:integration
    else
      echo "Genuine test failure — not retrying"
      exit 1
    fi

Step 6: Ephemeral Test Environments

# Fresh Kubernetes namespace per PR
name: Create test namespace
  run: |

    NAMESPACE="test-pr-${{ github.event.pull_request.number }}"
    kubectl create namespace $NAMESPACE
    helm install test-env ./chart \
      --namespace $NAMESPACE \
      --set image.tag=${{ github.sha }} \
      --wait --timeout=5m

name: Run integration tests
  run: |

    NAMESPACE="test-pr-${{ github.event.pull_request.number }}"
    kubectl run test-runner \
      --namespace $NAMESPACE \
      --image=test-image:${{ github.sha }} \
      --restart=Never \
      --wait \
      -- npm run test:integration

name: Cleanup
  if: always()

  run: |
    NAMESPACE="test-pr-${{ github.event.pull_request.number }}"
    kubectl delete namespace $NAMESPACE --ignore-not-found

Prevention: Test Design Principles

---

Frequently Asked Questions

What percentage of flaky tests is acceptable?

Industry benchmark: less than 0.5% flake rate. Google targets less than 1.5% of test runs being flaky. If more than 5% of your CI runs fail due to flakes, developers will stop trusting the pipeline. Track your flake rate weekly and set a team goal to reduce it.

Should I use retries to handle flaky tests?

Retries are acceptable as a short-term mitigation while you fix root causes. But every retried test should be logged and tracked. If a test needs retries consistently, it should be quarantined and fixed, not permanently retried.

How do I convince my team to invest in fixing flaky tests?

Measure the cost: track CI minutes wasted on flakes, how often developers re-run pipelines, and time spent investigating false failures. A typical team wastes 2-4 hours per developer per week on flaky test investigation.

What's the difference between unit test and integration test flakiness?

Unit tests should never be flaky — if they are, it's a design flaw. Integration tests are inherently more prone to flakiness because they involve real services and networks. The solution is better isolation (containers, ephemeral environments) and proper wait/retry patterns.

How do I handle tests that depend on external APIs?

Never call real external APIs in CI. Use contract testing (Pact) to verify API shape, mock servers (WireMock, MSW) for integration tests, and recorded HTTP fixtures for deterministic responses. Reserve real API calls for staging smoke tests only.

---

Related Resources

• GitHub Actions CI/CD Complete Guide — Pipeline configuration
• GitHub Actions Cache Optimization — Speed up test workflows
• Docker Compose Environment Variables — Test environment configuration