Our CI pipeline was broken 30% of the time. Not broken in obvious ways — broken in the worst way: randomly. Tests that passed locally, failed on CI. A re-run would make them green. Sound familiar?
This is the story of how we diagnosed and fixed it, and the mental model I now use for every CI setup.
The Three Categories of Flakiness
After logging every failure for two weeks, every flaky test fell into one of three buckets:
1. Order-dependent tests — Tests that silently relied on shared state left behind by a previous test. Passing order was coincidental in local dev but different in CI's parallel runner.
2. Timing issues — await asyncio.sleep(0.1) that worked on a beefy dev machine but not on a 2-core GitHub-hosted runner under load.
3. Environment drift — The CI container had a slightly different version of a transitive dependency. Not pinned, not noticed, silently causing edge-case failures.
Diagnosing Order-Dependent Tests
The trick I use: run your test suite in random order, then in reverse order. If failures appear that weren't there before, you have ordering dependencies.
# pytest with randomized order
- name: Run tests (randomized)
run: pytest --randomly-seed=random --randomly-dont-shuffle-module
For the failures we found, the fix was almost always: move database seeding into fixtures with proper teardown, not into module-level setup.
# Bad — shared state leaks between tests
_test_user = create_user("test@test.com")
# Good — isolated per test
@pytest.fixture
def test_user(db_session):
user = create_user("test@test.com", db=db_session)
yield user
db_session.rollback()
Fixing Timing Issues
Hard sleeps are a code smell. Replace them with polling:
import asyncio
async def wait_for_condition(fn, timeout=5.0, interval=0.1):
deadline = asyncio.get_event_loop().time() + timeout
while asyncio.get_event_loop().time() < deadline:
if await fn():
return True
await asyncio.sleep(interval)
raise TimeoutError("Condition not met within timeout")
# Usage
await wait_for_condition(lambda: job_queue.is_empty())
On slow CI runners, a job that takes 50ms locally might take 800ms. Polling with a generous timeout is robust; sleep(0.5) is a ticking time bomb.
Locking Dependencies
This one's embarrassing to admit: we weren't pinning transitive dependencies. We had:
fastapi>=0.100.0
sqlalchemy>=2.0.0
...and a transitive dep shipped a patch release that changed default behavior. The fix:
# Pin everything
pip freeze > requirements-lock.txt
# In CI, install from the lock file
pip install -r requirements-lock.txt
For Node projects, commit package-lock.json and use npm ci in CI, never npm install.
The Pipeline Structure That Actually Works
After all the fixes, here's the pipeline structure I now use as a template:
jobs:
test:
runs-on: ubuntu-latest
services:
postgres:
image: postgres:16
env:
POSTGRES_PASSWORD: test
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
cache: "pip"
- run: pip install -r requirements-lock.txt
- run: pytest --randomly-seed=12345 -x --tb=short
Key choices: always use service health checks before tests run, cache pip installs to reduce variance from slow downloads, and fail fast with -x so you see the first failure clearly.
The Biggest Insight
Flaky tests are not a testing problem. They're a design problem. Each one is a symptom of code that has hidden coupling, hidden state, or hidden assumptions about the execution environment.
Fixing flakiness isn't about retrying — it's about finding and eliminating those hidden dependencies. Retries mask the symptom. Good fixtures, locked deps, and explicit waits cure the disease.