CI/CD Pipeline Optimization for Monorepos

A monorepo CI pipeline that rebuilds and retests every package on every commit gets slower with each package you add, until a one-line change waits twenty minutes for green. This guide covers the three levers that keep monorepo CI fast — affected detection, cross-run caching, and parallel sharding — and it extends the Monorepo Architecture & Orchestration section with the CI-side of the task runner you have already chosen.

Concept overview and where it fits

The core insight is that most commits touch a small fraction of the repo, so most of CI's work is recomputing results that have not changed. Three techniques exploit that: affected detection skips packages no changed file can reach, caching replays outputs for inputs that have been built before, and sharding spreads the remaining work across parallel runners. They compose — affected narrows the set, cache replays what it can, and sharding parallelizes the rest.

The problem a monorepo CI pipeline solves badly by default is that its cost scales with the size of the repository instead of the size of the change. Every commit rebuilds and retests every package, so a one-line edit to a leaf library waits behind the whole suite, and the wait grows with every package added. The three levers in this guide attack that from different angles: affected detection cuts the amount of work, caching replays work already done, and sharding parallelizes whatever remains. They are complementary, not alternatives, and a fast pipeline uses all three in that order.

Ordering the levers correctly matters as much as adopting them. Affected detection comes first because it removes work entirely — there is no point caching or parallelizing a package the change cannot reach. Caching comes second, replaying any survivor whose inputs already have a stored result on this or another machine. Sharding comes last, spreading the genuinely-necessary remainder across runners. Applying them out of order — sharding the full suite on every commit, for instance — pays for parallelism on work that affected would have skipped, which is why teams that shard first often see less benefit than they expected.

Initialization: give CI the history it needs

Start by giving CI the history it needs to compute a diff. A shallow clone has no base commit to compare against, so affected detection silently falls back to 'everything changed'. Fetch enough history and pass an explicit base.

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0   # full history so the merge-base exists
      - uses: pnpm/action-setup@v4
        with: { version: 10 }
      - run: pnpm install --frozen-lockfile --ignore-scripts
      - run: pnpm turbo run build test --filter='...[origin/main]'

Affected detection is only as good as the git history the runner can see, which is why the checkout configuration is the first thing to get right. Computing what changed means diffing against a base commit, and a shallow clone frequently lacks that commit, so the runner cannot compute a diff and conservatively falls back to treating everything as affected — silently erasing the speed-up. Fetching full history and passing an explicit base ref is the precondition that makes every downstream optimization possible.

The base ref you compare against shapes what counts as a change. For a pull request, diffing against the merge base with the target branch selects exactly the packages the PR touches; for a push to the main branch, diffing against the previous commit selects what that commit changed. Being explicit about the base — rather than relying on the runner's guess — keeps the affected set correct and reproducible across the different contexts CI runs in.

Beyond fetch depth, the initialization step sets up the caching that makes installs fast, which is a precondition for a fast pipeline even before affected detection and task caching apply. Caching the package-manager store — pnpm's content-addressed store, for instance — keyed on the lockfile hash turns a cold dependency download into a fast linking step on subsequent runs, so the fixed per-job overhead that sharding cannot reduce is itself minimized. A pipeline that skips this pays the full install cost on every job, which often dominates the runtime once the build work is optimized away.

The install step should also be locked down as part of initialization: a frozen lockfile so the resolved graph is exactly the reviewed one, and ignored scripts so a compromised dependency cannot execute during install with the job's credentials. These are not just security measures but reproducibility ones — a frozen, script-free install produces the identical tree every run, which is what makes the downstream cache keys stable across CI and local machines. Getting the initialization right is what lets everything after it be both fast and correct.

Architecture: affected runs on the graph

Affected detection works off the dependency graph, not the file list. A changed file marks its own package dirty, and the tool then walks dependents — every package that imports the dirty one — because their behavior could change too. This is why the graph must be accurate: an undeclared import means a real dependent is missed and shipped untested. The same graph powers pnpm's filter selectors and Nx affected.

Affected detection works on the dependency graph, not on the raw file list, and that distinction is the source of both its power and its one failure mode. A changed file marks its own package dirty, and the runner then walks the graph to include every package that depends on the dirty one, because their behavior could change too — the changed package plus its dependents is the correct set to validate. Testing only the changed package would ship its consumers untested, which is why the dependent traversal is not optional.

The traversal is only as accurate as the declared edges, so an inaccurate graph produces an incorrect affected set. An undeclared cross-package import — a deep relative path into another package's source instead of a workspace dependency — is an edge the runner cannot see, so a real dependent is silently excluded and ships without being tested. This makes an accurate dependency graph a correctness requirement for fast CI, not merely good hygiene: the same graph that makes builds fast is the one that makes them safe.

Because affected is graph-driven, the same investment that keeps it accurate — declared dependencies, enforced module boundaries, no deep cross-package imports — is the investment that keeps the whole monorepo maintainable. A repo whose graph is honest gets fast, correct affected builds almost for free; a repo whose graph is full of undeclared edges gets an affected set you cannot trust, which forces teams back toward full runs to be safe. Fast CI and clean architecture are, at the graph level, the same problem viewed from two angles.

Execution strategy: cache then shard

Layer a remote cache on top so a result computed on one runner is reused on every other, including local developer machines. With remote caching configured, a shard that already built a package downloads the artifact instead of rebuilding it.

      - run: pnpm turbo run build --filter='...[origin/main]'
        env:
          TURBO_TOKEN: ${{ secrets.TURBO_TOKEN }}
          TURBO_TEAM: ${{ vars.TURBO_TEAM }}

The compounding effect of the three levers is easiest to see with a concrete progression. A naive pipeline runs every package on every commit — cost proportional to the repo. Turn on affected, and a one-line change runs only the touched package and its dependents. Add a remote cache, and even those survivors are downloaded rather than rebuilt when their inputs are unchanged, so the first affected build after a dependency bump is fast for everyone who follows. Finally shard the genuinely-new work, and the wall-clock drops toward a fixed floor set by checkout and install. Each lever multiplies the previous one's benefit rather than merely adding to it.

The floor those levers approach is the fixed per-job overhead — checkout, dependency install, and runner startup — which is why optimizing that floor pays off across every job. Caching the package-manager store so installs are near-instant, keeping the checkout shallow enough to be fast but deep enough for the merge base, and reusing a warm container image all lower the constant that sharding cannot reduce. Past a point, the constraint on CI speed is no longer the work but the setup, and that is where the remaining wins live.

Security and isolation

Caching is a trust boundary: a poisoned cache entry can inject a malicious artifact into every consumer of it. Scope cache write access to trusted branches only, keep pull-request builds read-only against the shared cache, and include the lockfile in the cache key so a dependency change can never replay a stale artifact. Run installs with --ignore-scripts in CI so a compromised dependency cannot execute during the build.

The shared cache deserves the same threat modeling as any other shared, mutable resource. Because a cache hit replays a stored artifact verbatim, an attacker who can write a poisoned entry can inject a compromised build into every consumer that later replays it — a supply-chain attack that bypasses code review entirely. The mitigations are to make pull-request builds read-only against the shared cache, restrict write access to trusted branches, and include enough in the cache key that a malicious change cannot produce a colliding key for a different, trusted input.

Install-time safety applies in CI as much as anywhere. Running installs with ignored scripts on the runner prevents a compromised dependency from executing a lifecycle payload with the job's credentials, which in CI often include cache-write tokens and deploy keys. Pairing that with a frozen lockfile means the graph the runner installs is exactly the reviewed one, so neither the dependency set nor its install-time code can drift between what was approved and what runs.

The two credentials a CI job most needs to protect are the cache-write token and any deploy or publish credential, and both are exposed if a dependency can execute during install. Running installs with ignored scripts stops a compromised transitive package from running its lifecycle code with the job's environment, and scoping cache-write access to trusted branches keeps a pull request — which runs untrusted contributor code — from poisoning the shared cache. Treating the pipeline's own credentials with the same least-privilege discipline as any production secret is what keeps a fast pipeline from also being a soft target.

CI/CD integration: matrix sharding

Sharding splits the remaining task set across N runners using a matrix. Combine it with affected so each shard only runs its slice of the changed set.

  test:
    strategy:
      matrix:
        shard: [1, 2, 3, 4]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }
      - run: pnpm turbo run test --filter='...[origin/main]'
          -- --shard=${{ matrix.shard }}/4

A matrix build needs a clean completion contract, because a merge gate that depends on a variable number of shard jobs is awkward to express. The idiomatic pattern is a single aggregating job that depends on the whole shard matrix and simply succeeds if they all did, so the branch protection rule references one stable check name regardless of how many shards you run. Disabling fail-fast on the matrix means a failure in one shard does not cancel the others, so a single run surfaces every failure at once instead of forcing a fix-and-rerun cycle per shard.

Sharding interacts with flaky and order-dependent tests in ways worth anticipating. Because each shard runs a disjoint subset in its own process, tests that secretly depend on shared external state — a database row, a fixed port, a global temp file — can collide when shards run in parallel, surfacing failures that never appeared in a serial run. That is usually a latent bug the parallelism exposed rather than a sharding problem, and fixing the hidden shared state makes the suite both correct and parallelizable.

A subtlety of matrix sharding is that the shards must partition deterministically, or a flaky partition can hide a failure on one run and surface it on another. Framework-native sharding — a test runner's own shard flag — partitions by test file deterministically, which is why it is preferable to an ad-hoc split by directory that can drift as files are added. Combined with a single aggregating check that depends on every shard and fail-fast disabled, deterministic sharding gives you a stable required status and a full picture of every failure in one run, rather than a moving target that fails differently each time.

Common Pitfalls and Remediation

The failures in this space cluster around three misconfigurations: a shallow clone that breaks the diff and forces full runs, a cache key that omits an input and either misses or replays staleness, and sharding applied before affected so every shard runs the whole suite. Each has a one-line diagnosis — check the fetch depth, diff the dry-run hash inputs, print the affected plan — and each is far cheaper to catch in review than to debug after CI has been slow for a sprint. Treating the pipeline configuration as code that is reviewed and tested, rather than settings nobody revisits, is what keeps these from recurring.

Caching that is correct as well as fast

Caching turns the affected set from a smaller amount of work into a smaller amount of new work, by replaying any task whose inputs already have a stored result. The correctness of that replay hinges entirely on the cache key, which must include every input that could change the output: the source files, the resolved dependency graph, the task configuration, and any environment variables the task reads. Leave one out — an undeclared env var, an unpinned tool version — and the cache can either miss when it should hit, wasting the benefit, or worse, replay a stale result that no longer matches the inputs.

A remote cache extends the replay across machines, so a result computed on one runner is reused by every other runner and by developer laptops. That shared store is a trust boundary as much as a speed feature: a poisoned entry would inject a bad artifact into everyone who replays it. The safe configuration scopes cache write access to trusted branches, keeps pull-request builds read-only against the shared cache, and includes the lockfile in the key so a dependency change can never replay an artifact built against the old graph. Configured this way, the cache is a reproducible accelerator; configured carelessly, it is a new attack surface.

Sharding the remaining work across runners

When affected and caching have removed all the redundant work, what remains is genuine and serial — and sharding is how you make it fast without making it less. A framework-native shard flag partitions the test set deterministically across N runners, so each executes a disjoint slice and the wall-clock time drops toward the slowest single shard rather than the sum. Combined with the affected filter, each shard runs its portion of the changed set, never the whole repo.

Two details separate effective sharding from wasteful sharding. The shard count should be chosen so per-shard runtime approaches the fixed overhead of checkout and install but does not drop below it — past that point, adding shards just multiplies startup cost. And the partition should be balanced, because wall-clock time is set by the slowest shard: if one shard holds all the slow integration tests, the others finish early and idle. Measure the spread, isolate known-slow files, and gate the merge on a single required check that depends on every shard with fail-fast disabled, so one shard's failure surfaces the full picture rather than cancelling the rest.

Measuring the pipeline so it stays fast

A fast pipeline degrades quietly as the repo grows, so the levers need measurement, not just adoption. Track three numbers over time: the size of the affected set relative to the change (is affected still narrowing effectively, or has an over-broad input started pulling in the whole graph?), the cache hit rate (are keys stable across environments, or is something diverging?), and the per-shard runtime spread (is the partition still balanced?). Each has a specific diagnostic — a dry-run of the affected plan, a cache summary that shows hash inputs, and per-shard timings — that turns a vague 'CI feels slow again' into a concrete cause.

The discipline is to treat CI time as a budget with alerts rather than a number you notice when it hurts. A regression usually traces to one lever: a global file that started invalidating everything, an undeclared env var that broke cache sharing between CI and local, or a slow test that unbalanced the shards. Catching it on the change that introduced it — through a tracked metric or a dry-run diff in review — keeps the pipeline fast as a steady state rather than something you periodically rescue.

What legitimately forces a full pipeline run

Not every full run is a failure of optimization; some changes genuinely affect everything and should rebuild everything. A change to the root lockfile, a shared base tsconfig, the CI workflow itself, or a widely-imported foundational package can alter every package's output, so the safe and correct result is a full run. Recognizing these cases prevents the mistake of suppressing a legitimate full run and shipping under-tested code to save a few minutes.

The pathological case is the opposite: a full run triggered by a change that touches nothing global — a single leaf-package edit that somehow rebuilds the whole repo. That points at a specific, fixable cause: a shallow clone with no merge base, an over-broad input glob that folds unrelated files into every package's hash, or an unreachable base ref. The runner's dry-run output distinguishes the two — a full set with a global file in the diff is expected; a full set with only leaf changes is a configuration bug. Keeping the global-input list small and deliberate is what keeps legitimate full runs rare and accidental ones rarer.

Frequently Asked Questions

How much history does affected detection actually need?

Enough to reach the merge-base with your comparison branch. fetch-depth: 0 is the safe default; a shallow clone often lacks the base commit and forces a full rebuild.

Do I need a task runner to get affected builds?

A task runner (Turborepo, Nx) or pnpm's --filter makes it turnkey. You can script it by hand with git diff plus the workspace graph, but the runners already model dependents correctly.

Will caching make my CI non-deterministic?

Only if the cache key is wrong. Include every input that affects the output — source, lockfile, task config, and declared env — and a cache hit is provably identical to a rebuild.

Should pull-request builds populate the remote cache?

Keep them read-only. Let trusted branch builds write the shared cache so an untrusted PR cannot inject an artifact that other builds later replay.

In what order should I apply affected, caching, and sharding?

Affected first (it removes work), caching second (it replays work already done), sharding last (it parallelizes the remainder). Sharding first pays for parallelism on work affected would have skipped, which is why it under-delivers when applied out of order.

Why does my remote cache miss between CI and local?

A hashed input differs between environments — commonly an undeclared env var, a different Node version, or an unpinned tool. Declare every input the task reads, pin toolchain versions, and diff the dry-run hash inputs to find the divergence.

How many shards should I use?

Enough that per-shard test time approaches the fixed checkout-and-install overhead, but not so many that overhead dominates. Time the affected suite, divide by a two-to-three-minute target, and re-measure as the suite grows.

Why does a lockfile change rebuild the whole monorepo?

Because the resolved dependency graph is an input to every package's build hash, and a lockfile change can alter any package's dependencies. The runner conservatively — and correctly — treats the whole graph as affected. This is a legitimate full run, not a bug.

Why do tests pass serially but fail when sharded?

Usually a latent shared-state bug the parallelism exposed: tests that depend on a fixed port, a database row, or a global temp file collide when shards run at once. Fixing the hidden shared state makes the suite both correct and parallelizable.

What's the fastest way to speed up a slow monorepo pipeline?

Turn on affected detection first — it usually gives the biggest single win by running only what a change can reach. Then add remote caching to replay unchanged work and sharding to parallelize the rest. Applied in that order, the three make CI cost track the change rather than the repo.

Why does my affected build sometimes run everything?

Usually a shallow clone with no merge-base, so the runner can't compute the diff and falls back to everything, or a change to a genuinely global file (the lockfile, a base tsconfig) that legitimately invalidates the whole graph. Fetch full history, and check the dry-run plan to tell the two apart.

Related

• Choosing a Monorepo Task Runner — the runner that supplies affected detection and caching.
• Remote Caching Setup — the shared cache that makes CI reuse work across runners.
• pnpm Workspace Filtering — change-aware filters that select only affected packages.
• Configuring Nx Affected Commands in CI — the Nx-specific affected setup for pipelines.

← Monorepo Architecture & Orchestration