Terragrunt Parallel Apply in GitHub Actions: 2.7× Faster

The change began by discarding a 113‑line “order.txt” that forced every one of the 14,551 Terragrunt modules in inDrive’s monorepo to execute sequentially, even when they were unrelated. Engineers built a true dependency graph from Terragrunt’s own `dependency` blocks, repaired missing references, and locked the graph with an Open Policy Agent policy that blocks PRs lacking explicit dependencies or containing cycles. The pipeline then slices a PR’s modules into topologically ordered layers, launching a matrix job for each layer in GitHub Actions. Each layer can run up to 20 modules concurrently, and the workflow caps the total number of layers at ten, rejecting oversized PRs with guidance to split them. Over 3½ months, 355 parallel rollouts consumed 64 hours of runner time; the same work, if run serially, would have required roughly 171 hours, delivering a 2.7× aggregate reduction. The benefit is uneven: large PRs with many independent modules saw dramatic cuts (e.g., a 64‑module change finished in 31 minutes versus an estimated 4 h 40 m), while small changes often ran slower because the overhead of graph construction and layer orchestration outweighed any concurrency gains.

The move reflects a broader shift in infrastructure‑as‑code tooling toward graph‑aware execution and CI parallelism. As cloud environments grow more complex, static ordering files become brittle and costly; leveraging Terragrunt’s native dependency declarations aligns with practices seen in tools like Pulumi’s automation API and Terraform Cloud’s policy checks. GitHub Actions’ matrix strategy, combined with OPA enforcement, gives inDrive a reproducible, self‑correcting pipeline that can scale with its monorepo size. However, the approach also introduces new failure modes: a single slow module can stall an entire layer, and the ten‑layer ceiling forces engineers to restructure large PRs, potentially fragmenting logical changes.

Future watch points include how inDrive refines its layer limits and parallelism caps to balance AWS API throttling against runner availability, and whether the OPA policy evolves to automatically suggest missing dependencies rather than merely reject them. Monitoring the proportion of rollouts that exceed their serial baseline will indicate if the overhead can be trimmed further, especially for small PRs. Additionally, adoption of similar graph‑driven pipelines by other firms could pressure Terraform ecosystem tools to embed native topological scheduling, reducing reliance on custom CI glue.