DevOps & monitoring

CI optimization results: faster queues, higher maintenance, and a 16-hour outage

A detailed post-mortem reveals that while Node job queue times dropped significantly, image build waits increased and a web bug caused a major production block.

Illustration of a server rack showing mixed performance status with green and red indicators
Illustration created for this article

A software team published a follow-up analysis to their previous report on continuous integration costs, measuring the impact of recent infrastructure changes over a short window in November 2026. The data shows mixed results: while wait times for standard Node jobs improved dramatically, image builds suffered due to a configuration error, and a separate web application bug caused a sixteen-hour production delay.

What happened

The team measured four key metrics across two distinct periods: a thirty-day baseline from mid-October to mid-November, and a shorter "after" window from November 18 to November 20, 2026. They focused on queue times, cache hit rates, maintenance labor, and failure recovery. The goal was to determine if shifting workloads to dedicated runner pools and adjusting cache strategies actually improved developer experience and system reliability.

The most significant improvement appeared in Node job queue times. At the 90th percentile, wait times dropped from 11.3 minutes to just 1.7 minutes, even as the load remained consistent. However, this gain was offset by a regression in image builds and end-to-end tests. Due to a setting that was never switched on, these heavier jobs continued to queue behind pull request checks, causing their 90th percentile wait time to rise from 3.0 minutes to 16.9 minutes. The team corrected this configuration late on November 18, but the damage to the metrics for that period was already done.

Beyond performance metrics, the team highlighted a severe incident unrelated to the runner infrastructure. A merge on November 19 introduced a web bug that broke the signed-in home page in the development environment. Because subsequent merges only involved documentation or infrastructure changes, they skipped end-to-end tests, allowing the broken state to persist. This triggered a deployment gate that halted production promotions for over sixteen hours until the root cause was identified and fixed.

Key details

  • Node queue times: The 90th percentile wait time fell from 11.3 minutes to 1.7 minutes at similar load levels.
  • Image build regression: Wait times for image builds and end-to-end tests rose from 3.0 to 16.9 minutes because a new runner pool was not correctly configured until late in the measurement window.
  • Merge to production: The median time for user-facing merges remained steady at roughly 50 minutes, though low-load merges after the fix completed in under 20 minutes.
  • Cache performance: Docker build cache hit rates increased from 70.5% to 77.6%, but the sample size was small enough that the team considers this change within the noise.
  • Maintenance spike: Daily changes to runner and CI infrastructure jumped from 1.3 to 7.1 per day, with all changes in the second window generated by automated agents.
  • Production block: A web bug caused a 16.3-hour delay in production deployments, during which eight merges were held back.

Background

Continuous Integration (CI) pipelines automate the process of testing and building code whenever developers submit changes. In self-hosted environments, teams manage their own "runners," which are servers that execute these tasks. Queue time refers to how long a job waits before a runner becomes available to process it. Reducing this wait is critical for developer productivity, as long delays break flow and context.

Cache hit rate measures how often the system can reuse previously computed results instead of rebuilding everything from scratch. Higher cache hits mean faster builds and lower compute costs. However, optimizing these systems requires constant tuning. Teams often split workloads into different pools—for example, separating quick linting tasks from heavy image builds—to prevent one type of job from blocking another. This strategy only works if the routing configuration is correct.

Why it matters

For teams running their own software, this case study illustrates that infrastructure changes rarely yield uniform improvements. Optimizing one part of the pipeline, such as Node jobs, can inadvertently highlight bottlenecks elsewhere, like image builds. The fact that the team’s median merge time did not improve significantly suggests that serial dependencies in the deployment lane remain a hard constraint. Engineers must look beyond average metrics and examine tail latencies to understand the true user experience.

The incident involving the sixteen-hour block underscores a critical operational risk: automated gates are only as effective as the tests they run. When non-code changes skip comprehensive testing suites, they can create blind spots that allow bugs to slip into critical paths. Furthermore, the reliance on automated agents for infrastructure changes increased the volume of modifications but did not necessarily reduce human cognitive load. Diagnosing why a gate failed still required deep investigation, proving that automation aids execution but not necessarily diagnosis.

What you can do

  • Verify runner routing: Ensure that new runner pools are actively receiving traffic by checking job logs, rather than assuming configuration changes are live immediately.
  • Monitor queue tails: Focus on 90th and 99th percentile wait times rather than averages, as these reflect the worst-case delays that frustrate developers.
  • Audit test coverage for skips: Review which types of commits bypass end-to-end tests and consider adding lightweight smoke tests to those paths to catch breaking changes early.
  • Track maintenance velocity: Measure the frequency of infrastructure changes to identify if optimization efforts are creating excessive churn or technical debt.
  • Set realistic cache expectations: Recognize that small sample sizes can make cache improvements appear statistically insignificant; monitor trends over longer periods.
  • Prepare for false positives: When deployment gates fail, investigate potential application bugs before assuming test flakiness to avoid wasting time on incorrect fixes.

More news

All news