Automating chaos engineering failover tests in AWS CI/CD pipelines

A detailed technical tutorial published on November 2, 2026, demonstrates how to embed automated reliability testing directly into continuous integration and deployment workflows. The guide, authored by Gbenga Ogbonyenitan for the DEV Community, walks engineers through using AWS Fault Injection Service (FIS) to intentionally stop container tasks and verify that the system recovers within acceptable time limits.

What happened

The article provides a step-by-step implementation of a chaos engineering pattern where a pipeline deploys a service, deliberately breaks one of its running components, and then validates that the infrastructure self-heals. The setup uses Amazon ECS (Elastic Container Service) with Fargate, an Application Load Balancer, and AWS CodePipeline. The core innovation is adding a "FailoverTest" stage that runs after deployment, ensuring that every release is accompanied by a proof of resilience.

The process involves two main phases: building the underlying service infrastructure and configuring the injection mechanism. First, the engineer sets up security groups, target groups, and the load balancer to handle traffic. Then, they configure FIS to target specific ECS tasks. The tutorial emphasizes that this is not just about breaking things, but about measuring the recovery time against a predefined threshold, such as 300 seconds. If the service does not return to a healthy state within that window, the pipeline fails, preventing potentially unstable configurations from being considered successful.

The author notes that while the demo uses a simple nginx web server, the pattern is designed for real-world applications. By automating this test, teams move from assuming their high-availability design works to proving it on every single release. The guide includes both a manual console-based approach for learning and a CloudFormation-based approach for production-ready infrastructure as code.

Key details

• The pipeline consists of three stages: Source, Deploy, and FailoverTest, with the final stage executing the chaos experiment.
• AWS Fault Injection Service is configured to stop exactly one task in a two-task ECS service, ensuring partial availability during the test.
• A custom script running in AWS CodeBuild checks pre-conditions, starts the FIS experiment, waits for completion, and validates recovery metrics.
• Recovery validation requires both the ECS running task count and the load balancer healthy target count to match the desired state.
• The experiment uses a CloudWatch alarm as a stop condition to monitor healthy hosts, though the primary safety control is limiting the fault to one task.
• Cleanup scripts are provided to handle resources not managed by CloudFormation stacks, such as log groups and task definition revisions.

Background

Chaos engineering is the discipline of experimenting on a system to build confidence in its capability to withstand turbulent conditions in production. Traditionally, this might involve manual game days or separate testing environments. Integrating it into CI/CD brings these tests closer to the code changes, catching regression in resilience logic early. AWS Fault Injection Service is a managed service that helps perform these experiments safely by defining scopes, actions, and stop conditions.

In this context, the "failover" refers to the automatic replacement of a failed container task by the ECS scheduler. When a task stops, the service detects the discrepancy between the desired count and the running count, launches a new task, and registers it with the load balancer. The test verifies that this entire loop completes without human intervention and within a timeframe that meets service level objectives.

Why it matters

For teams managing self-hosted software or internal platforms, reliability is often assumed rather than tested. Design documents may claim that a service is highly available, but configuration drift, permission errors, or network policies can silently break failover mechanisms. By automating this test, engineers convert abstract design claims into concrete, pass/fail checks. This reduces the risk of discovering recovery failures during actual outages, when stakes are highest.

Furthermore, this approach highlights the importance of recovery time objectives. It is not enough for a service to eventually come back; it must do so quickly enough to meet user expectations. The pipeline enforces a strict timeout, forcing teams to optimize their health check intervals, container startup times, and load balancer deregistration delays. This shifts reliability engineering from a reactive post-mortem activity to a proactive development requirement.

However, implementing this requires careful permission management. The tutorial details several common pitfalls, such as missing IAM permissions for creating service-linked roles or incorrect target group ARNs. These details underscore that chaos engineering introduces new operational complexities that must be managed alongside standard deployment tasks.

What you can do

• Review your current CI/CD pipelines to identify where resilience testing could be added as a post-deployment stage.
• Start with a non-production environment to test the FIS integration, ensuring you understand the failure modes before touching live traffic.
• Define clear recovery time objectives for your critical services and use them as the timeout threshold in your automated tests.
• Audit your IAM roles to ensure your build agents have the minimum necessary permissions to start FIS experiments and query ECS status.
• Implement pre-checks in your test scripts to verify the system is healthy before injecting faults, avoiding false positives from existing issues.
• Document the expected behavior of your services during partial outages and use these tests to validate those expectations continuously.