Chaos Engineering: How to Create an Automated Chaos Gauntlet using Gremlin & Jenkins

In this tutorial, we will demonstrate how to use Jenkins to create an automated chaos gauntlet. This will be done using Jenkins Pipelines and Stages to inject a controlled amount of failure with Gremlin. We then add a final stage that allows you to optionally halt the attack from the pipeline, rather than having to wait for the full duration of the attack.

Step 1 — Add Gremlin API Keys To Jenkins

In this step, you’ll enter your gremlin-api-key credentials into the Jenkins instance.

Open the following in your browser:

From Jenkins > Credentials > System > Global credentials (unrestricted), set the Kind and Scope as shown here. Then, enter your Gremlin Secret. Enter `gremlin-api-key` as the ID. Click OK to save.

Step 2 — Create your Chaos Staging Pipeline

In this step, you’ll create a Jenkins pipeline that uses the Gremlin API to run a Chaos Guantlet (series of Gremlin attacks) on your staging environment.

Click New Item > Pipeline and enter the name “chaos-staging-pipeline”. Add the source for your application/code, e.g. https://github.com/tammybutow/jenkins-gremlin.

Add your source (github project)

Scroll down to Advanced Project Options and add in the Pipeline script that will be used to call Gremlin.

Add your pipeline script (see the example below)

This is an example of a Pipeline script for your Chaos Staging Pipeline. It will run a Chaos Gauntlet that includes a CPU attack.

pipeline {
    agent none
    environment {
        ATTACK_ID = ''
        GREMLIN_API_KEY = credentials('gremlin-api-key')
    }
    parameters {
        string(name: 'CPU_LENGTH', defaultValue: '1000', description: 'Duration of CPU attack')
        string(name: 'CPU_PERCENT', defaultValue: '50', description: 'Percentage of CPU attack')
        string(name: 'CPU_CORE', defaultValue: '4', description: 'Number of cores to impact')
        string(name: 'TARGET_IDENTIFIER', defaultValue: '172.31.56.245', description: 'Host to target')
    }
    stages {
        stage('Chaos Staging Gauntlet') {
            agent any
            steps {
                script {
                    ATTACK_ID = sh (
                        script: "curl -s -H 'Content-Type: application/json' -H 'X-Gremlin-Agent: jenkins' -H 'Authorization: Key ${GREMLIN_API_KEY}' https://api.gremlin.com/v1/attacks/new --data '{ \"command\": { \"type\": \"cpu\", \"args\": [\"-c\", \"$CPU_CORE\", \"-l\", \"$CPU_LENGTH\", \"-p\", \"$CPU_PERCENT\"] },\"target\": { \"type\": \"Exact\", \"hosts\" : { \"ids\": [\"$TARGET_IDENTIFIER\"] } } }' --compressed",
                        returnStdout: true
                    ).trim()
                echo "see your attack at https://app.gremlin.com/attacks/${ATTACK_ID}"
                }
            }
        }
        stage('Observe and/or Halt Chaos') {
            agent any
            input {
                    message 'Do you want to halt attack?'
                    parameters {
                        choice(choices: ['yes' , 'no'], name: 'HALT', description: '')
                    }
            }
            steps {
                script {
                    if (env.HALT=='yes') {
                        sh "curl -s -X DELETE https://api.gremlin.com/v1/attacks/${ATTACK_ID} -H 'X-Gremlin-Agent: jenkins' -H 'Authorization: Key ${GREMLIN_API_KEY}' --compressed"
                    }
                }
            }
        }
    }
}

I also recommend adding more attacks to your Chaos Gauntlet, e.g. Disk attack, IO attack and Memory attack. You can use the built-in Gremlin API Examples feature to create the curl commands to run these attacks.

Now you are ready to Build your Staging Chaos Pipeline. To get started, click Build. You will see the Chaos Gauntlet commence running the attacks.

Staging Chaos Pipeline

Now that we’ve created our Staging Chaos Pipeline, let’s integrate it with the Build-Test-Deploy pipeline for our sample application.

Conclusion

This tutorial walked you through how to create a Chaos Pipeline that runs an automated Chaos Gauntlet. This makes use of the Gremlin API and Jenkins. We used Jenkins to create these CI/CD pipelines but you can also use Gremlin with other tools such as AWS Pipelines and Spinnaker.