Jenkins Agents on EC2 Spot Instances

Photo by Kevin Ache on Unsplash

Inspired by how Lyft scaled their Jenkins infrastructure, I put together a proof of concept for running Jenkins agents on EC2 Spot instances instead of always-on boxes, using the EC2-Fleet plugin and an Auto Scaling Group to provision agents only when there's actually a job to run.

What this was meant to prove

• Jobs can run on spot instances without issues
• Agents scale automatically with the job queue
• Spot pricing meaningfully cuts the cost of CI infrastructure
• The master node itself can be downsized once it's no longer running builds directly

The stack

Jenkins (master + agents), EC2 Spot Instances, the EC2-Fleet plugin, an Auto Scaling Group, and IAM.

How it fits together

The Jenkins master runs on a normal EC2 instance. The EC2-Fleet plugin watches the build queue and provisions spot instance agents on demand, within limits you set on the Auto Scaling Group.

Setting up the Jenkins master

Launch an Amazon Linux 2 instance (a t2.micro is enough) with this user data:

#!/bin/bash
wget -q -O - https://pkg.jenkins.io/debian-stable/jenkins.io.key
sudo sh -c 'echo deb http://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list'
sudo yum update -y
sudo yum install java-1.8.0-openjdk -y
sudo wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat/jenkins.repo
sudo rpm --import https://pkg.jenkins.io/redhat/jenkins.io.key
sudo amazon-linux-extras install epel -y
sudo yum install jenkins -y
sudo service jenkins start

Use a VPC with DNS enabled, a public subnet with an internet gateway, and turn on auto-assigned public IPs.

Once it's up, hit the instance's public IP on port 8080. Grab the initial admin password with:

sudo cat /var/lib/jenkins/secrets/initialAdminPassword

Run through the setup wizard: install the suggested plugins and create an admin user.

Give the fleet plugin an IAM identity

Create an IAM user (Ec2-fleet-user) with programmatic access and this policy:

{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Effect":"Allow",
         "Action":[
            "ec2:DescribeSpotFleetInstances",
            "ec2:ModifySpotFleetRequest",
            "ec2:CreateTags",
            "ec2:DescribeRegions",
            "ec2:DescribeInstances",
            "ec2:TerminateInstances",
            "ec2:DescribeInstanceStatus",
            "ec2:DescribeSpotFleetRequests"
         ],
         "Resource":"*"
      },
      {
         "Effect":"Allow",
         "Action":[
            "autoscaling:DescribeAutoScalingGroups",
            "autoscaling:UpdateAutoScalingGroup"
         ],
         "Resource":"*"
      },
      {
         "Effect":"Allow",
         "Action":[
            "iam:ListInstanceProfiles",
            "iam:ListRoles",
            "iam:PassRole"
         ],
         "Resource":"*"
      }
   ]
}

Download the access key CSV: the plugin needs it during configuration.

Launch template + Auto Scaling Group

Create a launch template (EC2-fleet-launch-template) using the amzn2-ami-hvm AMI on a t3.small, with spot instance requests enabled, and this user data:

#!/bin/bash -xe
sudo amazon-linux-extras install epel -y
sudo yum update -y
sudo yum remove java-1.7.0-openjdk
sudo yum install java-1.8.0-openjdk -y
sudo yum install git -y
sudo yum -y update aws-cli

Build an Auto Scaling Group from that template (desired 1, minimum 1, maximum 4), spread across the same VPC's private subnets as the master.

Installing the plugin

Manage Jenkins → Manage Plugins, search for ec2-fleet, install without restart, then restart Jenkins.

Wiring it up

Manage Jenkins → Manage Nodes and Clouds → Configure Clouds, add an Amazon EC2 Fleet cloud:

1. Add AWS credentials using the IAM user's access key and secret
2. Pick the right region
3. The Auto Scaling Group should auto-populate
4. Test the connection
5. Set the launcher to SSH, using the Jenkins master's private key
6. Configure: 1 executor, 5 max idle minutes before scale-down, minimum cluster size 1, maximum cluster size 5, label spot-agents
7. Use the non-verifying verification strategy, and select Private IP for internal traffic

A test build

A freestyle job (sai-test-build) pointed at https://github.com/psdilip/python.git, with an Execute Shell step running:

python3 basics.py

Kick off the build and watch the Build Executor Status. You'll see the fleet provision an instance in real time, and the console log fills in as usual.

The result

The EC2-Fleet plugin deployed spot instances as Jenkins agents and ran jobs in parallel without any manual provisioning, all while cutting infrastructure cost and taking load off the master node entirely. It's a solid foundation to build a larger CI/CD setup on top of.

Practical guide: setting this up yourself

A condensed, in-order checklist of everything above.

1. Launch the Jenkins master. A t2.micro running Amazon Linux 2 is enough, with a user data script that installs Java and Jenkins and starts the service, in a VPC with DNS enabled and a public subnet (internet gateway plus auto-assigned public IPs).
2. Finish the setup wizard. Hit the instance's public IP on port 8080, grab the initial password with sudo cat /var/lib/jenkins/secrets/initialAdminPassword, then install the suggested plugins and create an admin user.
3. Create an IAM user for the fleet plugin. Name it Ec2-fleet-user, give it programmatic access, and attach a policy covering EC2 spot fleet actions (DescribeSpotFleetInstances, ModifySpotFleetRequest, TerminateInstances, etc.), Auto Scaling Group actions (DescribeAutoScalingGroups, UpdateAutoScalingGroup), and IAM actions (ListInstanceProfiles, ListRoles, PassRole). Download the access key CSV.
4. Build a launch template for agents. Call it EC2-fleet-launch-template, base it on the amzn2-ami-hvm AMI on a t3.small, enable spot instance requests, and give it user data that installs Java, git, and updates the AWS CLI.
5. Build the Auto Scaling Group. Use that launch template, set desired 1, minimum 1, maximum 4, and place it in the same VPC's private subnets as the master.
6. Install the EC2-Fleet plugin. Under Manage Jenkins → Manage Plugins, search for ec2-fleet, install without restart, then restart Jenkins.
7. Configure the Amazon EC2 Fleet cloud. Under Manage Jenkins → Manage Nodes and Clouds → Configure Clouds, add the IAM user's credentials, pick the region, confirm the Auto Scaling Group auto-populates, test the connection, set the launcher to SSH with the master's private key, and configure 1 executor, 5 max idle minutes, minimum cluster size 1, maximum cluster size 5, and label spot-agents (non-verifying verification strategy, Private IP for internal traffic).
8. Run a test build. A freestyle job pointed at a git repo with an Execute Shell step is enough to watch the fleet provision a spot instance in real time and confirm the job actually runs on it.