Recently I posted about how to gracefully handle the termination of AWS Spot Instances. After some time in production it turned out not to be as reliable as I wished. Sometimes the Ansible environment wasn't in order or the script exited for no apparent reason. It was time for an update.
The requirements were:
- must be portable and independent
- must provide a safe Ansible environment including boto for AWS API interaction
The solution was obviously a Docker container. But why would you create a ~120MB container just to run some Bash and Ansible? Mostly because of the advantages that Ansible brings in terms of AWS API interaction. With this one task entry the instance is deregistered from all connected ELBs:
- name: Deregistering target instance from ELBs
local_action:
module: ec2_elb
instance_id: "{{ ansible_ec2_instance_id }}"
region: "{{ ansible_ec2_placement_region }}"
state: "absent"
In order to query the ECS-Agent API, that is only exposed on localhost, the container has to run with --net=host. As this is not possible to specify in an ECS-TaskDefinition, the container has to be started manually on each ECS instance. We do this directly during instance provisioning. Apart from deregistering the instance from ELBs and ECS, you can call other actions. I have included a Slack notification for example.
The ECS-SpotInstance-Terminationspotter is available on GitHub. Please build it yourself, because the container is not on DockerHub.