I’ve been doing a fair bit of investigation around running jobs on AWS. There are a lot of different strategies we can go about this and there is no “one size fits all” so figured It might be a great idea to document the different options I’ve come across so far!
Using lambda
Cloud watch events can be used to trigger Lambda functions on a schedule. It‘s also possible to be able to react to events and trigger lamda functions eg cloud watch/S3 events, which is a great use case if you want to be able to react to different events rather than the traditional time based schedule. Lambda is serverless so it's a great use case when your not wanting to run jobs 24/7- so takes the need out of needing to provision an EC2 instance and paying for the uptime. Please note though this is not suitable for any jobs that will take a long period of time to run due to lambda time out limits- so it's worth checking this before you go down this route! It's a great use case for when you need to run quick jobs.
Provision an EC2 instance for running cron jobs:
Some benefits of spinning up your on EC2 instance and running your own cron jobs is the set up is relatively easy however it isn’t always the most cost effective option, especially if you only need your jobs to run at selected times. Jobs set up on EC2 instances can be quiet hard to monitor and it isn’t really a scalable solution if you are expecting to manage loads of jobs at one time.
Using ECS and Fargate:
An ECS container is an EC2 instance that runs the container Agent. It allows you to run docker images. You can create your very own images using Amazon ECR (Elastic Container Registery) which fully integrates with ECS. The EC2 instances are owned and managed by you however with Fargate it takes out the need to manage the EC2 instance so you just manage the tasks themselves. Giving you a serverless option.
AWS Batch
AWS batch can be used to schedule in jobs and also reactive to events. It runs jobs as docker containers and is able to dynamically provision the optimal resources to run your jobs which makes it easy to manage. It‘s really scalable and allows you to run thousands of concurrent jobs at any one time. AWS batch is completely free you only pay for the underlying resources that have been provisioned. AWS batch plans an executes your batch using EC2 instances/ Fargate. As per the example below it can be triggered with a lamda function, cloud watch event or cloud watch schedule. If you have a lot of jobs you need to manage then i'd definitely consider AWS Batch as a good option.
Summary
As per the above you can see there are a lot of different ways to go about scheduling tasks- there may also be more out there that I have yet to discover. The beauty of it is because of the variety you have a great range of different choices and you can find the service which specifically suits your needs! Have you used any of the above? Have you come across any different solutions? Feel free to email me beckydevops@hotmail.com- I'd love to have a conversation about it!
Comments