|Sent on:||Tuesday, March 5, 2013 1:45 PM|
Thanks for the explanation. I do need asynchronous jobs. Can I have a controller that would be an always-running micro instance (to cut down on cost), that would look in the queue and pass jobs to the auto-scaler? I see the auto-scaler a starting new instances (larger than micro) that would actually run the job. I’m looking for this setup because I won’t always have jobs that need to be run and I don’t want a big instance always running because of the costs. Also, do you know I can find some code examples for async queues?
i'm not sure what you mean by "controller".
for aws auto scaling, you just need to create an image that automatically "scales" based on events that are being tracked in cloudwatch. once it sees that you need more instances, it launches one - which is why your first instance can run the cron job with no problem.
if you want asyncronous jobs, you use queues (sqs, rabbitmg, gearman, etc). your application would send a job to the queue. at that point a worker would pick up the job and execute it. how you want to setup your worker is up to you. For instance your worker could be a cron job that looks into the queue every hour then works on the job. Or the worker can be a daemon that is continually connected to your queue listening for jobs to hit the queues.
for swf, my understanding of it is that its a workflow manager. it still needs something to tell it to run. this could potentially be your worker.
one possibility to use queues as sort of a cron job would be to pick up the job, do it, then put it back with a delay for the next interval to run it at.