At work i've got quite of an interesting project which was sending an email while attaching 2 reports that can be quite process intensive. So, since the Django app already includes Celery so i decided to take advantage of it.
The general algorithm looks something along like:
- Generate Report A
- Generate Report B
- send emaill
The tricky part about this is that we can do 2 things at once but we cannot send the email until the 3 first algorithm execution steps have been completed so! thanks to celery i was able to make this happens as following:
from celery.app import task from celery import group @task() def generate_report_a(report_a_data): url = None ... # logic to generate the report and retrieve the url return url @task() def generate_report_g(report_b_data): url = None ... # logic to generate the report and retrieve the url return url @task() def send_email_content(reports, email_data): report_a, report_b = reports from_email = None to_email = None subject = None body = None ... # logic to parse email ... # send email with both reports # algorithm execution with celery parallel_tasks = [ generate_report_a.s(report_a_data), generate_report_b.s(report_b_data) ] chord(parallel_tasks, send_email_content.s(email_data) ).delay()
Basically the first 2 reports will get processed in parallel. Notice that when dealing with the chord module we're only dealing with signatures. Also notice that each report can be reused independently as well as within a more complex setup.
Isn't this a beauty? thanks Celery :)
The documentation is very clear! http://docs.celeryproject.org/en/latest/userguide/canvas.html#groups