At work i've got quite of an interesting project which was sending an email while attaching 2 reports that can be quite process intensive. So, since the Django app already includes Celery so i decided to take advantage of it.

The general algorithm looks something along like:

  • Generate Report A
  • Generate Report B
  • send emaill

The tricky part about this is that we can do 2 things at once but we cannot send the email until the 3 first algorithm execution steps have been completed so! thanks to celery i was able to make this happens as following:

from celery.app import task
from celery import group

@task()
def generate_report_a(report_a_data):
    url = None
    ... # logic to generate the report and retrieve the url
    return url

@task()
def generate_report_g(report_b_data):
    url = None
    ... # logic to generate the report and retrieve the url
    return url

@task()
def send_email_content(reports, email_data):
    report_a, report_b = reports
    from_email = None
    to_email = None
    subject = None
    body = None

    ... # logic to parse email
    ... # send email with both reports

# algorithm execution with celery

parallel_tasks = [
    generate_report_a.s(report_a_data),
    generate_report_b.s(report_b_data)
]

chord(parallel_tasks, send_email_content.s(email_data) ).delay()

Basically the first 2 reports will get processed in parallel. Notice that when dealing with the chord module we're only dealing with signatures. Also notice that each report can be reused independently as well as within a more complex setup.

Isn't this a beauty? thanks Celery :)

The documentation is very clear! http://docs.celeryproject.org/en/latest/userguide/canvas.html#groups