test
Just Queue It!
Queue Everything & Delight Everyone.
About This Talk.
What Is Task Queue?
How Task Queue Works?
Why Task Queues?
Task Queue Implementation.
What Is A Task Queue?
Taks Queue is a mechanism to distribute work across machines/threads.
A Simple Queue
A Simple Python Process
Token Based Queue System
A task queue
Why Task Queues?
- Running Tasks In Background.
- Automatic Retrying Failed Tasks.
- Multi Processing.
- Scheduling Periodic Work.
Celery
- It’s a task queue with focus on real-time processing, while also supporting task scheduling.
Terminlology
Producer: Code to place tasks in broker.
Broker: Holds tasks & delivers to workers.
Consumer: Take tasks from broker & execute them.
Getting Started
Install Celery.
Install a broker.
Create tasks & start processing.
Tasks
# task.py
from celery import Celery
app = Celery('tasks', backend='amqp',
broker='amqp://guest@localhost//')
@app.task()
def add(x, y):
return x + y
# add tasks into queue result = add.delay(3, 4)
# check result result.ready() # result.status result.get()
Workers
celery worker --help # all options celery worker -A task -l info -n foo_worker
celery inspect active celery inspect active_queues
Monitoring.
Flower: Real-time monitor & web admin for Celery.