DEP 0014:
Django Background Workers

Joe Riddle

Outline

Use cases

  • Sending emails
  • Web scraping
  • Nightly jobs
  • Anything that takes a long time
  • Anything that needs to run periodically

The state of background workers

DEP 0014

This proposal sets out to provide an interface and base implementation for long-running background tasks in Django.

— https://github.com/django/deps/blob/main/accepted/0014-background-workers.rst

DEP 0014 — History

DEP 0014 — Benefits

  • lower barrier to entry for offloading computation to the background
  • allow third-party libraries to offload some of their computation

DEP 0014 — Task Backends

Django will ship with three task backends:

  1. ImmediateBackend — Run tasks immediately without a background process. Useful for transitioning.
  2. DummyBackend — Do nothing but store tasks in memory. Useful for tests.
  3. DatabaseBackend — Use the ORM as a task store. Useful for realsies.

Demo 👨‍💻

Features DEP 0014 Celery django-rq (Redis) Huey
Background tasks ✅ ✅ ✅ ✅
Deferred tasks ✅ ✅ ✅ ✅
Recurring scheduling ❌ ✅ With django-celery-beat ✅ With rq-scheduler ✅
Task results ✅ ✅ With django-celery-results ✅ ✅
Automatic retry ❌ ✅ ✅ ✅
Timeouts ❌ ✅ ✅ ✅
Hooks ❌
but Django Signals		 ✅
Signals		 ✅
dependencies and callbacks		 ✅
pipelines
Monitoring ❌ ✅ various ways, Flower ✅ With rq-dashboard ❌
Setup effort Less:
management command (worker)		 Lots:
queue, worker, beat		 Some:
redis, worker		 Some:
redis, worker
Brokers Database RabbitMQ, Redis, others Redis Redis, sqlite, FS, in-memory

Questions 🎤

http://127.0.0.1:8000/admin/django_tasks_database/dbtaskresult/