How to Schedule Tasks in Django
Scheduled tasks are an essential feature of many Django websites.
There’s a variety of reasons you might want to schedule tasks in Django.
You might have a weather app that needs to collect weather data daily, and update your website with the latest weather.
You might have a crypto price tracker app that needs to check the price of a crypto-currency every one minute and notify you if the press crosses a certain threshold.
You might have a web scraper that needs to scrape a job board daily to monitor for new listings, and notify you when new listings are found.
Since you’re here you may already have a use-case in mind. There is no ‘ultimate answer’ to the question of which solution is best, it really depends on your app’s needs —and that may change over time as your app evolves.
Let’s dive right in to some options.
How to Choose the Right Django Task Scheduler:
During my research for this article, I came across far too many solutions to include to be useful, so I will only be featuring the “best” ones that were recommended most often, and those that have the most docs / resources available for learning.
It’s especially important when choosing a solution, for any tech you’ll be adding to your project to consider the size of the community for that tech — that is, how many other people are out there using it, and discussing it.
Choosing the most ‘popular’ , highly recommended projects isn’t for everyone, some people like to be on the bleeding edge, but for those of who are learning and just want to make the process easier, this is the way to go — besides, they are often the ‘most popular’ for a reason.
Which option you choose depends on what your needs are.
As a general principle, try to follow these rules:
Keep it as simple as possible. Don’t over-engineer the solution.
Look for libraries that have an active and growing community where you can ask questions and get support. Reddit & Discord are great resources, and often have dedicated rooms for specific libraries.
Choose libraries that are proven, battle-tested; not the latest experimental one that hasn’t been tested extensively or used in many production environments.
The “Best” Ways to Schedule Tasks in Django:
Cron Jobs
Cron Jobs run on the server at specific intervals. There are limitation to this method, most notably, that it is less reliable than the others — if your server happens to go down at the time your task is scheduled to run, there is no built-in mechanism for ensuring it runs later.
There’s also no mechanism for ensuring that tasks are queued, or restarted if they fail. Cron is really best for super simple tasks that need to be run on a regular basis and are not essential to the app functioning.
For this reason, if reliability is of major concern, you’ll probably want a task queue / task scheduler / job queue option, like those mentioned below.
Pros: Available on most hosting providers, simplest option available.
Cons: Not 100% reliable. There is no mechanism for retrying failed tasks or tracking which tasks fail or succeed.
Advanced Python Scheduler (APScheduler)
APScheduler is a simple, yet customizable scheduler that runs in your project code — essentially operates as a ‘While Loop’ that runs scheduled tasks.
APScheduler does have some options for job persistence and can be configured to store data in-memory, or in database, depending on your needs.
APScheduler is also only meant to run on one server at a time, so if you need to spread your tasks across multiple servers, APScheduler would not be able to communicate amongst the various installations.
Pros: More feature-rich than simple Cron Jobs, providing multiple ways to store, queue and run tasks. Comes with persistence features to retry tasks in the event they are interrupted or incomplete.
Cons: Not ideal for large-scale projects that requires communication across multiple servers for tasks to be run.
APScheduler Resources:
Heroku Scheduler - Building a Django Bitcoin Price Alert app
Scheduled Jobs with Custom Clock Processes in Python with APScheduler
Scheduling Tasks in Django with the Advanced Python Scheduler
Whats the best Task Queue/Scheduler that could run my API calls in the background?
Use APScheduler to Schedule Tasks: Update a Model in Django Daily (Real-World Example)
Celery + RabbitMQ
Celery is a job queue that helps run tasks ‘asynchronously’ (at the same time), by routing tasks to execute on various ‘worker’ servers ; RabbitMQ is a ‘message broker’, that routes and manages messages on their way to queues for execution.
These two work together to create, schedule, and run tasks. Put simply: Celery creates passes the action to be scheduled to RabbitMQ, then RabbitMQ uses a queuing system to route that action back to Celery, where a Celery ‘worker’ executes the action.
Pros: Flexible - can be used with RabbitMQ or other message brokers.
Cons: More complicated than other options, due to having many options.
Resources for Celery + RabbitMQ:
Redis + Redis Queue (RQ) +
Redis is an in-memory database that can ALSO be used as a message broker, similar to RabbitMQ ; The two are different in that RabbitMQ is a ‘dedicated’ message broker (no additional features like caching) whereas Redis can also be used for caching pages in Django and makes them load faster by storing data in-memory.
Redis Queue (RQ), is a library for queueing jobs to be processed by Redis. RQ is similar to Celery, but is often said to be simpler and easier to set-up, whereas Celery is more robust in terms of configuration.
Pros: Simple Configuration. If you want caching as well as scheduled tasks, this can serve as a two-in-one solution.
Cons: RQ can only use Redis for message delivery, which can be a limitation if you want to use other message brokers.
Redis + Redis Queue (RQ) Resources:
Additional Django Task Schedulers:
As I mentioned earlier, there are far too many task scheduling options for Django to go over. There are 692+ GitHub Repositories on this topic. I would recommend sticking to the ‘most popular’ schedulers, listed above, but in the interest of being thorough, here are a few standouts from the ‘less popular’ category.
Django Extensions - Job Scheduling - A suite of extensions for Django that includes a job scheduling extension.
Django Kronos - Kronos uses Cron, but instead of scheduling many tasks on Cron, you schedule one task to run Kronos, which in turn runs cron modules in your project. Cron tasks are outlined in your project root and in application folders where tasks are needed.
Django “Poor Man’s Cron” - A task management app for Django that relies on traffic from the many bots around the web, such as search engine crawlers, to run scheduled tasks.
Dramatiq - A background task library for Python that positions itself as a more stable, simpler alternative to Celery, created by developers who previously used Celery and found it lacking.
Django Q - 'Django Q is a native Django task queue, scheduler and worker application using Python multiprocessing.
Rocketry (Formerly ‘Red Engine’)- Rocketry is a modern scheduling framework for Python applications. The developers state that their goal is to make it simple, clean and extensive.
Heuy - A ‘minimalist’ task scheduler for Django.
What tasks do you want to schedule?
👉 Comment below and share what automated tasks you want to schedule.