Dev Web & SaaS

Python Celery Redis: The 8 Key Steps to Go from Zero to Operational

Python Celery Redis: The Essentials in One Article — Real Code, Diagrams and Concrete Steps, Excerpts from a 24-Lesson Course.

REHOUMA Haythem

12 Jun 2026 • 11 min read

Everyone can learn Python Celery Redis — provided they follow the steps in the right order. We have condensed a complete 24-lesson course into a clear path, with the most useful code snippets.

tl;dr

Why Celery
Redis essentials
Celery setup
Tasks and workflows
Celery Beat

~$ cat ./parcours.md # Python Celery Redis — 8 chapters

Why Celery

→ Limits of synchronous→ Architecture Broker / Worker / Backend+ 1 more lessons

Redis essentials

→ Redis data types→ Pub/Sub and Streams+ 1 more lessons

Celery setup

→ First worker and first task→ Configuration and best practices+ 1 more lessons

Tasks and workflows

→ Retries and timeouts→ Chain, Group, Chord+ 1 more lessons

Celery Beat

→ Periodic tasks→ Crontab schedules+ 1 more lessons

Monitoring

→ Flower dashboard→ Structured logs and Sentry+ 1 more lessons

Advanced patterns

→ Rate limiting and quotas→ Idempotence and deduplication+ 1 more lessons

Final project ML email pipeline

→ Project - Specifications→ Implementation of tasks+ 1 more lessons

🏁

Final project

→ You leave with a concrete and demonstrable project

Dynamic Scheduling via DB

NOTEGoal — Allow users/admins to modify schedules without redeploying.

Problem with beat_schedule in code

NOTEIf you want to add/modify a schedule in prod, you must redeploy. For user-defined tasks (newsletters, client reports), you need a dynamic scheduler.

django-celery-beat (recommended)

output

pip install django-celery-beat

# settings.py
INSTALLED_APPS = [..., "django_celery_beat"]
CELERY_BEAT_SCHEDULER = "django_celery_beat.schedulers:DatabaseScheduler"

# Run migrations
python manage.py migrate

# Beat reads the schedule from the DB every 5s
celery -A myproject beat -l info

Tables created

Create a schedule via Python

output

from django_celery_beat.models import CrontabSchedule, PeriodicTask

# 1. Define the timing
schedule, _ = CrontabSchedule.objects.get_or_create(
    minute="30",
    hour="9",
    day_of_week="mon-fri",
    day_of_month="*",
    month_of_year="*",
    timezone="Europe/Paris"
)

# 2. Create the periodic task
PeriodicTask.objects.create(
    crontab=schedule,
    name="Newsletter Acme Corp",
    task="app.tasks.send_newsletter",
    args=json.dumps(["acme_corp"]),
    kwargs=json.dumps({"template": "weekly"}),
    enabled=True,
    one_off=False,
)

Admin API via Django admin

output

# The Django admin already exposes the models
# You can directly go to /admin/django_celery_beat/periodictask/
# to add, edit, disable via the UI

Custom REST API

output

@app.post("/schedules")
def create_schedule(data: ScheduleCreate, db: Session = Depends(get_db)):
    schedule = CrontabSchedule.objects.create(
        minute=data.minute, hour=data.hour, ...
    )
    PeriodicTask.objects.create(
        name=data.name,
        crontab=schedule,
        task=data.task_name,
        args=json.dumps(data.args)
    )
    return {"status": "created"}

@app.delete("/schedules/{name}")
def delete_schedule(name: str):
    PeriodicTask.objects.filter(name=name).delete()

Without Django: celery-redbeat

output

pip install celery-redbeat

# celery_app.py
celery_app.conf.beat_scheduler = "redbeat.RedBeatScheduler"
celery_app.conf.redbeat_redis_url = "redis://redis:6379/2"

# Create a schedule
from redbeat import RedBeatSchedulerEntry
from celery.schedules import crontab

entry = RedBeatSchedulerEntry(
    name="my-task",
    task="app.tasks.my_task",
    schedule=crontab(hour=9, minute=0),
    app=celery_app,
    args=["arg1"]
)
entry.save()

# Delete
entry.delete()

Use case: multi-tenant newsletters

output

# DB model
class Newsletter(models.Model):
    name = models.CharField(max_length=200)
    schedule_cron = models.CharField(max_length=100)        # "0 9 * * mon"
    enabled = models.BooleanField(default=True)
    
    def save(self, *args, **kwargs):
        super().save(*args, **kwargs)
        self.sync_celery_beat()
    
    def sync_celery_beat(self):
        m, h, dom, mon, dow = self.schedule_cron.split()
        schedule, _ = CrontabSchedule.objects.get_or_create(
            minute=m, hour=h, day_of_month=dom,
            month_of_year=mon, day_of_week=dow
        )
        PeriodicTask.objects.update_or_create(
            name=f"newsletter_{self.id}",
            defaults={
                "crontab": schedule,
                "task": "app.tasks.send_newsletter",
                "args": json.dumps([self.id]),
                "enabled": self.enabled
            }
        )

Project - Requirements

NOTEMission — Build a product recommendation system sent by email to customers every week.

Specifications

NOTEFunctional:

For each active user: retrieve purchase history
Generate 5 recommendations via ML (cosine similarity or more advanced model)
Compose a personalized HTML email
Send via SendGrid with retry
Track delivery, open, click
Frequency: weekly, Monday 9am

Non-functional:

Scale: 100k+ users
Idempotency: no duplicate sends
SendGrid rate limit: 100 emails/second
Error recovery without replaying everything

Target architecture

output

+---------------+
| Celery Beat   |  cron mon 9h
| (1 instance)  |
+-------+-------+
        |
        v
+---------------+      +-----------+
| dispatch_     |--->  | Redis     |
| weekly_emails |      | broker    |
| (1 task)      |      +-----+-----+
+---------------+            |
                             v
                +------------+------------+
                |                         |
        +-------v------+         +--------v-------+
        | Worker emails|         | Worker ML      |
        | (10 procs)   |         | (2 procs, GPU) |
        +-------+------+         +--------+-------+
                |                         |
                v                         v
        +-------+------+         +--------+-------+
        | SendGrid     |         | Recommendation Model|
        | (rate limit) |         | + History DB   |
        +--------------+         +----------------+
                |
                v
        +-------+------+
        | Webhook      |
        | open/click   |
        +--------------+

Project structure

output

recommender/
├── docker-compose.yml
├── Dockerfile
├── requirements.txt
├── app/
│   ├── celery_app.py
│   ├── config.py
│   ├── tasks/
│   │   ├── orchestrator.py      # dispatch_weekly_emails
│   │   ├── ml.py                # compute_recommendations
│   │   ├── email.py             # render + send
│   │   └── tracking.py          # webhooks
│   ├── models/                  # SQLAlchemy
│   ├── ml/
│   │   └── recommender.py
│   └── templates/
│       └── weekly.html
└── tests/

DB models

output

class User(Base):
    id: Mapped[int] = mapped_column(primary_key=True)
    email: Mapped[str]
    active: Mapped[bool] = mapped_column(default=True)
    last_recos_sent: Mapped[datetime | None]

class Product(Base):
    id: Mapped[int] = mapped_column(primary_key=True)
    name: Mapped[str]
    category: Mapped[str]
    embedding: Mapped[list[float]]    # vector for ML

class Order(Base):
    id: Mapped[int] = mapped_column(primary_key=True)
    user_id: Mapped[int] = mapped_column(ForeignKey("users.id"))
    product_id: Mapped[int] = mapped_column(ForeignKey("products.id"))
    created_at: Mapped[datetime]

class EmailJob(Base):
    """Track sends for idempotency"""
    id: Mapped[int] = mapped_column(primary_key=True)
    user_id: Mapped[int] = mapped_column(ForeignKey("users.id"))
    week_key: Mapped[str]                  # "2026-W21"
    status: Mapped[str]                     # pending|sent|failed|opened|clicked
    sendgrid_msg_id: Mapped[str | None]
    created_at: Mapped[datetime] = mapped_column(default=datetime.utcnow)
    
    __table_args__ = (UniqueConstraint("user_id", "week_key"),)

Multi-queue configuration

output

celery_app.conf.task_queues = (
    Queue("orchestrator"),
    Queue("ml"),
    Queue("emails"),
    Queue("tracking"),
)

celery_app.conf.task_routes = {
    "app.tasks.orchestrator.*": {"queue": "orchestrator"},
    "app.tasks.ml.*": {"queue": "ml"},
    "app.tasks.email.*": {"queue": "emails"},
    "app.tasks.tracking.*": {"queue": "tracking"},
}

celery_app.conf.beat_schedule = {
    "weekly-recos": {
        "task": "app.tasks.orchestrator.dispatch_weekly_emails",
        "schedule": crontab(hour=9, minute=0, day_of_week="mon")
    }
}

Target SLOs

NOTE

All emails sent in < 2h for 100k users
Final failure rate < 0.5%
Tracking webhook latency < 1s
Resilience: if SendGrid is down for 10min, automatic recovery

Summary

NOTEKey takeaways

Separate ML (heavy), emails (throughput), orchestrator (control)
EmailJob with UniqueConstraint = idempotency
Beat for weekly triggering
Tracking via SendGrid webhook

Persistence and eviction

NOTEGoal — Configure persistence and memory eviction in production.

RDB: snapshots

output

# redis.conf
save 900 1          # snapshot if >=1 change in 15min
save 300 10         # snapshot if >=10 changes in 5min
save 60 10000       # snapshot if >=10000 changes in 1min

dbfilename dump.rdb
dir /var/lib/redis

NOTERDB: compact binary snapshot, fast restore. But may lose a few minutes on crash.

AOF: append-only log

output

# redis.conf
appendonly yes
appendfilename "appendonly.aof"

appendfsync always       # slow but 0 loss
appendfsync everysec     # default: 0-1s loss
appendfsync no           # OS decides

auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb

NOTEAOF: every write is logged. Stronger guarantees, but larger on disk.

Hybrid: RDB + AOF

output

# Recommended in prod: both enabled
save 900 1
appendonly yes
appendfsync everysec

# On restart: Redis loads AOF (more up-to-date)

maxmemory and eviction

output

# Memory limit
maxmemory 2gb

# Policy when full
maxmemory-policy allkeys-lru        # evict least recently used (cache)

Eviction policies

Policy	Behavior
noeviction	Reject writes (default)
allkeys-lru	Evict LRU across all keys
volatile-lru	LRU on keys with TTL
allkeys-lfu	Least frequently used
allkeys-random	Random
volatile-ttl	Key with shortest TTL first

Backups

output

# Manual snapshot
redis-cli BGSAVE              # asynchronous
redis-cli SAVE                # synchronous (blocking!)

# Copy the dump
cp /var/lib/redis/dump.rdb /backup/dump-$(date +%F).rdb

# Restore
systemctl stop redis
cp /backup/dump-2026-05-27.rdb /var/lib/redis/dump.rdb
systemctl start redis

Master-replica replication

output

# On the replica
replicaof master.redis.local 6379

# Check
redis-cli INFO replication
# role:slave
# master_link_status:up

# Automatic failover: Redis Sentinel
# Sharding: Redis Cluster

go-further

This article covers the most useful snippets — the complete Python Celery Redis course (8 chapters, 24 lessons, corrected exercises and final project) takes you all the way.

./access-the-full-course free course: Vibe Coding

FAQ

How long does it take to learn Python Celery Redis?

With a structured progression (8 chapters, 24 short and practical lessons), you reach an operational level in a few weeks at 30 to 60 minutes per day. The key is to practice each concept immediately.

Are there any prerequisites?

Basic computer knowledge is enough. If you can use a terminal and read simple code, you are ready.

Where to start concretely?

Reproduce the commands in this article, then follow the complete Python Celery Redis course: it chains the 24 lessons in order, with exercises and a final project.

./read-also

→ Get started with Portfolio IA SEO Vercel: your first concrete step today → IA Stripe GitHub SaaS in practice: the code and commands that really matter → Python Requests APIs explained simply (with diagrams and real code)

📬 Want to receive this type of guide every week? Subscribe for free — real code, zero fluff.

Dynamic Scheduling via DB

Problem with beat_schedule in code

django-celery-beat (recommended)

Tables created

Create a schedule via Python

Admin API via Django admin

Custom REST API

Without Django: celery-redbeat

Use case: multi-tenant newsletters

Project - Requirements

Specifications

Target architecture

Project structure

DB models

Multi-queue configuration

Target SLOs

Summary

Persistence and eviction

RDB: snapshots

AOF: append-only log

Hybrid: RDB + AOF

maxmemory and eviction

Eviction policies

Backups

Master-replica replication

FAQ

Stay up to date