Mastodon Sidekiq Queue Backlog Error: How to Clear

When you run a Mastodon instance, Sidekiq processes background tasks like delivering posts to remote servers, sending notifications, and updating timelines. A backlog occurs when the number of queued jobs exceeds Sidekiq’s ability to process them, causing delays in federation, missing notifications, and slow timeline updates. This article explains why the Sidekiq queue backlog happens and provides step-by-step methods to clear it safely. You will learn how to monitor queue size, restart workers, and purge stuck jobs without losing important data.

Why the Sidekiq Queue Backlog Occurs on a Mastodon Instance

Sidekiq uses multiple worker threads to process jobs from Redis queues. The default queues are default, push, pull, mailers, and scheduler. A backlog builds when the job arrival rate exceeds the processing rate. Common causes include a sudden spike in federated activity after a popular post, insufficient worker count for the instance size, a slow database or network link to remote servers, and a stuck job that blocks the queue. When a job fails repeatedly, Sidekiq moves it to the retry queue and eventually to the dead queue after exhausting retries. If the dead queue is not cleaned, it accumulates and contributes to the backlog. The backlog also grows when the Sidekiq process crashes or is killed without gracefully finishing its work.

How Queue Priority Affects Backlog

Mastodon assigns different priorities to queues. The push queue delivers outgoing activities to remote instances. The pull queue fetches data from remote instances. The default queue handles tasks like link preview generation and media processing. When the push queue backs up, your posts reach followers on other instances late or not at all. When the pull queue backs up, your local timeline and notifications from remote accounts become stale. Understanding which queue is overloaded helps you target the correct fix.

Steps to Clear the Sidekiq Queue Backlog in Mastodon

Before clearing the backlog, check the current queue size using the Sidekiq web interface or command line. Then apply the method that matches your situation. Always back up your Redis data before purging queues.

Method 1: Restart Sidekiq Workers

1. Open a terminal session on your Mastodon server
   Log in via SSH as a user with sudo privileges.
2. Check the current queue size
   Run systemctl status mastodon-sidekiq to see the process state and recent log lines. Use redis-cli -n 0 LLEN queue:push to check the push queue length. Replace push with the queue name you want to inspect.
3. Restart the Sidekiq service
   Run sudo systemctl restart mastodon-sidekiq. This kills all existing worker processes and starts fresh ones. The command takes a few seconds to complete.
4. Monitor the queue after restart
   Run redis-cli -n 0 LLEN queue:push again. If the queue length decreases over the next minute, the backlog is clearing. Wait five minutes and check again.

Method 2: Purge All Jobs from a Specific Queue Using Rails Console

If restarting does not reduce the queue size, you can purge all pending jobs from a queue. This method deletes jobs without processing them. Use it only when you understand that unprocessed jobs will be lost.

1. Open the Rails console
   Run cd /home/mastodon/live then RAILS_ENV=production bin/rails console as the mastodon user.
2. Select the queue to clear
   Type queue = Sidekiq::Queue.new("push") and press Enter. Replace push with the queue name you need to clear.
3. Check the number of pending jobs
   Type queue.size to confirm the backlog size.
4. Clear all jobs from the queue
   Type queue.clear and press Enter. The output shows the number of jobs removed.
5. Verify the queue is empty
   Type queue.size again. It should return 0. Exit the console by typing exit.

Method 3: Clear the Dead Job Queue

Dead jobs are those that exhausted all retries. They remain in the dead queue and count toward the total backlog. To clear them:

1. Open the Rails console
   Follow the same steps as Method 2 to enter the console.
2. Access the dead set
   Type ds = Sidekiq::DeadSet.new.
3. Check the dead job count
   Type ds.size.
4. Clear all dead jobs
   Type ds.clear. The output shows the number of dead jobs removed.
5. Exit the console
   Type exit.

If the Backlog Returns After Clearing

A recurring backlog indicates a systemic issue rather than a one-time spike. Address the root cause to prevent future problems.

Sidekiq Queue Backlog Returns Immediately After Restart

If the queue fills up again within seconds, a single stuck job may be re-enqueued repeatedly. Use the Sidekiq web UI at https://yourinstance.example.com/sidekiq to find jobs with high retry counts. Click the job ID and view its error message. Common errors include network timeouts to a specific remote server or database connection failures. Delete or retry that specific job from the UI.

High Memory Usage by Sidekiq Workers

Each Sidekiq worker consumes memory. If your instance has too many workers configured, the system runs out of RAM and kills processes. Check the Sidekiq configuration in /home/mastodon/live/config/sidekiq.yml. Reduce the concurrency value to 5 or 10 for small instances. Restart Sidekiq after changing the configuration.

Database Slowdown Causes Queue Growth

Mastodon jobs that write to the database, such as creating notifications or updating accounts, wait for database queries. If PostgreSQL is overloaded, jobs time out and retry. Check database performance with pg_top or htop. Add indexes on frequently queried columns or increase database memory settings in postgresql.conf.

Mastodon Sidekiq Queue Backlog vs High Retry Count: Key Differences

Item	Queue Backlog	High Retry Count
Definition	Pending jobs waiting to be processed	Jobs that failed and are scheduled for retry
Location in Sidekiq UI	Queues tab	Retries tab
Primary cause	Too many jobs arriving too fast	Persistent errors in individual jobs
Best fix	Restart workers or increase worker count	Delete or fix the failing job
Risk of data loss	Low if you restart quickly	Low if you fix the underlying error

After clearing the backlog, monitor the Sidekiq queues for at least 24 hours. Set up a monitoring tool like Prometheus with the Sidekiq exporter to alert you when queue size exceeds a threshold. For persistent backlogs, consider adding more Sidekiq processes by increasing the concurrency value in sidekiq.yml or running multiple systemd service units. Always test configuration changes on a staging instance first.