How to Set Up Sidekiq Queues for a Self-Hosted Mastodon

When you run your own Mastodon server, background jobs like sending emails, processing media uploads, and delivering posts to other instances are handled by Sidekiq. If Sidekiq queues are not configured correctly, your instance can become slow, unresponsive, or fail to deliver messages. This article explains how Sidekiq queue configuration works in Mastodon and provides step-by-step instructions to set up multiple Sidekiq processes for a self-hosted instance. By the end, you will understand how to allocate resources to each queue and avoid common performance pitfalls.

What Are Sidekiq Queues in Mastodon and Why They Matter

Mastodon uses Sidekiq to process background jobs asynchronously. Each job belongs to a queue based on its type. The default Mastodon queue setup includes five main queues: default, push, mailers, pull, and scheduler. The default queue handles media processing, link preview generation, and local timeline updates. The push queue delivers posts to remote instances. The mailers queue sends confirmation and notification emails. The pull queue fetches data from remote instances. The scheduler queue runs periodic tasks like cleaning expired records.

Without dedicated Sidekiq processes for each queue, a single process must handle all job types. If the push queue is overloaded, email delivery can be delayed. If media processing is heavy, federation can slow down. Assigning separate Sidekiq processes to different queue groups prevents one job type from starving others. This setup is essential for instances with more than a few hundred active users or for those that prioritize low-latency federation.

Prerequisites

Before you begin, ensure your Mastodon installation is running on a Linux server with systemd. You need root or sudo access to modify service files. Your Mastodon deployment should use the standard directory structure, typically located at /home/mastodon/live. You also need to know the number of CPU cores and the amount of RAM available on your server, as these determine the concurrency values you will set.

Steps to Configure Sidekiq Queues for Mastodon

Follow these steps to create dedicated Sidekiq processes for each queue group. This example assumes you want four Sidekiq processes: one for default and push, one for pull, one for mailers, and one for scheduler. Adjust the queue groups based on your instance’s workload.

1. Stop the existing Sidekiq service
   Run sudo systemctl stop mastodon-sidekiq to stop the default single Sidekiq process. Then disable it with sudo systemctl disable mastodon-sidekiq to prevent it from starting on boot.
2. Create a Sidekiq service file for the default and push queues
   Create a file at /etc/systemd/system/mastodon-sidekiq-default.service with the following content. Replace youruser with the Mastodon system user, usually mastodon. Set the concurrency value based on your CPU core count; a common starting point is 10 threads per core.

[Unit]
Description=mastodon-sidekiq default+push
After=network.target

[Service]
Type=simple
User=mastodon
WorkingDirectory=/home/mastodon/live
Environment="RAILS_ENV=production"
Environment="DB_POOL=25"
Environment="QUEUES=default,push"
ExecStart=/home/mastodon/.rbenv/shims/bundle exec sidekiq -c 15 -q default -q push
TimeoutSec=300
Restart=always

[Install]
WantedBy=multi-user.target

3. Create a Sidekiq service file for the pull queue
   Create /etc/systemd/system/mastodon-sidekiq-pull.service with the same structure but change the QUEUES environment variable and the ExecStart line to use the pull queue. For example: ExecStart=/home/mastodon/.rbenv/shims/bundle exec sidekiq -c 10 -q pull.
4. Create a Sidekiq service file for the mailers queue
   Create /etc/systemd/system/mastodon-sidekiq-mailers.service. Set the concurrency lower, such as 5, because email delivery is I/O bound and does not need many threads. Use ExecStart=/home/mastodon/.rbenv/shims/bundle exec sidekiq -c 5 -q mailers.
5. Create a Sidekiq service file for the scheduler queue
   Create /etc/systemd/system/mastodon-sidekiq-scheduler.service. The scheduler queue runs periodic tasks and should have a concurrency of 1 or 2. Use ExecStart=/home/mastodon/.rbenv/shims/bundle exec sidekiq -c 2 -q scheduler.
6. Reload systemd and enable the new services
   Run sudo systemctl daemon-reload. Then enable each service: sudo systemctl enable mastodon-sidekiq-default, sudo systemctl enable mastodon-sidekiq-pull, sudo systemctl enable mastodon-sidekiq-mailers, and sudo systemctl enable mastodon-sidekiq-scheduler.
7. Start all Sidekiq services
   Run sudo systemctl start mastodon-sidekiq-default, then start the other three services in the same way. Verify each is running with sudo systemctl status mastodon-sidekiq-default.
8. Monitor queue performance
   Use the Sidekiq web interface at https://yourdomain/sidekiq to view queue sizes and processing latency. Adjust concurrency values if any queue consistently backs up above 100 jobs.

Common Mistakes and Things to Avoid When Configuring Sidekiq Queues

Setting Concurrency Too High

A common mistake is setting the concurrency value to the total number of CPU threads without considering memory. Each Sidekiq thread consumes approximately 50 MB of RAM. If you set concurrency to 25 and run four processes, you need at least 5 GB of RAM just for Sidekiq. Monitor memory usage with htop or free -m and reduce concurrency if the server starts swapping.

Running All Queues in a Single Process

Some administrators keep the default single Sidekiq process and only increase concurrency. This approach causes all job types to compete for the same thread pool. If the push queue is busy delivering to thousands of remote instances, the mailers queue will have to wait. Separate processes prevent this contention and improve delivery reliability.

Ignoring the DB_POOL Environment Variable

Sidekiq processes need database connections. The DB_POOL environment variable in the service file must be at least as high as the concurrency setting. If DB_POOL is lower than concurrency, Sidekiq threads will block waiting for a database connection, causing timeouts and failed jobs. Set DB_POOL to a value 10 to 20 percent higher than the concurrency to accommodate overhead.

Sidekiq Queue Configuration: Single Process vs Multiple Processes

Item	Single Sidekiq Process	Multiple Sidekiq Processes
Queue handling	All queues in one worker	Each queue group in a dedicated worker
Resource usage	Lower total memory, but all jobs compete	Higher total memory, but no queue starvation
Concurrency tuning	Single high value must fit all job types	Per-queue tuning based on workload
Federation latency	Delayed if other queues are busy	Consistent low latency for push and pull
Email delivery speed	Delayed if media processing is heavy	Independent, fast email processing
Complexity	Simple, one service file	Requires multiple service files and monitoring

Setting up multiple Sidekiq processes is the recommended approach for any Mastodon instance with more than 500 active users or for administrators who need reliable federation and email delivery. The small increase in configuration complexity is offset by significant improvements in job processing speed and system stability.