How to Detect a Misbehaving Mastodon Relay That Floods Sidekiq
🔍 WiseChecker

How to Detect a Misbehaving Mastodon Relay That Floods Sidekiq

Mastodon relays distribute public posts across instances, but a misconfigured or malicious relay can flood your Sidekiq queue with thousands of jobs per minute. This flood slows down or completely halts federation, local timeline updates, and email delivery. The root cause is usually a relay that sends too many activities or sends them too fast for your instance to process. This article explains how to identify the offending relay, confirm it is flooding Sidekiq, and take corrective action.

Key Takeaways: Detecting a Bad Mastodon Relay That Floods Sidekiq

  • Sidekiq dashboard at /sidekiq: Shows queue sizes, retry counts, and job latency in real time.
  • Mastodon admin dashboard at /admin/relays: Lists all relays with their status and last activity timestamp.
  • Instance audit log at /admin/action_logs: Records relay enable, disable, and error events for forensic analysis.

ADVERTISEMENT

Why a Relay Floods Sidekiq and How to Spot It

A Mastodon relay works by subscribing to a shared inbox. When the relay receives a new public post from any subscribed instance, it forwards that post to all other subscribers. If the relay is misconfigured — for example, it has a bug that causes it to resend the same post repeatedly, or it subscribes to too many instances and forwards every post without deduplication — your instance will receive a high volume of ActivityPub::ProcessingWorker and ActivityPub::DistributionWorker jobs.

The first sign of a flood is a sudden increase in Sidekiq queue size. You can check this by visiting https://your-instance.example/sidekiq and looking at the default queue. A healthy queue should have fewer than 100 pending jobs during normal operation. If you see thousands or tens of thousands of jobs, and they are all related to a single relay domain, that relay is likely the cause.

Another indicator is job latency. Sidekiq shows the time the oldest pending job has been waiting. If latency exceeds 30 seconds, your instance is struggling to keep up. In extreme cases, latency can reach several minutes or even hours.

Steps to Identify the Flooding Relay

  1. Open the Sidekiq dashboard
    Navigate to https://your-instance.example/sidekiq and log in with your admin credentials. Look at the default queue. If the queue size is above 500 and growing, a flood is in progress.
  2. Check the job arguments
    In the Sidekiq dashboard, click on the default queue to see the list of pending jobs. Each job shows its arguments, which include the activity ID and the source domain. Look for one domain appearing in many jobs — that is your suspect relay.
  3. Inspect the relay list in Mastodon admin
    Go to https://your-instance.example/admin/relays. You will see all relays your instance is subscribed to. Note the last_activity_at column. A relay that is flooding will show a very recent last activity time — often within the last few seconds — even if no new posts are being created on your instance.
  4. Review the instance audit log
    Navigate to https://your-instance.example/admin/action_logs. Filter by action type relay. Look for repeated enable or disable actions. A misbehaving relay may have been toggled multiple times as admins tried to fix the flood.
  5. Monitor Sidekiq job types
    In the Sidekiq dashboard, click on the Busy tab to see currently running workers. If most workers are running ActivityPub::ProcessingWorker or ActivityPub::DistributionWorker with the same relay domain in the arguments, that confirms the flood source.

ADVERTISEMENT

If Mastodon Still Has Issues After Identifying the Relay

Queue keeps growing after disabling the relay

If you disabled the relay but the queue continues to grow, the flood may have already queued thousands of jobs before you disabled it. These jobs will still be processed. To clear them, you can restart Sidekiq or use the Sidekiq API to delete jobs from the queue. On the Sidekiq dashboard, click Queues, then click the delete icon next to the default queue. This removes all pending jobs without processing them.

Relay re-enables itself after being disabled

Some third-party relay management scripts or cron jobs may re-enable a relay automatically. Check your server crontab and any custom scripts that interact with the Mastodon API. Also verify that no other admin user has re-enabled the relay. Use the audit log to see who performed the last enable action.

Other relays also start flooding after one relay is removed

This is rare but can happen if the first relay was a hub that forwarded activities from many other relays. When you remove the hub, the remaining relays may start sending directly to your instance. In this case, consider limiting the number of relays your instance subscribes to. Mastodon documentation recommends subscribing to no more than three relays for a medium-sized instance.

Mastodon Relay vs Direct Federation: Sidekiq Load Comparison

Item Relay Subscription Direct Federation
Job type ActivityPub::ProcessingWorker for each forwarded post ActivityPub::ProcessingWorker for posts from followed accounts only
Job volume per minute 100–5000+ depending on relay size 1–50 for a typical single-user instance
Control over content None — relay forwards everything Full — only follow accounts you choose
Risk of flood High — a single misconfigured relay can overwhelm Sidekiq Low — each followed account has rate limits

Direct federation gives you granular control over which posts enter your instance. Relays are convenient for discovering new content but require active monitoring. If you run a small instance, consider disabling all relays and relying on direct follows and local discovery instead.

You can now detect a misbehaving Mastodon relay by monitoring the Sidekiq dashboard, checking job arguments, and reviewing the relay list in your admin panel. The next step is to disable the offending relay and clear any queued jobs using the Sidekiq interface. For ongoing protection, set up a cron job that checks Sidekiq queue size every five minutes and alerts you if it exceeds 200 pending jobs.

ADVERTISEMENT