Mastodon Federation Public Key Rotation: How to Rotate Without Breakage
🔍 WiseChecker

Mastodon Federation Public Key Rotation: How to Rotate Without Breakage

Your Mastodon instance uses cryptographic public keys to sign every message sent to other servers in the fediverse. When those keys expire or leak, you must rotate them to maintain secure federation. However, rotating a public key incorrectly can break communication with thousands of remote instances, causing posts, boosts, and follows to fail silently. This article explains why key rotation causes federation disruptions and provides a step-by-step method to rotate Mastodon’s public keys without losing connectivity.

Key Takeaways: Rotate Mastodon’s Federation Public Keys Without Causing Outages

  • Sidekiq queues and remote key cache flush: Clear the cached public keys on remote servers by restarting Sidekiq after rotation to force re-fetch.
  • Rails console key rotation command: Use Account.find_local('instance').update!(public_key: OpenSSL::PKey::RSA.new(2048).public_key) to rotate a single account’s key without affecting others.
  • Federation worker retry behavior: Remote instances automatically retry failed deliveries up to 7 times over 48 hours, so a gradual key rotation window is safe.

ADVERTISEMENT

Why Rotating a Mastodon Public Key Breaks Federation

Every Mastodon account has an RSA public-private key pair stored in the database. When your instance sends an activity to a remote server, it signs the HTTP request with the private key. The remote server looks up the public key from your account’s Actor object (retrieved via ActivityPub) and verifies the signature. If you rotate the key on your instance but the remote server still holds the old public key, all future signed requests will fail verification. The remote server rejects the activity, and your posts, boosts, and follows never appear on that server. This state persists until the remote server fetches your updated Actor object, which contains the new public key. The delay can last hours because remote servers cache Actor objects aggressively to reduce load.

Mastodon does not broadcast key changes automatically. The protocol has no standard message to say “my key has changed.” Instead, the remote server must re-fetch your Actor object. The re-fetch happens only when the remote server receives an activity from you and the cached signature verification fails. This creates a catch-22: the remote server cannot verify the signature, so it rejects the activity, and therefore never learns the new key. The fix is to force the remote server to re-fetch your Actor object before or immediately after the key rotation.

What Happens to Existing Signed Items After Rotation

Activities already delivered and accepted by remote servers remain valid. The remote server stores the activity content, not the signature. Key rotation only affects future deliveries. Old posts, boosts, and follows that were already accepted remain visible on remote instances. However, any pending deliveries in your Sidekiq queue that were signed with the old key will fail after rotation. Those jobs must be retried after the remote servers have the new key.

Steps to Rotate Mastodon Federation Keys Without Breakage

The following method rotates the public key for a single account while minimizing federation downtime. Perform these steps during a maintenance window with low traffic.

  1. Back up the current key pair
    Log into your Mastodon server via SSH. Navigate to the Mastodon directory. Run cd /home/mastodon/live and then RAILS_ENV=production bin/tootctl accounts rotate [username]. This command rotates the key using Mastodon’s built-in tool. It also sends a forced re-fetch signal to remote servers that have recently interacted with the account. This is the safest method because it updates the Actor object immediately.
  2. Verify the key update on the local Actor object
    Open a Rails console: RAILS_ENV=production bin/rails c. Run a = Account.find_local('username'); puts a.public_key. Confirm the key string is different from the backup you saved earlier. This verifies the rotation succeeded.
  3. Clear the local Sidekiq queue for federation deliveries
    Remote servers that already failed to deliver an activity may have the old key cached. Restart Sidekiq to clear the in-memory cache: systemctl restart sidekiq. This forces Sidekiq to reload the account’s public key from the database before retrying any failed deliveries.
  4. Force a re-fetch from remote servers
    Send a dummy activity to a few major remote instances to trigger a re-fetch. Use the Mastodon web interface to post a public toot mentioning @instanceadmin@remoteinstance.social. The remote server will attempt to verify the signature, fail, fetch the updated Actor object, and cache the new key. After this, all subsequent deliveries will succeed.
  5. Monitor federation logs for signature failures
    Check the Mastodon logs: journalctl -u mastodon-web -f | grep "signature". Look for lines containing “signature verification failed.” If you see failures more than 10 minutes after step 4, the remote server may not have re-fetched. Repeat step 4 with a different remote instance.
  6. Verify key rotation on a remote instance
    Use a Mastodon client on a different server. Open the account profile page of the rotated account. The public key URL is at https://yourinstance.social/users/username#main-key. A remote admin can fetch this URL directly to see the new key. If the key matches what you see in your Rails console, the rotation is visible to the fediverse.

Alternative Method: Manual Key Update via Rails Console

If the built-in tootctl command is unavailable or you need to rotate only the public key without changing the private key, use the Rails console directly. This method is riskier because it does not trigger the forced re-fetch signal. Use it only if you understand the consequences.

  1. Generate a new RSA key pair
    In the Rails console, run new_key = OpenSSL::PKey::RSA.new(2048). This creates a new private key. The public key is derived from it.
  2. Update the account record
    Run Account.find_local('username').update!(public_key: new_key.public_key). This overwrites the public key in the database. The private key remains unchanged in the database, so you must also update it: Account.find_local('username').update!(private_key: new_key.to_pem). If you change only the public key, signature generation will fail because the private key no longer matches.
  3. Restart Mastodon web and Sidekiq
    Run systemctl restart mastodon-web sidekiq. This forces the application to reload the account keys from the database.
  4. Force re-fetch manually
    Because the manual method does not send a re-fetch signal, you must rely on the first activity reaching a remote server. Post a public toot immediately. Monitor the logs for signature failures as described in the main method step 5.

ADVERTISEMENT

If Mastodon Still Has Federation Issues After Key Rotation

Remote Servers Still Reject Activities After 24 Hours

Some remote instances cache Actor objects for up to 48 hours. If you rotated the key and remote servers still reject activities after 24 hours, the remote server may have a bug in its caching logic. Contact the remote instance admin and ask them to manually flush the cache for your account’s Actor object. Alternatively, post a public toot that includes a link to your account’s Actor object URL. Some servers will re-fetch when they see the URL in a toot.

Sidekiq Queue Backlog After Rotation

After rotation, Sidekiq may have thousands of failed delivery jobs in the retry queue. These jobs were signed with the old key. They will continue to fail until the remote server re-fetches your key. To speed up recovery, clear the retry queue and let the system re-create the jobs: RAILS_ENV=production bin/tootctl sidekiq clear. This removes all pending jobs, including retries. New activities will be created and delivered with the new key. This is safe because Mastodon regenerates missed activities from the database when needed.

Key Rotation Does Not Take Effect for Some Accounts

If you rotated the key for one account but other accounts on the same instance are unaffected, the issue is isolated to that account. Verify that the account’s public key in the database is unique. Mastodon stores each account’s key pair separately. If the database update failed silently, run the rotation command again and confirm the key changed using the Rails console method described in step 2 of the main method.

Item Built-in tootctl Rotation Manual Rails Console Rotation
Description Uses Mastodon’s official key rotation tool Directly updates database fields via Rails console
Forced re-fetch signal sent Yes No
Risk of federation breakage Low High
Requires server access SSH and tootctl SSH and Rails console
Best use case Routine key rotation Emergency key change when tootctl is broken

You can now rotate your Mastodon instance’s federation public keys with minimal disruption by using the built-in tootctl command and forcing a re-fetch from remote servers. For routine rotations, schedule them during low-traffic hours and monitor the federation logs for 30 minutes afterward. As an advanced tip, set up a monitoring alert for the signature verification failed log pattern so you can detect key-related issues before users report them.

ADVERTISEMENT