How much will Mastodon instances try to re-send messages?
Simon Willison had some DNS issues which meant his personal Mastodon instance (which is similar to my Masto.host setup) was knocked offline for a day or so. He was wondering whether this would cause a flood of traffic when it came back online:
I wonder how much of an impact this had on the invisible Mastodon infrastructure…
Every time someone I follow posted in the past 24 hours their instance would have tried and failed to deliver to my instance - how hard do these things get retried?
Am I about to see a massive flood of incoming deliveries that finally work?
And there were a couple of replies to his post, explaining what would happen, including a link to the relevant part of the Mastodon codebase – if a job fails, it gets retried with an exponential delay.
clifff (15 March 2024):
@simon by default, sidekiq jobs that throw an error will be retried 25 times over the course of 20 days https://github.com/sidekiq/sidekiq/wiki/Error-Handling#automatic-job-retry
it’s possible mastodon has customized this but I haven’t specifically looked. would expect you’ll get more traffic than usual but shouldn’t be all at once
Jason Culverhouse (15 March 2024):
@cliff @simon
Here is the retry algorithm, if you were down a day. They will trickle in vs a massive flood
(0..16).each do |count| delay = (count**4) + 15 jitter = rand(0.5 * (count**4)) total_delay = delay + jitter puts "Retry #{count}: #{total_delay} seconds" endRetry 0: 15.713002155688077 seconds
Retry 1: 16.48915648567112 seconds
Retry 2: 31 seconds
Retry 3: 124 seconds
Retry 4: 382 seconds
Retry 5: 938 seconds
Retry 6: 1319 seconds
Retry 7: 2617 seconds
Retry 8: 6006 seconds
Retry 9: 9468 seconds
Retry 10: 12612 seconds
Retry 11: 19804 seconds
Retry 12: 26167 seconds
Retry 13: 31795 seconds
Retry 14: 55286 seconds
Retry 15: 67782 seconds
Retry 16: 89823 seconds