Retries That Fail Better: Backoff, Budgets, and Dead Letters

By Sergey Nosov

16 October 2026

At 2:19 a.m. Pacific time on 20 September 2015, a brief network disruption hit some of Amazon DynamoDB’s storage servers in US-East. They asked an internal metadata service for their membership, the table partitions each holds. The largest answers, already close to the servers’ time limit, came too late, and those servers took themselves out of service. AWS’s summary describes what followed: “Unavailable servers continued to retry requests for membership data, maintaining high load on the metadata service.”

The error rate on customer requests settled near 55 percent. At 5:06 a.m., engineers paused requests to the metadata service, a step that “decreased retry activity, which relieved much of the load,” and then added capacity.

In Worstward Ho (1983), Samuel Beckett wrote, “Ever tried. Ever failed. No matter. Try again. Fail again. Fail better.” Naive retry code stops at “try again”: the same request, sent at once, into the same struggling dependency. Failing better is a retry with a plan. The plan settles four things before the first failure: which failures deserve another attempt, how long to wait, when to stop, and where the work goes when the retries end. This article takes them in that order, with C# code tested on .NET 10.

How a Retry Storm Forms

A retry storm needs no bug. A dependency slows down, perhaps a database under a bad query plan. Its callers time out and retry at once. The retries time out too, so the load doubles, then triples, and the slow dependency stops answering. When it recovers, every caller retrying in step arrives in the same second, and it falls over again. Three properties of retries drive the storm:

Retry Only What Says “Not Now”

The first decision is not how to retry but whether to. A failure deserves another attempt only when the dependency said “not now”:

Other failures mean “no,” and the second attempt gets the same answer as the first, at the same cost:

Treat both lists as defaults that an API’s contract can override, and when a failure fits neither, do not retry it. A fresh token fixes a 401 for an expired one, but that is a new request, not a retry. In an eventually consistent store, a 404 right after a write can be temporary, as my article on eventual consistency shows. A 409 that means the first attempt is still running deserves another try. A 500 is a judgment call. The libraries below retry it; my loop does not, because a bug fails the same way every time.

Above all, retry only what is safe to repeat. RFC 9110, the HTTP standard, says a client “SHOULD NOT automatically retry a request with a non-idempotent method.” The exceptions are a client that knows the request is idempotent anyway and one that can tell the request “was never applied.” A timeout tells you neither. The first attempt may have charged the card after your timer fired, and a careful-looking retry charges it again. My article on idempotency in APIs covers the keys that make such a request safe to repeat; every attempt must send the same one.

Libraries cannot know which of your requests are safe. By default, Microsoft’s standard resilience handler for HttpClient retries every HTTP method. In my test, it sent a POST that kept failing with 503 four times. One line, DisableForUnsafeHttpMethods(), cut that to one.

Back Off with Full Jitter, inside the Caller’s Deadline

Waiting between attempts is backoff, and doubling the wait each time is exponential backoff. That spreads one client’s attempts out in time, but callers that failed together still retry together, on the same schedule. Marc Brooker showed this in 2015 on the AWS Architecture Blog: in his simulation of many clients updating the same database row, plain exponential backoff still left “clusters of calls.”

His fix was to add jitter, randomness in the wait. The variant he named Full Jitter waits a random time between zero and the capped exponential delay. Plain backoff took more work and more time than any jittered variant; Full Jitter did the least work, though Decorrelated Jitter finished slightly sooner. His verdict: jittered backoff “should be considered a standard approach for remote clients.”

The loop below puts these rules together. It makes at most five attempts. Each wait is random, from zero up to a ceiling that starts at 200 milliseconds and doubles after every failure, to at most 10 seconds.

static readonly TimeSpan BaseDelay =
    TimeSpan.FromMilliseconds(200);
static readonly TimeSpan MaxDelay =
    TimeSpan.FromSeconds(10);
// The least time worth starting
// an attempt with.
static readonly TimeSpan MinAttempt =
    TimeSpan.FromMilliseconds(100);
const int MaxAttempts = 5;

static async Task<T> RetryAsync<T>(
    Func<CancellationToken, Task<T>> call,
    Func<Exception, bool> isTransient,
    TimeSpan timeLimit,
    CancellationToken ct)
{
    using var cts = CancellationTokenSource
        .CreateLinkedTokenSource(ct);
    cts.CancelAfter(timeLimit);
    var clock = Stopwatch.StartNew();

    for (var attempt = 1; ; attempt++)
    {
        try
        {
            return await call(cts.Token);
        }
        catch (Exception ex) when (
            !cts.IsCancellationRequested
            && isTransient(ex))
        {
            if (attempt == MaxAttempts)
                throw;

            var wait = NextWait(attempt, ex);
            if (clock.Elapsed + wait + MinAttempt
                >= timeLimit)
                throw;

            await Task.Delay(wait, cts.Token);
        }
    }
}

// Full jitter: random, from zero
// to a ceiling that doubles.
// Retry-After wins when longer.
static TimeSpan NextWait(
    int attempt, Exception ex)
{
    var ceilingMs = Math.Min(
        MaxDelay.TotalMilliseconds,
        BaseDelay.TotalMilliseconds
            * Math.Pow(2, attempt - 1));
    var wait = TimeSpan.FromMilliseconds(
        ceilingMs * Random.Shared.NextDouble());

    var asked = (ex as RetryLaterException)
        ?.RetryAfter;
    return asked > wait ? asked.Value : wait;
}

The call turns HTTP answers into exceptions the loop can classify. A “not now” status becomes a RetryLaterException, which carries any Retry-After value. Any other unsuccessful status throws from EnsureSuccessStatusCode, and the loop lets it through:

sealed class RetryLaterException(
    HttpStatusCode status,
    TimeSpan? retryAfter)
    : Exception($"HTTP {(int)status}")
{
    public TimeSpan? RetryAfter { get; }
        = retryAfter;
}
static async Task<string> GetStockAsync(
    HttpClient http, string sku,
    CancellationToken ct)
{
    var path = "stock/"
        + Uri.EscapeDataString(sku);
    using var resp =
        await http.GetAsync(path, ct);
    var code = (int)resp.StatusCode;
    if (code is 408 or 429 or 502
        or 503 or 504)
    {
        throw new RetryLaterException(
            resp.StatusCode,
            RetryAfter(resp));
    }
    resp.EnsureSuccessStatusCode();
    return await resp.Content
        .ReadAsStringAsync(ct);
}

static TimeSpan? RetryAfter(
    HttpResponseMessage resp) =>
    resp.Headers.RetryAfter switch
    {
        { Delta: { } delta } => delta,
        { Date: { } date } =>
            date - DateTimeOffset.UtcNow,
        _ => null,
    };

static bool IsTransient(Exception ex) => ex
    is RetryLaterException
    or HttpRequestException
    {
        HttpRequestError:
            HttpRequestError.ConnectionError
            or HttpRequestError.ResponseEnded,
    }
    or HttpRequestException
    {
        HttpRequestError:
            HttpRequestError.Unknown,
        InnerException: IOException,
    }
    or OperationCanceledException
        { InnerException: TimeoutException };

A caller gives the client a timeout for each attempt and the loop a time limit for the whole operation:

var inventory = new HttpClient
{
    BaseAddress = inventoryUri,
    Timeout = TimeSpan.FromSeconds(2),
};

var stock = await RetryAsync(
    ct => GetStockAsync(
        inventory, "A17", ct),
    IsTransient,
    TimeSpan.FromSeconds(5),
    cancellationToken);

Five details carry the design:

Know When to Stop: Budgets and Breakers

The loop knows when one call should stop. It does not know when the whole service should. Five attempts on every call add up to a fivefold flood, and the flood arrives exactly when the dependency is already drowning. Two mechanisms see past the single call.

A retry budget limits retries across all calls rather than within one. Google’s SRE book pairs a limit of three attempts per request with a budget per client. Each client tracks the ratio of retries to all requests, and “a request will only be retried as long as this ratio is below 10%.”

gRPC’s retry design builds the budget as a token bucket. Every attempt that fails with a retryable error takes a token, and every success returns a fraction of one. Once half the tokens are gone, retries stop until successes refill them. Here it is in C#, with the design document’s example values:

// gRPC-style retry throttling:
// a failure costs one token, a
// success earns a tenth back,
// and retries stop once half
// the tokens are gone.
sealed class RetryBudget
{
    const double MaxTokens = 10;
    const double TokenRatio = 0.1;
    readonly Lock gate = new();
    double tokens = MaxTokens;

    public void OnSuccess()
    {
        lock (gate)
            tokens = Math.Min(MaxTokens,
                tokens + TokenRatio);
    }

    public bool OnFailure()
    {
        lock (gate)
        {
            tokens = Math.Max(0, tokens - 1);
            return tokens > MaxTokens / 2;
        }
    }
}

The budget plugs into the loop without changing it. Report every success, and let the classifier ask the budget before each retry. One budget serves every call to the same dependency:

var stock = await RetryAsync(
    async ct =>
    {
        var s = await GetStockAsync(
            inventory, "A17", ct);
        inventoryRetries.OnSuccess();
        return s;
    },
    ex => IsTransient(ex)
        && inventoryRetries.OnFailure(),
    TimeSpan.FromSeconds(5),
    cancellationToken);

The arithmetic sets a tipping point. A success returns a tenth of a token and a failure costs a whole one, so the tokens hold steady when about one attempt in eleven fails. I simulated 10,000 calls of at most five attempts against a dependency that failed attempts at random. At a 1 percent failure rate, the budget changed nothing: 10,116 attempts with it and without it. At 50 percent, it cut 19,357 attempts to 10,005, and in a total outage, 50,000 to 10,004.

The price shows in between. At 20 percent, the plain loop rescued all but one call, while the budget gave up on 2,033. That is the design: at one failure in five, the budget sees trouble, not a blip. The threshold is a choice. AWS documents that, with three attempts, its updated SDK quota starts to drain only when more than about 22 percent of requests keep failing with transient errors. For throttling, the threshold is about 32 percent.

A circuit breaker stops calling a dependency that keeps failing. Michael Nygard popularized the pattern in Release It! (2007). In Martin Fowler’s description, once failures reach a threshold, the breaker trips, and every further call returns an error without reaching the dependency. After a timeout, a half-open state lets a trial call through to see whether the problem is fixed.

Retries are for a blip; the breaker is for an outage. A retry loop with neither a budget nor a breaker retries through the whole outage, which is the storm again, with extra steps. My article on graceful degradation covers what to serve while the breaker is open.

Breakers have critics. The Builders’ Library warns that they “introduce modal behavior into systems that can be difficult to test” and can slow recovery. AWS limits retries locally with a token bucket instead, the retry quota of its SDKs’ standard mode. A budget limits only what retries add; a breaker also stops first attempts, which is the modal behavior AWS warns about.

Order decides what each layer sees. The standard resilience handler nests five strategies, outermost first: a rate limiter, a total timeout, the retry, the circuit breaker, and a timeout per attempt. The breaker sits inside the retry, so it counts every attempt. Once it opens, each attempt fails at once with a BrokenCircuitException, which the handler does not retry.

In my test, 60 concurrent calls went to a dependency that answered everything with 503; three retries each would have meant 240 requests. The dependency saw about 100 before the breaker opened, and all 60 calls ended with a broken circuit. The next call failed in under a millisecond, without sending a request.

Where the Work Goes When Retries Run Out

A message consumer has no caller to hand its failure to, so the failure must land where a person will see it. Enterprise Integration Patterns (2003) separates two cases. A message the messaging system “cannot or should not deliver” goes to a Dead Letter Channel. One that makes no sense to its receiver goes to an Invalid Message Channel. Azure Service Bus serves both with one dead-letter queue, a subqueue of every queue and subscription.

Here is a consumer, built on Azure.Messaging.ServiceBus and reusing IsTransient, NextWait, and MaxDelay. I tested its decisions against a stand-in receiver that records each settlement call, not a live namespace:

// The queue's MaxDeliveryCount.
const int MaxDeliveries = 10;

record Order(
    string Id, decimal Total);

// camelCase or PascalCase; a
// missing or null constructor
// argument throws.
static readonly JsonSerializerOptions Json =
    new(JsonSerializerDefaults.Web)
    {
        RespectRequiredConstructorParameters
            = true,
        RespectNullableAnnotations = true,
    };

static async Task HandleAsync(
    ProcessMessageEventArgs args)
{
    var msg = args.Message;
    var ct = args.CancellationToken;
    Order order;
    try
    {
        var body = msg.Body;
        order = body.ToObjectFromJson<Order>(Json)
            ?? throw new JsonException(
                "The body is null.");
    }
    catch (JsonException ex)
    {
        await args.DeadLetterMessageAsync(
            msg, "Unreadable",
            Cap(ex.Message));
        return;
    }

    try
    {
        await ProcessOrderAsync(order, ct);
    }
    catch (Exception ex) when (
        IsTransient(ex)
        && msg.DeliveryCount < MaxDeliveries)
    {
        // Cap the wait: it holds
        // the lock and one of the
        // processor's calls.
        var wait = NextWait(
            msg.DeliveryCount, ex);
        await Task.Delay(
            wait < MaxDelay ? wait : MaxDelay, ct);
        await args.AbandonMessageAsync(msg);
        return;
    }
    catch (Exception ex)
        when (!ct.IsCancellationRequested)
    {
        await args.DeadLetterMessageAsync(
            msg, ex.GetType().Name,
            Cap(ex.ToString()));
        return;
    }
    await args.CompleteMessageAsync(msg);
}

// Service Bus refuses a reason
// or a description over 4,096
// characters.
static string Cap(string text) =>
    text.Length <= 4096
        ? text : text[..4096];

Each path ends in one decision:

The processor below completes nothing on its own, because AutoCompleteMessages is off:

await using var processor =
    client.CreateProcessor("orders",
        new ServiceBusProcessorOptions
        {
            AutoCompleteMessages = false,
            MaxConcurrentCalls = 8,
        });
processor.ProcessMessageAsync += HandleAsync;
processor.ProcessErrorAsync += LogErrorAsync;
await processor.StartProcessingAsync();
// Disposing it stops processing.

With the default, true, the processor completes any message the handler returns without settling. If the handler throws, the processor abandons the message whatever that setting says, and that abandon is immediate too.

Waiting inside the handler holds the message’s lock and one of the processor’s eight concurrent calls, which suits waits of a few seconds, hence the cap. The lock lasts 1 minute by default, and the processor renews it for up to 5 minutes. Leave prefetch off, as here: a prefetched message’s lock runs while it sits in the buffer, and each expiry there counts as a delivery.

Service Bus has no built-in delay for an abandoned message; a request for one, first filed on GitHub in January 2020, is still open. For longer waits, Microsoft’s deferral guide suggests deferring the message and scheduling a reminder that carries its sequence number.

The same guide marks where waiting per message stops making sense: if every operation needs a database that is down, stop receiving until it is back. A circuit breaker on that dependency can trigger the pause, but not from inside a handler, which Microsoft warns can deadlock: StopProcessingAsync waits for every handler in flight.

Azure Functions has the immediate abandon built in: by default, its Service Bus trigger abandons the message when the function fails. Its Queue storage trigger tries a message five times with no delay, then moves it to a queue named <originalqueuename>-poison; Queue storage itself has no dead-lettering. Functions retry policies cover only Cosmos DB, Event Hubs, Kafka, and timer triggers.

Other brokers make other choices:

A Dead-Letter Queue Is a To-Do List

A dead-letter queue is a to-do list, not a graveyard. Each message in it is an order that did not ship or a payment that did not post, and somebody has to decide what happens to it. Four habits keep the list short:

Use the Retries You Already Have

In production, start from the retries your stack already has. In .NET, the Microsoft.Extensions.Http.Resilience package, built on Polly, adds the whole pipeline to an HttpClient in one call. By default it allows 30 seconds per operation. It retries three times with exponential backoff and jitter from a 2-second base and gives each attempt 10 seconds. Its breaker opens when 10 percent of at least 100 attempts in 30 seconds fail. Besides DisableForUnsafeHttpMethods(), check the timeouts before the first deployment:

services
    .AddHttpClient("inventory",
        c => c.BaseAddress = inventoryUri)
    .AddStandardResilienceHandler(o =>
    {
        o.Retry
            .DisableForUnsafeHttpMethods();
        o.AttemptTimeout.Timeout =
            TimeSpan.FromSeconds(2);
        o.TotalRequestTimeout.Timeout =
            TimeSpan.FromSeconds(5);
    });

Microsoft advises one resilience handler, not a stack of them, and the rule applies beyond its own package: many SDKs already retry on their own. Azure’s HTTP-based client libraries for .NET retry network failures and 408, 429, 500, 502, 503, and 504 responses by default. They make three retries, exponential from 0.8 seconds with 20 percent jitter either way, and never sooner than a Retry-After header asks. The Storage clients raise that to five retries.

Against a stub that always answered 503, one BlobClient call made six requests over about 26 seconds. Wrapped in the five-attempt loop from earlier, with a 5-minute limit and a classifier that accepted its 503, the same call made thirty requests over 2 minutes. Tune the client’s retry options, or turn them off and keep your own loop, but do not run both.

AWS plans to change its SDKs’ retry defaults in November 2026; most SDKs, .NET included, let you opt in now with AWS_NEW_RETRIES_2026=true. The updated standard mode backs off with full jitter from a 50-millisecond base, or 1 second after throttling, capped at 20 seconds.

Takeaways

My Software Development Principles series has entries on the Retry Pattern, Exponential Backoff, the Circuit Breaker, and the Dead Letter Queue. Each lists the signs that it is applied well or badly.

Try It on One Retry

Find one retry in a system you work on: a loop somebody wrote, a policy somebody configured, or a library default nobody looked at. Ask it four questions. Does it back off? Does it jitter? Does it stop, by attempts, by time, and by budget? And when it stops, where does the work go, and who finds out? Then ask what it repeats, whether that is safe, and whether anyone can see it retry. The point is to find the retry nobody knows is a storm.

Further Reading