Eventual Consistency: When the Read Outruns the Write

By Sergey Nosov

16 October 2026

In 1994, Douglas Terry and five colleagues at Xerox PARC proposed four guarantees for replicated data. The first, Read Your Writes, came with an example from Grapevine, which they called “the first widely used computing system with weak consistency.” Users who had just changed a password were sometimes told the new one was invalid. The login process, they explained, had “contacted a server to which the new password had not yet propagated.”

Three decades later, Microsoft documents the same kind of failure in Azure. Create a service principal, assign it a role right away, and the assignment can fail with PrincipalNotFound. The troubleshooting guide attributes this to a likely replication delay: the assignment can reach a region that has not replicated the principal yet.

Neither is a malfunction. Eventual consistency promises that once the writes stop, every replica will return the last write, but not when. So a replica’s “not found” can mean “not yet” or “never,” and the code that receives it has to know which.

Application code hits the same problem. One service writes an order and announces it on a queue; a second service takes the message and reads the order from a replica. In a test with PostgreSQL 18 and an in-memory queue, the replica was a streaming standby. PostgreSQL’s documentation says such a standby shows each commit after a small delay, “typically under one second assuming the standby is powerful enough to keep up with the load.” This one trailed the primary by about 2.5 milliseconds, and the message still won: the worker found no order for 1,000 of 1,000 messages.

Retrying, the usual fix, can only guess. This article shows how carrying the writer’s position in the message turns the guess into an answer. The C# samples were tested on .NET 10 with Npgsql 10.0.3, against PostgreSQL 18.6 and the fourth beta of PostgreSQL 19.

What “Eventually” Promises

In 2008, Werner Vogels, Amazon’s chief technology officer, defined eventual consistency this way: “the storage system guarantees that if no new updates are made to the object, eventually all accesses will return the last updated value.” Until the replicas converge, different readers may get different answers, and nothing has gone wrong.

That contract is weaker than it sounds. In 2013, Peter Bailis and Ali Ghodsi of UC Berkeley pointed out a hole in a looser definition, in which replicas merely end up agreeing. Under it, they wrote, “A database always returning the value 42 is eventually consistent, even if 42 was never written.” Vogels’s wording closes that hole: the replicas must converge on the last write. It still says nothing about what readers see before then. As the authors put it, “eventual consistency is purely a liveness property,” and “no behavior is ruled out in the meantime!”

Nor does the definition bound the wait. Vogels wrote that, if no failures occur, the maximum size of the inconsistency window can be worked out from communication delays, load, and the number of replicas. A slow replica, a partition, or a backlog stretches the window.

In practice, the window is often short. Bailis and Ghodsi report models predicting that LinkedIn’s production stores were consistent 99.9 percent of the time within 13.6 milliseconds, and Yammer’s within 202 milliseconds. Eventually consistent stores, they concluded, “are frequently consistent within tens or hundreds of milliseconds.” Frequently is not always.

I describe an eventually consistent system as one that is sometimes briefly wrong, on purpose. The convergence is guaranteed. The brevity is only typical.

Why accept that? A replica that does not wait for the others can keep answering during a network partition, and it answers sooner the rest of the time. My article on the CAP theorem and PACELC covers both trade-offs. Vogels argued that at Amazon’s scale, “events that normally have a low probability of occurrence are now guaranteed to happen,” and that a design must account for them up front.

Convergence itself takes work. In 1987, a Xerox PARC team that included Terry modeled update propagation on epidemics. In rumor mongering, a site spreads a fresh update to randomly chosen sites, like news in a small town, until too many of them already know it. Because “there is some chance that an update will not reach all sites,” the team backed it up with anti-entropy, a costlier comparison of whole databases, run less often. The guarantee comes from the backstop.

You Already Depend on It

The examples are everyday ones:

Eventual consistency is not exotic. The skill is knowing when you are in it.

The Read That Outruns the Write

Service A, an API, inserts an order on the primary and publishes an OrderPlaced message. In the test, an in-memory channel stands in for the message broker:

public sealed record OrderPlaced(Guid OrderId);

// Service A: the API
public static async Task PlaceOrderAsync(
    NpgsqlDataSource primary,
    ChannelWriter<OrderPlaced> queue,
    string customer, int totalCents)
{
    var id = Guid.NewGuid();
    await InsertOrderAsync(
        primary, id, customer, totalCents);
    await queue.WriteAsync(new OrderPlaced(id));
}

public static async Task InsertOrderAsync(
    NpgsqlDataSource db, Guid id,
    string customer, int totalCents)
{
    await using var cmd = db.CreateCommand(
        "INSERT INTO orders " +
        "(id, customer, total_cents) " +
        "VALUES ($1, $2, $3)");
    cmd.Parameters.Add(new() { Value = id });
    cmd.Parameters.Add(new() { Value = customer });
    cmd.Parameters.Add(new() { Value = totalCents });
    await cmd.ExecuteNonQueryAsync();
}

Service B, a worker, takes the message and reads the order from the standby:

// Service B: the worker
public static async Task HandleAsync(
    NpgsqlDataSource replica, OrderPlaced msg)
{
    await using var conn =
        await replica.OpenConnectionAsync();
    var order = await FindOrderAsync(
        conn, msg.OrderId);
    if (order is null)
        throw new InvalidOperationException(
            $"Order {msg.OrderId} not found");
    // ... ship the order ...
}

public sealed record Order(
    Guid Id, string Customer, int TotalCents);

public static async Task<Order?> FindOrderAsync(
    NpgsqlConnection conn, Guid id)
{
    await using var cmd = new NpgsqlCommand(
        "SELECT customer, total_cents " +
        "FROM orders WHERE id = $1", conn);
    cmd.Parameters.Add(new() { Value = id });
    await using var r = await cmd.ExecuteReaderAsync();
    return await r.ReadAsync()
        ? new Order(id, r.GetString(0), r.GetInt32(1))
        : null;
}

The primary and its standby ran in containers on one machine, with PostgreSQL’s default asynchronous replication. The test placed orders at least 5 milliseconds apart and gave each message its own handler. The handler threw “not found” for 1,000 of 1,000 orders, although every order existed. The message simply arrived first.

A separate run kept reading until each order appeared. All 1,000 did, a median 2.7 milliseconds after their commits returned, and 99 percent within 16.6 milliseconds.

To test a longer and steadier lag, some runs set the standby’s recovery_min_apply_delay to 500 milliseconds. PostgreSQL then delays only the commit records, and MVCC visibility rules keep each change invisible until its commit applies. With that delay, the handler also failed for all 1,000 orders.

A real broker adds its own delivery time, and a busy consumer may read late. Either can hide the race for a while. A design that is correct only while the queue is slower than replication is not correct.

PlaceOrderAsync has a second problem: if the process dies between the insert and the publish, the order exists and no one hears about it. My article on the transactional outbox shows the fix for that dual write. The outbox does not solve this race, though: its relay publishes after the commit, and the consumer can still ask a replica too early.

“Not found,” here, is timing, not truth. The trouble is that the worker cannot see which. The same empty result comes back when an order really is gone: deleted after the message was sent, or rolled back after a publish that came too early. A handler that treats every miss as “not yet” waits on messages that can never succeed. One that treats every miss as “never” drops real orders.

Retrying Only Guesses

The usual fix treats the miss as a normal, temporary state and retries with backoff. This version makes five attempts and doubles the backoff each time. It also uses what Marc Brooker of AWS called “Full Jitter” in 2015. Each wait is random, between zero and the current backoff, so that a crowd of retries does not arrive at once.

public static async Task<Order?> FindWithRetryAsync(
    NpgsqlDataSource replica, Guid id)
{
    var delay = TimeSpan.FromMilliseconds(100);
    for (var attempt = 1; ; attempt++)
    {
        await using (var conn =
            await replica.OpenConnectionAsync())
        {
            var order = await FindOrderAsync(conn, id);
            if (order is not null) return order;
        }
        if (attempt == 5) return null;
        // Full jitter: 0 to delay
        await Task.Delay(
            delay * Random.Shared.NextDouble());
        delay *= 2;
    }
}

On the healthy standby, every lookup found its order; the median wait was about 59 milliseconds, most of it jitter. With the standby half a second behind, the same code gave up on 169 to 184 of 1,000 orders across three runs. Every one of those orders existed. For an order that did not exist, the code took 0.2 to 1.2 seconds to reach the same verdict it had given those real orders.

Both failures come from the design, not the numbers. A bounded retry turns any lag longer than its budget into a false “never,” and a bigger budget only moves the line. With full jitter, the budget is itself random, from almost nothing to 1.5 seconds here. An unbounded retry turns a real absence into a message that never finishes. Long waits do not belong in a handler, either: past a few seconds, hand the message back to the broker with a delay.

Retries are still worth having for transient errors, and they make idempotent handlers essential, because a retried or redelivered message must not ship an order twice. My article on idempotency in APIs covers how. But a retry loop cannot decide whether to wait.

Carry the Writer’s Position

The writer knows something the worker does not: how far the database log had to go to include the order. A PostgreSQL primary appends every change to its write-ahead log, and a log sequence number, or LSN, marks each position in it. A standby replays the log in order. Once it has replayed past the order’s commit, the order and every earlier commit are visible there to any transaction that starts afterward. The API can put that position in the message:

public sealed record OrderPlaced(
    Guid OrderId, string CommitLsn);

public static async Task PlaceOrderAsync(
    NpgsqlDataSource primary,
    ChannelWriter<OrderPlaced> queue,
    string customer, int totalCents)
{
    var id = Guid.NewGuid();
    await InsertOrderAsync(
        primary, id, customer, totalCents);

    // Read it after the commit
    await using var pos = primary.CreateCommand(
        "SELECT pg_current_wal_insert_lsn()::text");
    var lsn = (string)(await pos.ExecuteScalarAsync())!;

    await queue.WriteAsync(new OrderPlaced(id, lsn));
}

Read the position after the commit returns, never inside the transaction. PostgreSQL 19’s documentation requires a target “at or after the end of the relevant write transaction’s COMMIT record on the primary.” A position read in the same batch as the INSERT comes before that record, because Npgsql runs a batch as one implicit transaction. With an outbox, the relay can add the position: once it sees the committed row, the current insert position is at or past that row’s commit record.

That batch mistake is easy to miss. With commit records held back by the 500-millisecond delay, a wait on a position read inside the batch succeeded every time. The order was missing every time: 200 of 200. Without the delay, the mistake never showed in 1,000 tries.

The code reads the insert position, as the documentation’s example does, in case synchronous_commit is off. When a commit ends exactly on a WAL page boundary, that position points past the next page’s header, and an idle standby reaches it only when more WAL arrives. In one run, 15 of 5,000 commits ended that way. With no other writes, the standby took up to 21 seconds to reach their insert positions, but their flush positions within 63 milliseconds. The quirk fails safe, with a false “not yet.” Where synchronous_commit is never off, pg_current_wal_flush_lsn() avoids it.

On PostgreSQL 18, the worker polls the standby until its replay position reaches the message’s position, then reads on the same connection. A second connection could land on a different standby, behind a load balancer or with Npgsql’s own Load Balance Hosts setting. A pooler such as PgBouncer, in transaction mode with several standby hosts, can switch servers between transactions even on one connection. Behind such a pooler, use a single standby, or wait and read in one transaction at READ COMMITTED, the default. At REPEATABLE READ, the first poll fixes the snapshot, and the read can miss the order.

// PostgreSQL 18: poll the replica
public static async Task<Order?> FindAfterAsync(
    NpgsqlDataSource replica, OrderPlaced msg,
    TimeSpan timeout)
{
    // Wait and read on one server
    await using var conn =
        await replica.OpenConnectionAsync();
    var clock = Stopwatch.StartNew();
    while (true)
    {
        // NULL: not a standby
        await using var check = new NpgsqlCommand(
            "SELECT CASE WHEN pg_is_in_recovery() " +
            "THEN pg_last_wal_replay_lsn() " +
            ">= $1::pg_lsn END", conn);
        check.Parameters.Add(
            new() { Value = msg.CommitLsn });
        var reached = await check.ExecuteScalarAsync();
        if (reached is DBNull)
            throw new InvalidOperationException(
                "Not a standby");
        if (reached is true)
            return await FindOrderAsync(
                conn, msg.OrderId);
        if (clock.Elapsed > timeout)
            throw new TimeoutException(
                "Replica is behind; retry later");
        await Task.Delay(20);
    }
}

Now “not found” has a meaning. Once the standby has replayed past the commit, an empty result says the order is gone, deleted or lost in a failover. Before moving the message to a dead-letter queue and raising an alert, confirm the absence on the primary: one read, only for the rare miss. A timeout is an honest “not yet,” and the TimeoutException lets the broker redeliver the message later. On a server that is not a standby, including one promoted in a failover, the CASE expression yields NULL, and the code throws rather than time out on every delivery.

In the test, every lookup found its order. The median wait was 29 milliseconds on the healthy standby, mostly the loop’s own sleep, and 525 milliseconds with the half-second lag. For an order that did not exist, the check reported it missing in 1 to 9 milliseconds. Pointed at the primary, it threw “Not a standby” after 2 milliseconds.

Terry’s 1994 design had the same shape. Before each read, a session manager checked that the server had every write the session had made, and if no server qualified, “it reports that the guarantee cannot be provided.” Here the message carries that record of writes from the API to the worker. Because one log orders every commit on a single primary, one number stands for all of them. With several primaries, as in a sharded database, the message needs a position from each primary the reader depends on.

Other stores expose the same idea under other names. A Cosmos DB session token is a position that one client instance can pass to another. In MongoDB, a causally consistent session tracks an operation time and a cluster time. A worker in another process can advance its own session to the writer’s times, carried in the message; a majority read from a secondary then reflects the writer’s majority write. My CQRS article waits on a read model’s position the same way.

PostgreSQL 19 Does the Waiting

PostgreSQL 19 adds a WAIT FOR LSN command. Its first release candidate came out on 15 October 2026, and general availability is planned for 19 November 2026. The documentation names this article’s use case: waiting for replay “is useful to achieve read-your-writes consistency while using an async replica for reads and the primary for writes.” The server does the waiting, so the loop disappears:

// PostgreSQL 19: the server waits
public static async Task<Order?> FindAfterWaitAsync(
    NpgsqlDataSource replica, OrderPlaced msg)
{
    // Throws unless it is an LSN
    var lsn = NpgsqlLogSequenceNumber.Parse(
        msg.CommitLsn);
    await using var conn =
        await replica.OpenConnectionAsync();
    await using var wait = new NpgsqlCommand(
        $"WAIT FOR LSN '{lsn}' " +
        "WITH (TIMEOUT '2s', NO_THROW)", conn);
    var status =
        (string)(await wait.ExecuteScalarAsync())!;
    return status switch
    {
        "success" => await FindOrderAsync(
            conn, msg.OrderId),
        "timeout" => throw new TimeoutException(
            "Replica is behind; retry later"),
        _ => throw new InvalidOperationException(
            $"WAIT FOR LSN: {status}")
    };
}

WAIT FOR LSN takes no bind parameter; in the test, WAIT FOR LSN $1 failed with a syntax error. So the code parses the message’s value as an LSN before it goes into the SQL. A message from a queue is input like any other, as my article on the boundary defense principle explains.

On the healthy PostgreSQL 19 standby, the median wait was 2.4 milliseconds, close to the replication lag itself and well under the poll’s 29. With the half-second lag, it was 498 milliseconds, and both runs found every order.

The status not in recovery means the server is not a standby, or no longer one, and the code throws on it. A failover can also mislead a standby that stays in recovery. WAIT “compares only the numeric LSN; it has no notion of which timeline a WAL record belongs to,” the documentation warns, and the PostgreSQL 18 check compares the same way. A standby on a different timeline from the writer’s can pass the number without ever holding the order. The primary check above catches that case.

Other Ways Out

Carry the data, not just the ID. If the message holds what the worker needs, the worker never reads the database, and the race is gone. It is often the cleanest fix. In 2017, Martin Fowler called it event-carried state transfer. He listed the costs, too: “lots of data schlepped around and lots of copies,” which he thought mattered less now that storage is abundant, and “more complexity on the receiver.”

I would add two costs. The data is a snapshot as of the commit, so if the order can change, include a version and let consumers ignore older events. And messages persist in queues, dead-letter queues, and logs, so copying personal data into them spreads it further than the database ever did.

Read the authority. For the one read that follows a write, read the primary. It is the simplest fix; to limit the primary’s load, keep it to the reads that need it. Be careful with a fallback that reads the primary whenever a standby is slow: standbys tend to fall behind under load, and the fallback moves that load onto the primary.

Pay at write time. PostgreSQL can make a commit wait until a synchronous standby has applied it, with synchronous_commit set to remote_apply. The setting can be chosen per user, per connection, or per transaction, so only the writes whose readers cannot wait need to pay. Where synchronous replication already guards durability, leave the default setting alone. On a primary that replicates asynchronously, name the standby by its application_name. Then set the default to local, which waits only for the local disk, so that other commits do not wait for the standby:

-- On the primary
ALTER SYSTEM SET
  synchronous_standby_names
  = 'replica1';
ALTER SYSTEM SET
  synchronous_commit = 'local';
SELECT pg_reload_conf();

Check that pg_stat_replication shows the standby’s sync_state as sync; a name that matches no standby makes these commits wait for one that never answers. Then give the writes that need it their own data source:

// Commits wait until the standby
// has applied them; a canceled
// wait only raises a warning
public static NpgsqlDataSource WaitForStandby(
    string connectionString)
{
    var builder =
        new NpgsqlDataSourceBuilder(connectionString);
    builder.ConnectionStringBuilder.Options =
        "-c synchronous_commit=remote_apply";
    return builder.Build();
}

Behind PgBouncer, that startup option works only if synchronous_commit is listed in track_extra_parameters; otherwise the pooler refuses it, or silently drops it if told to ignore it. Check it with SHOW synchronous_commit on a connection from the new data source.

In the test, every read of the standby right after such a commit found the order. The commits took a median 5.2 milliseconds on the healthy standby, against 4.2 for an ordinary commit, and about 500 milliseconds when the standby ran half a second behind. That is the latency side of PACELC, paid in the open.

The availability side is harsher. PostgreSQL’s documentation warns that such commits may never complete if a synchronous standby crashes. Naming several candidate standbys, as it recommends, keeps one crash from stalling them. But a commit waits only for as many standbys as synchronous_standby_names asks for, so a read on another standby can still miss it. Waiting for all of them brings the stall back. Consistency is a dial, not a switch, and the strongest setting is not the best one, only the most expensive.

A timeout ends the stall quietly. With the standby stopped, the test’s INSERT returned normally once its 3-second command timeout canceled the wait. The client received only a warning, through the connection’s Notice event. The server’s log added that the transaction had “already committed locally, but might not have been replicated to the standby.” The commit had become an ordinary one; Npgsql’s default command timeout does the same after 30 seconds. So handle the Notice event on every connection that makes such a write, and treat a write that drew the warning as one the standby may not have.

Tell the user. When people wait for their own write, a “processing” state beats pretending the lag away. Vogels framed user-perceived consistency in the user’s terms: the inconsistency window “needs to be smaller than the time expected for the customer to return for the next page load.” Where it may not be, show the state honestly, or keep the user’s position and wait on it. Pinning a user to one replica also works in a multi-leader store, where that replica takes the user’s writes, until it fails or the load balancer moves the user.

Measure the Window, and Test with It

Since the model does not bound the window, measure it. On a PostgreSQL primary, pg_stat_replication reports each directly connected standby’s lag. For an asynchronous standby, the documentation says, the replay_lag column roughly measures how long recent commits took to become visible there. On a caught-up, idle standby, the column shows NULL after a short time; that is not an error. Alert on the lag, and on a standby that drops out of the view; a cascading standby appears only in its upstream server’s view. The worst case comes after an outage, while a standby catches up.

SELECT application_name,
       replay_lag
FROM pg_stat_replication;

Cosmos DB measures staleness differently. Its Probabilistically Bounded Staleness metric, in the Azure portal, shows the probability of a consistent read for each pair of write and read regions.

Then test with lag. Tests against a single database never show this bug, while a 2.5-millisecond lag was enough to fail nearly every first read here. To make the failure reproducible, delay the commits on a test standby:

-- On a test standby only
ALTER SYSTEM
  SET recovery_min_apply_delay
  = '500ms';
SELECT pg_reload_conf();

The Security Lens

For authorization, the direction of staleness decides the risk. A new grant that has not taken effect fails closed: a 401 or a PrincipalNotFound error costs a retry. A revocation that has not taken effect fails open. Microsoft says role-assignment changes, removals included, can take up to 10 minutes to take effect because Azure Resource Manager sometimes caches configurations and data. For a managed identity’s group membership, it adds, a back-end cache per resource URI lasts around 24 hours. Until those caches catch up, a removed permission can keep working.

The read that outruns the write is worse in this direction. Suppose an offboarding service disables an account and publishes AccountDisabled, and a session service handles the event by reading the account from a replica. It finds the account still enabled and keeps the sessions alive. Nothing failed, and no alert fired. The fixes are the ones above: carry the decision in the event, wait for the writer’s position before trusting a replica, or read the authority. If the wait times out, end the sessions anyway: for a revocation, “not yet” must fail closed.

Partitions make this harder, because an attacker who controls the network can cause the delay; my article on the CAP theorem covers that case. Revocation and authorization belong at the strong end of the dial.

A Checklist for Reads That Follow Writes

My Software Development Principles series lists warning signs for eventual consistency, such as client code that assumes writes are immediately visible to all readers without session guarantees. These questions keep each read deliberate:

The goal is not to avoid eventual consistency. The goal is to never be surprised by it, and to never guess what “not found” means.

Further Reading