Storage systems use “log,” “high watermark,” and “low watermark” for related but distinct ideas. Treating them as interchangeable creates subtle recovery and data-loss bugs.

This article separates three layers:

  1. a local write-ahead log that makes state recoverable after a crash;
  2. a replicated log whose commit position determines what is safe to expose;
  3. retention boundaries that determine what old history may be discarded.

The names vary across systems. The safety argument matters more than the label.

1. A write-ahead log is an ordering rule

A write-ahead log (WAL) records a change before the corresponding data pages are considered durable. On recovery, the system replays durable records to reconstruct state that had not yet reached its final location.

A typical update path is:

  1. assign a monotonically increasing log sequence number (LSN);
  2. encode the logical or physical change as a framed record;
  3. append it to the current log segment;
  4. make the log durable according to the requested durability policy;
  5. apply the change to in-memory or cached data pages;
  6. acknowledge only at the point promised by the API.

The essential invariant is:

A data page containing a change must not be made durable before the log record needed to explain that change is durable.

Otherwise a crash can leave a page with effects that recovery cannot interpret or roll back.

Record framing and torn writes

A WAL is not merely newline-delimited JSON. Recovery must distinguish a complete record from a partial tail left by power loss or a short write. A record commonly includes:

length | version | flags | lsn | transaction-id | payload | checksum

Recovery scans records in order, validates lengths and checksums, and stops or repairs at the first invalid tail according to the storage format. Checksums detect corruption; they do not by themselves make a multi-sector write atomic.

Segments make rotation, archival, and truncation manageable. A segment should be published with a clear protocol—for example, write and synchronize its contents, write and synchronize metadata if required, and atomically install the final name. Exact steps depend on the filesystem and durability contract.

write is not necessarily durable

A successful userspace write generally means the kernel accepted the bytes. It does not necessarily mean stable storage has persisted them. The durability boundary may require fsync, fdatasync, direct-I/O semantics, a database-specific flush, and storage hardware that honors flushes correctly.

Calling a synchronization operation for every transaction is expensive, so databases use group commit:

  1. several transactions append records under one ordered log position;
  2. one flusher synchronizes through LSN N;
  3. every waiter whose required LSN is at or below N may complete.

The flusher must publish the durable LSN only after the synchronization succeeds. An in-memory “last appended LSN” is not the same as “last durable LSN.”

Idempotent recovery and checkpoints

Recovery normally starts from a checkpoint and scans forward. A checkpoint does not have to contain every page; it identifies a point from which the remaining log is sufficient.

Redo operations should either be idempotent or guarded by page/version LSNs. A common rule is: apply log record r to page p only if p.page_lsn < r.lsn. Transactional systems may also need undo, compensation records, or commit/abort metadata.

A safe checkpoint protocol establishes which dirty pages and log ranges are covered before it advances the truncation boundary. Deleting old segments merely because “a checkpoint started” can remove the only recovery record for a page not yet persisted.

2. Replication adds a different notion of commitment

In a replicated state machine, each node may have a local log end, but clients should usually observe only entries that the replication protocol has committed.

For Raft, a leader tracks a matchIndex for every server. Informally, an index is replicated on a majority when a quorum’s matchIndex values are at least that index. Raft adds an important leader rule: the leader advances commitIndex by counting replicas only for an entry from its current term. Earlier entries become committed indirectly once a current-term entry is committed.

Conceptual pseudocode is:

for N from last_log_index down to commit_index + 1:
    replicated = count(server where match_index[server] >= N)
    if replicated >= quorum_size and log[N].term == current_term:
        commit_index = N
        break

This is not the median of follower offsets, and the leader is part of the quorum. The term check is part of Raft’s safety proof; omitting it can incorrectly “commit” an entry from an earlier term.

The commit index and the local durable index are separate dimensions. A protocol must specify when followers acknowledge an append, whether that acknowledgement implies stable storage, and when the leader responds to a client.

Kafka’s high watermark is system-specific

Kafka exposes a partition high watermark representing the upper boundary of records replicated sufficiently for ordinary consumer visibility. Its exact advancement rules are tied to Kafka leaders, follower fetch state, and the in-sync replica set. Transactional consumption also uses a last stable offset, which is another boundary.

That is why “high watermark = last offset on a majority” is too broad. In one system the term means a Raft commit index; in another it is a broker-specific replication boundary; elsewhere it may mean an event-time observation. Always define:

  • who computes it;
  • which participants count;
  • whether acknowledged bytes are durable;
  • whether the value survives leadership change;
  • what readers are allowed to observe below or above it.

3. Retention needs a low boundary

Old log entries can be removed only when no required consumer depends on them. A retention boundary is often the minimum of several constraints:

retain_from = min(
    recovery_checkpoint_lsn,
    slowest_required_replica_lsn,
    oldest_snapshot_lsn,
    oldest_changefeed_lsn,
    backup_or_archive_lsn
)

The minimum is conceptual; a real system must define inclusive/exclusive offsets and segment boundaries precisely.

This “low watermark” is not necessarily a single field. It can be derived from leases and pins owned by:

  • crash recovery;
  • lagging replicas that are still eligible to catch up incrementally;
  • long-running readers or MVCC snapshots;
  • change-data-capture consumers;
  • point-in-time recovery and backups;
  • legal or operational retention policy.

If one consumer falls too far behind, the system needs policy: retain unbounded history, evict the consumer, install a snapshot, or fail writes under disk pressure. Silently deleting required history converts an availability problem into correctness failure.

Snapshot installation changes the dependency

A replicated system can compact its log after creating a snapshot that contains state through an included index and term. A lagging replica behind that boundary cannot receive the missing entries individually; it must install the snapshot and then resume from later log entries.

Safe deletion ordering is roughly:

  1. build a snapshot from a well-defined committed state;
  2. persist and validate the snapshot;
  3. publish snapshot metadata atomically;
  4. ensure the replication protocol can install it;
  5. only then remove covered log segments not pinned for another purpose.

Deleting first and hoping snapshot construction finishes later is unsafe.

Fencing leaders and writers

Durable logging does not prevent two processes from believing they are the writer. Replicated systems use terms, epochs, or fencing tokens so an old leader cannot continue publishing after a new leader is elected.

Every append, acknowledgement, and commit advancement should be interpreted within an epoch. A delayed response from an earlier epoch must not advance the current leader’s state. Storage layers may also persist the accepted epoch to reject stale writers after restart.

Recovery and retention checklist

Local WAL

  • Are records length-framed, versioned, and checksummed?
  • Is the difference between appended, written, and durable LSN explicit?
  • Can recovery detect and handle a partial final record?
  • Is replay idempotent or guarded by page/version LSNs?
  • Does group commit wake only transactions covered by a successful flush?
  • Is checkpoint publication ordered before log truncation?

Replication

  • Does the quorum count include the leader where the protocol requires it?
  • Are acknowledgements tied to the advertised durability mode?
  • Is commit advancement scoped to the current term/epoch where required?
  • Are stale leaders fenced?
  • Can a new leader recover the committed prefix without exposing uncommitted suffixes?

Retention

  • Which readers, replicas, snapshots, and backups pin history?
  • Are offsets inclusive or exclusive?
  • Can a lagging replica install a snapshot?
  • What happens under disk pressure?
  • Are segment deletion and metadata updates crash-safe?

The core distinction

The WAL answers: what must survive a local crash?

The commit boundary answers: what has the replication protocol made safe to expose?

The retention boundary answers: what old history is no longer required?

Reliable systems track all three. Naming them separately makes the safety proof reviewable and prevents a fast-moving offset from being mistaken for a durable or discardable one.

Sources and further reading