A write-ahead log is simple to describe: record a change durably before treating the corresponding data mutation as durable. In a replicated system, however, one log position can participate in several different guarantees.

A record may have been appended locally but not flushed. It may be durable on one node but not replicated to a quorum. It may be committed by the replication protocol but not yet applied to the state machine. And even after it is old, it may still be required for replica catch-up or crash recovery.

That is why it is useful to separate four ideas that are often compressed into one word such as “offset” or “watermark”:

  1. write position — how far the local log has been appended;
  2. durable position — how far the local log is known to survive the process or machine failure covered by the durability contract;
  3. commit index — how far the replication protocol considers entries committed;
  4. retention watermark — how far old log data may be removed without violating recovery or replication requirements.

The names differ across systems. The invariants matter more than the vocabulary.

Write-ahead logging is a durability protocol

Suppose a storage engine receives a command that changes key k from old to new. Updating an in-memory tree first and planning to persist it later creates an obvious failure window:

mutate memory
          |
          | process crashes here
          v
      persist later

After restart, the acknowledged state is gone.

A WAL reverses the dependency:

encode log record
          |
      append record
          |
      make record durable according to policy
          |
      acknowledge / apply

The durable log becomes the recovery source. The main data structure can be flushed or checkpointed asynchronously because replay can reconstruct changes that were committed after the last checkpoint.

A production record usually needs enough framing to distinguish a complete record from a torn or corrupt tail. A conceptual layout is:

+----------+----------+----------+------------------+
      | length   | checksum | sequence | command payload  |
      +----------+----------+----------+------------------+

On recovery, the engine scans records in order, validates framing and checksums, and stops or repairs according to the system's corruption policy.

write() is not the same as durable

A successful operating-system write normally means bytes have been accepted by the kernel. It does not automatically mean the storage device has made them durable against every failure the application cares about.

A useful local state model is:

memory state
          |
          v
      appended to OS-visible file  ---- write position
          |
          v
      flushed according to policy  ---- durable position

The actual persistence primitive may be fsync, fdatasync, direct I/O, a database-specific group-commit mechanism, or something else. Hardware caches and filesystem semantics matter. The application must define the failure model its durability claim covers.

Group commit is a common optimization. Instead of forcing one persistence operation per request, several records can share one flush:

r1 ----+
      r2 ----+--> one durable flush --> acknowledge r1..rN
      r3 ----+

This trades a small amount of batching latency for much higher throughput.

Replication introduces another boundary

Now put the WAL behind a replicated state machine.

A leader may have entries that followers have not yet received:

index:        41 42 43 44 45 46
      leader:       ✓  ✓  ✓  ✓  ✓  ✓
      follower A:   ✓  ✓  ✓  ✓  ✓
      follower B:   ✓  ✓  ✓  ✓
      follower C:   ✓  ✓  ✓

The leader's local write position is 46. That does not mean 46 is globally committed.

Replication protocols keep per-replica progress—often called a match index or acknowledged offset—and derive a commit index from their own quorum and term rules.

For a Raft-like protocol, “replicated to a majority” is part of the commit rule, but it is not sufficient to summarize the full algorithm. Current-term restrictions and leadership semantics are important for safety. Other replicated logs define high-water marks differently.

The safe general statement is:

The commit index is the protocol-defined prefix whose entries are safe to expose as committed state according to that protocol's replication rules.

Do not infer it from a local WAL position alone.

Committed and applied are also different

Even after an entry is committed, a state machine may not have applied it yet.

log end        = 46
      commit index   = 44
      applied index  = 42

Entries 43 and 44 are committed but still waiting to be reflected in the local materialized state.

This separation matters for reads. A server answering directly from the state machine cannot claim to reflect commit index 44 if it has only applied through 42.

A compact progress structure makes the distinctions visible:

#[derive(Debug, Clone, Copy)]
      struct LogProgress {
          appended: u64,
          durable: u64,
          committed: u64,
          applied: u64,
      }
      
      impl LogProgress {
          fn is_valid(&self) -> bool {
              self.applied <= self.committed
                  && self.committed <= self.appended
                  && self.durable <= self.appended
          }
      }

The exact ordering between durable and committed depends on the replication and acknowledgement policy. Some systems require local durability before an entry participates in a commit; others define durability at the replicated-system level. The structure is useful because it forces that contract to be explicit instead of hiding everything behind one offset.

Why “high-water mark” is ambiguous

Many systems use a term such as high-water mark, but it is not a universal distributed-systems primitive with one definition.

Depending on the system it may mean:

  • the highest committed log offset;
  • the highest replicated offset visible to consumers;
  • the highest processed event sequence;
  • the latest timestamp considered complete;
  • a resource-allocation threshold.

So documentation should say what the mark protects rather than rely on the name.

For a replicated log, I prefer the more specific term commit index when I mean the consensus boundary.

Retention needs the opposite question

The commit index moves forward as new work becomes safe to expose.

Log retention asks a different question:

How far back must history remain available?

Deleting everything below the commit index is usually wrong. Old entries may still be needed by:

  • a follower that is catching up;
  • a recovery process after a checkpoint;
  • a change-data-capture consumer;
  • an audit or replication stream;
  • a snapshot-transfer protocol;
  • an operator-defined retention window.

A retention watermark is therefore a minimum safe truncation boundary, not simply “the oldest committed entry.”

Conceptually:

snapshot covers through          1000
      slowest required replica needs    940
      CDC consumer needs                970
      policy time-window floor          920
      
      safe truncation boundary = min(required boundaries)
                               = 920

The exact calculation depends on what the system promises to preserve.

Checkpoints let the log forget

A WAL grows forever unless some other durable representation absorbs its history.

Suppose a checkpoint contains state through log index 1,000. If every recovery path can start from that checkpoint, entries much older than 1,000 no longer need to be retained for local crash recovery.

WAL:       1 ------------------------------ 1200
      checkpoint:                    ^ 1000
      
      local recovery requires:       1001 .. 1200

But replication may still need an older entry. That is why truncation uses the most conservative active requirement, not the checkpoint alone.

A lagging replica can be handled in two broad ways:

  1. keep old log segments until it catches up;
  2. declare it too far behind and reseed it from a snapshot/checkpoint.

The second option prevents one unhealthy replica from pinning unbounded retention.

Segment the WAL

Deleting bytes from the middle of one giant append-only file is awkward. Real logs are usually segmented:

000001.log   indexes 1..10000
      000002.log   indexes 10001..20000
      000003.log   indexes 20001..30000

Once the retention watermark passes the end of a segment, the entire segment becomes eligible for deletion.

That makes retention a metadata decision followed by coarse-grained file cleanup rather than constant rewriting of one log file.

Retention is a safety rule before it is a disk-space rule

It is tempting to treat cleanup as housekeeping: “disk is getting full, remove old logs.” In a distributed database, cleanup changes the recovery surface.

Deleting too early can make a follower impossible to catch up incrementally. Deleting the only history after a checkpoint can break crash recovery. Deleting data still needed by an external consumer can create an irreversible gap.

A safe cleanup loop therefore asks:

What is the oldest log position still required by every guarantee I promise?

Only segments strictly older than that boundary are candidates for deletion.

Slow consumers create retention pressure

The downside of conservative retention is that one stalled participant can make the log grow without bound.

That is not a reason to weaken the watermark calculation silently. It is a reason to define policy.

For example:

  • replicas may be reseeded after exceeding a lag threshold;
  • CDC consumers may have a documented maximum retention SLA;
  • snapshots may be generated more aggressively when lag is large;
  • operators may receive alerts on retained bytes, oldest required offset, and lag age.

The policy decides when a participant stops being a requirement. The cleaner should not make that decision on its own.

Observe every boundary separately

Useful metrics include:

  • current append position;
  • durable position and flush latency;
  • commit index;
  • applied index and apply lag;
  • per-replica match index;
  • oldest retained index;
  • retention watermark;
  • WAL bytes on disk;
  • age and size of the oldest segment;
  • checkpoint index and checkpoint age.

A single “replication lag” number hides too much. A node can be network-caught-up but slow to apply, or locally appended but not durably flushed.

A complete mental model

The whole lifecycle looks like this:

client command
           |
           v
      append WAL -------------------------- write position
           |
           v
      satisfy local durability policy ----- durable position
           |
           v
      replicate under protocol rules
           |
           v
      advance commit boundary ------------- commit index
           |
           v
      apply to materialized state --------- applied index
           |
           v
      checkpoint / snapshot state
           |
           v
      advance safe retention boundary ----- retention watermark
           |
           v
      delete obsolete log segments

The important lesson is not the terminology. It is that each boundary answers a different question:

  • Was the record written?
  • Will it survive the promised failure?
  • Has the replicated system committed it?
  • Has this node applied it?
  • Can old history now be removed safely?

Treating those as separate invariants makes storage and replication code much easier to reason about—and makes failures far easier to diagnose.