Linux traditionally exposes I/O through one system call per operation: read, write, accept, fsync, and so on. Readiness APIs such as epoll reduce blocking, but applications still submit work through individual system calls and must translate “this descriptor is ready” into an actual operation.

io_uring adds a shared-memory submission queue and completion queue between userspace and the kernel. An application describes operations in submission queue entries (SQEs), the kernel performs them, and completion queue entries (CQEs) report results. Batching and shared rings can reduce system-call traffic; operation-based completion also fits files and other resources that do not behave like readiness-driven sockets.

That summary is simple. Correct buffer lifetimes, cancellation, backpressure, and kernel-version differences are where real implementations become interesting.

The two rings

An io_uring instance contains:

  • a submission queue (SQ) of indexes pointing to SQEs;
  • a completion queue (CQ) of CQEs written by the kernel;
  • shared head and tail counters coordinating producer and consumer progress;
  • an SQE array describing opcodes, file descriptors, addresses, lengths, flags, and application data.

The broad loop is:

  1. reserve an SQE;
  2. populate every field required by the opcode;
  3. publish it to the SQ;
  4. submit queued work, possibly asking the kernel to wait for completions;
  5. consume CQEs and advance the CQ head.

The kernel and application are concurrent participants. Libraries hide the required atomic loads, stores, and memory ordering, but they cannot decide what an application-owned pointer refers to or how long it remains valid.

A low-level Rust read

The io-uring crate provides typed opcode builders while remaining close to the kernel API.

use io_uring::{opcode, types, IoUring};
use std::{fs::File, io, os::fd::AsRawFd};

fn main() -> io::Result<()> {
    let mut ring = IoUring::new(32)?;
    let file = File::open("input.dat")?;
    let mut buffer = vec![0_u8; 4096];

    let read = opcode::Read::new(
        types::Fd(file.as_raw_fd()),
        buffer.as_mut_ptr(),
        buffer.len() as u32,
    )
    .offset(0)
    .build()
    .user_data(1);

    // Safety: the fd and buffer referenced by this SQE remain valid and
    // the buffer is not accessed again until the completion is consumed.
    unsafe {
        ring.submission()
            .push(&read)
            .map_err(|_| io::Error::other("submission queue is full"))?;
    }

    ring.submit_and_wait(1)?;

    let cqe = ring
        .completion()
        .next()
        .ok_or_else(|| io::Error::other("missing completion"))?;

    if cqe.user_data() != 1 {
        return Err(io::Error::other("unexpected completion"));
    }

    let result = cqe.result();
    if result < 0 {
        return Err(io::Error::from_raw_os_error(-result));
    }

    let bytes_read = result as usize;
    println!("read {bytes_read} bytes");
    Ok(())
}

The unsafe block is not about pushing to a vector in general. The SQE stores raw references that the kernel may use after submission. Rust cannot prove that file stays open, that buffer is not reallocated, or that no code reads it while the kernel writes it.

A production abstraction normally transfers ownership of the buffer into an in-flight operation and returns it with the result. That is the design used by higher-level Rust runtimes.

Completion semantics

A CQE contains an application-defined user_data, a result, and flags. Several details matter:

  • a negative result is -errno, not a Rust io::Error;
  • a non-negative result often reports bytes processed, and partial completion is normal;
  • completion does not imply application-level durability unless the operation itself provided it—for example, a completed buffered write is not an fsync;
  • some multishot operations produce multiple CQEs for one SQE, with flags indicating whether more may follow;
  • CQEs may arrive in a different order from submissions unless an ordering mechanism is requested.

user_data should identify both the logical operation and its generation. Reusing a small integer too early can let a late completion be mistaken for newer work.

Submission does not remove backpressure

An application cannot push unlimited work. Queue entries are finite, kernel resources are finite, and each outstanding operation may retain a buffer or file reference.

Robust code needs explicit limits:

  • reserve capacity before accepting more upstream work;
  • stop producing when the SQ is full;
  • cap bytes as well as operation count;
  • drain the CQ promptly;
  • decide what happens when one operation in a linked chain fails;
  • expose queue depth and completion latency as metrics.

Increasing ring depth can improve device utilization until it increases memory retention and tail latency. The correct depth is a property of the device and workload, not a universal constant.

Registered files and buffers

Applications can register file descriptors or memory regions with a ring. Later SQEs refer to table indexes rather than making the kernel repeatedly acquire the same resources.

Registration can reduce per-operation setup and is especially relevant at high IOPS. It does not automatically make an operation zero-copy. A registered read buffer is still a destination for data, and a registered send buffer may still be copied by the networking stack depending on the opcode and platform path.

Registration also introduces lifecycle policy:

  • registered memory is pinned or otherwise accounted for and should be bounded;
  • a fixed-file table needs a safe replacement protocol;
  • buffer pools need ownership states such as free, selected by kernel, in flight, and returned;
  • failures during registration or update need a fallback path.

Provided-buffer rings let the kernel select an available buffer for receive-like operations. This reduces per-request buffer selection overhead but makes completion handling responsible for recycling the selected buffer ID.

Polling modes are specialized tools

SQPOLL lets a kernel thread poll the submission queue, reducing the need for submission system calls while it remains active. It consumes CPU and may require permissions or resource limits depending on the kernel configuration.

IOPOLL asks for polled I/O completion on devices and files that support it, generally with direct I/O constraints. It is not a drop-in optimization for arbitrary buffered files or sockets.

Both modes trade CPU and operational complexity for latency. They should be selected from measurements on the deployed kernel and storage stack.

Cancellation is a race, not time travel

An async cancel request can race with the original operation. Possible outcomes include:

  • cancellation wins and the original completes as canceled;
  • the operation completes first and cancellation reports that it did not find a target;
  • a multishot operation has already produced some completions before it stops;
  • linked operations fail or continue according to their link type.

Dropping a Rust future therefore cannot make the kernel forget a pointer. A runtime must retain operation state and buffers until the kernel has produced the terminal completion, even if no task is waiting for the result.

File-descriptor reuse creates another hazard. Closing a descriptor while an operation still refers to its integer value can allow the number to be reused for a different resource. Safe runtimes keep resources logically alive or use fixed-file indirection until in-flight work is resolved.

A higher-level ownership model with tokio-uring

tokio-uring combines a Tokio-compatible runtime with resource types backed by io_uring. Its file API takes ownership of a buffer for an operation and returns the buffer with the result:

use tokio_uring::fs::File;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    tokio_uring::start(async {
        let file = File::open("input.dat").await?;
        let buffer = vec![0_u8; 4096];
        let (result, buffer) = file.read_at(buffer, 0).await;
        let bytes_read = result?;
        println!("{:?}", &buffer[..bytes_read]);
        Ok(())
    })
}

Passing the buffer by value prevents application code from accessing it while the kernel owns the operation. This is a good example of Rust making an external protocol safer without pretending the protocol is synchronous.

An io_uring runtime is not automatically a faster replacement for an existing readiness runtime. Compatibility, scheduling, supported resource types, and ecosystem integration all matter. A mixed system may use a conventional Tokio runtime for most networking and a dedicated ring for a storage path.

Kernel features must be discovered

io_uring arrived in Linux 5.1 and has expanded rapidly. Individual opcodes and flags appeared later, and behavior has been fixed or hardened across kernel releases. Checking only “kernel >= 5.1” is insufficient.

A deployable program should:

  1. probe required opcodes and features during startup;
  2. distinguish required capabilities from optional fast paths;
  3. keep a fallback for older kernels or unsupported filesystems/devices;
  4. test on the same kernel family and security configuration used in production;
  5. pin and review library versions because wrappers evolve with the UAPI.

Kernel support also has a security history. Container runtimes and sandbox policies may restrict io_uring even when the host kernel implements it.

What to benchmark

Compare the entire application, not a single operation in isolation:

  • throughput at several queue depths;
  • p50, p95, and p99 completion latency;
  • CPU time and system calls per request;
  • memory retained by in-flight buffers;
  • behavior under cancellation, timeouts, and overload;
  • direct versus buffered I/O;
  • registered versus unregistered resources;
  • fairness between connections or tenants.

io_uring is most compelling when batching, operation-based completion, and resource reuse fit the workload. Its real contribution is a richer asynchronous I/O interface—not a promise that every operation becomes zero-copy or that every application becomes faster.

Sources and further reading