io_uring is Linux’s modern completion-oriented asynchronous I/O interface. The important idea is not simply that it is “faster async I/O.” It gives user space shared submission and completion rings so applications can describe operations in batches and collect results with fewer transitions through the traditional one-syscall-per-operation interface.

The design changes the ownership model of I/O. With a normal blocking read, a buffer is borrowed only for the duration of the call. With an asynchronous submission, the kernel may still need the file descriptor, address, and metadata after the submitting function returns. A safe Rust wrapper therefore has to model in-flight ownership, not merely expose the raw kernel structs.

The two rings

At a high level:

user space                                  kernel
      
      prepare SQEs
          |
          v
      +--------------------+      submit      +------------------+
      | Submission Queue   | ---------------->| execute requests |
      |   SQE, SQE, SQE    |                  +------------------+
      +--------------------+                           |
                                                       v
      +--------------------+     completions   +------------------+
      | Completion Queue   | <---------------- | finished results |
      |   CQE, CQE, CQE    |                  +------------------+
      +--------------------+
          |
          v
      match user_data to in-flight operation

A Submission Queue Entry (SQE) describes work: operation code, file descriptor, offsets, buffer information, flags, and a user_data field chosen by the application.

A Completion Queue Entry (CQE) reports the result. Its user_data lets the application associate the completion with the state it stored when the request was submitted. res is the operation result: for many operations a non-negative value is a count or returned descriptor, while a negative value represents a negated errno.

That last detail is easy to mishandle when working below a wrapper library. A CQE does not use the normal libc convention of returning -1 and setting thread-local errno.

Why shared rings help

Traditional synchronous I/O often looks like:

prepare one request
          -> system call
              -> kernel work
          <- return
      prepare next request

io_uring allows a program to place multiple descriptions into the submission ring and notify the kernel about a batch. Completions accumulate in another ring.

The potential wins come from several places:

  • batching submissions and completions;
  • reducing system-call frequency;
  • avoiding repeated setup for registered resources;
  • keeping a larger number of operations in flight;
  • linking operations so the kernel can carry a small workflow forward;
  • using specialized modes such as SQ polling where appropriate.

None of these makes an application automatically faster. If storage latency, application serialization, memory bandwidth, or lock contention is the bottleneck, replacing one I/O API may do little.

A Rust-level example

The io-uring crate exposes the kernel interface while staying close to the underlying model.

use io_uring::{opcode, types, IoUring};
      use std::{fs::File, os::fd::AsRawFd};
      
      fn read_at(file: &File, buffer: &mut [u8], offset: u64) -> std::io::Result<i32> {
          let mut ring = IoUring::new(8)?;
      
          let entry = opcode::Read::new(
              types::Fd(file.as_raw_fd()),
              buffer.as_mut_ptr(),
              buffer.len() as _,
          )
          .offset(offset)
          .build()
          .user_data(1);
      
          unsafe {
              ring.submission()
                  .push(&entry)
                  .map_err(|_| std::io::Error::other("submission queue full"))?;
          }
      
          ring.submit_and_wait(1)?;
      
          let cqe = ring
              .completion()
              .next()
              .ok_or_else(|| std::io::Error::other("missing completion"))?;
      
          Ok(cqe.result())
      }

The unsafe boundary is meaningful. The kernel will use the buffer pointer after the SQE is pushed. The caller must ensure the memory remains valid and is not aliased in an invalid way until completion.

A higher-level runtime can make that ownership rule easier to uphold by moving an owned buffer into the operation and returning it with the result.

Submission queue mechanics

The submission side is a producer/consumer ring with indices shared between user and kernel space. Conceptually:

head ------------------------------ tail
              entries ready for kernel

User space fills SQEs, places their indices into the submission queue, and advances the tail with the ordering required by the ABI. The kernel consumes entries and advances the head.

The actual memory-ordering rules matter. Hand-implementing the rings with raw atomics is an educational exercise, but it is not where I would spend complexity in production. The maintained Rust crate already encodes the ABI layout and synchronization requirements.

The engineering lesson is broader: shared memory removes a syscall boundary only by introducing a concurrency boundary. Correct visibility between producer and consumer becomes part of the protocol.

Completion is a separate phase

Once a request is submitted, the application should treat its associated resources as in flight until a completion says otherwise.

A useful state machine is:

Prepared
         |
         v
      Submitted  ----> kernel owns temporary use of resources
         |
         v
      Completed
         |
         v
      resource can be reused / freed

This is where Rust ownership can help enormously. If a runtime represents Submitted by moving the buffer into an internal operation table, ordinary application code cannot accidentally drop or mutate it before the CQE arrives.

If an API instead exposes raw pointers, the caller owns that proof manually.

user_data is your correlation key

An application may have thousands of concurrent requests. Completion order need not match submission order.

submit: A B C D
      complete: C A D B

The user_data field exists so each CQE can be mapped back to its request state. A common design is to store an integer operation ID that indexes an in-flight table rather than trying to encode a raw pointer directly.

The table may own:

  • the buffer;
  • operation-specific state;
  • a task waker or callback;
  • timeout/cancellation metadata;
  • tracing context.

Only after processing the completion should that slot be recycled.

Registered buffers and files

Repeatedly describing the same resources has overhead. io_uring supports registering files and buffers so the kernel can retain metadata across operations.

This can reduce per-I/O work for stable resource pools, but it changes lifecycle management. A registered buffer cannot simply be reallocated or dropped while the ring still refers to it. Registered-file tables similarly need explicit updates when descriptors change.

Use registration when the workload has long-lived, reusable resources and measurements show the setup cost matters. For highly dynamic workloads, the bookkeeping can outweigh the benefit.

Fixed buffers, provided buffers, and buffer selection

Network servers often do not know which receive buffer should be used for the next incoming message. Buffer-selection features let the application provide a group of buffers and allow the kernel to choose one for an operation.

That can make receive paths more efficient, but a new invariant appears: the application must not return a buffer to the pool until every consumer of the completion has finished with it.

A good Rust abstraction should make the buffer’s state explicit:

free -> provided -> selected/in-flight -> application-owned -> free

The optimization is only correct if no two states believe they exclusively own the same memory.

Linked operations

SQEs can be linked so a later operation depends on an earlier one. This supports patterns such as:

read -> write
      request -> timeout
      open -> read -> close

without returning to user space between every step.

Links are powerful, but they also move control flow into the kernel submission graph. Error propagation and cancellation semantics must be understood: a failed operation can affect later linked entries, and timeout behavior differs depending on how the chain is constructed.

Use links for a workflow that is genuinely I/O-shaped and well-defined, not as a replacement for all application state machines.

Polling modes are workload-specific

io_uring offers polling options that can reduce wake-up overhead by dedicating CPU resources to observing the rings or devices.

Polling can improve latency for high-rate workloads on suitable hardware. It can also burn CPU while idle. A server with bursty traffic and tight resource budgets may be worse off than with normal interrupt-driven completion.

This is a classic systems trade-off: trade CPU residency for lower coordination latency. Benchmark it under the actual arrival pattern.

Cancellation is not time travel

Asynchronous code often assumes that cancelling a future means the underlying operation never happened. That is unsafe as a general model.

By the time cancellation is requested:

  • the SQE may not have been consumed yet;
  • the kernel may be executing it;
  • it may already have completed while the CQE is still unread.

A cancellation request is another operation with race semantics. Application protocols must tolerate the possibility that the original operation completed.

This matters especially for writes and other side effects. Idempotency and higher-level transaction semantics cannot be delegated to the I/O runtime.

Multishot operations

Some io_uring operations can produce multiple completions from one submission. This is useful for patterns such as accepting many connections or receiving repeated events without constantly resubmitting the same request.

That changes the usual one-SQE/one-CQE assumption. A multishot operation remains alive until a completion indicates that more results will not follow or the operation is cancelled.

An in-flight table must therefore model a stream of completions rather than freeing state after the first CQE.

Where higher-level Rust runtimes help

Most applications should not manipulate SQEs and CQEs directly. A runtime can provide:

  • owned-buffer APIs;
  • futures that wake on completion;
  • operation tables and IDs;
  • cancellation handling;
  • buffer pools;
  • accept/read/write abstractions;
  • integration with task scheduling.

The important question when evaluating a wrapper is how faithfully it models completion-based ownership. A pleasant async API that hides the lifetime of in-flight memory can be dangerous; a good one should make invalid reuse difficult or impossible.

Performance: benchmark the complete path

io_uring is most compelling when an application has a large amount of concurrent I/O and can take advantage of batching or reduced setup costs.

Measure at least:

  • operations per second;
  • p50/p95/p99 latency;
  • system calls per operation;
  • CPU utilization per core;
  • queue depth;
  • submission batch size;
  • time from submission to completion;
  • time a completion waits before user space processes it;
  • buffer-pool pressure;
  • storage or network device saturation.

A ring with deep queues can increase throughput while making tail latency worse. An application can also submit faster than the downstream device can serve, turning the ring into another queue that hides overload.

Backpressure still matters.

The deeper model

The easiest way to understand io_uring is not as an async version of read and write, but as a small shared-memory command/completion protocol between user space and the kernel.

application owns request state
              |
              v
      publish command through SQ
              |
              v
      kernel executes asynchronously
              |
              v
      publish result through CQ
              |
              v
      application reclaims request state

That model explains the performance opportunities and the safety challenges at the same time.

Rust fits the interface well because ownership is exactly the question a completion API must answer: who is allowed to touch this resource while an operation is in flight, and when can it be reclaimed?

Once that invariant is clear, the rings themselves become much easier to reason about.