ClickHouse is often described with a list of performance techniques: columnar storage, vectorized execution, compression, sparse indexes, parallelism, and distributed aggregation. The source is more useful when those terms are connected to concrete data structures and execution boundaries.

This article is a map for reading upstream ClickHouse, not a substitute for the code. Internal interfaces and file locations are not stable public APIs, so pin a commit before using this guide for an investigation.

The central idea is that performance does not come from one “fast” class. It comes from preserving columnar batches and useful ordering across storage, planning, and execution while avoiding unnecessary reads, conversions, synchronization, and data movement.

1. Columns are the unit of computation

Start with src/Columns/IColumn.h. IColumn is the polymorphic boundary for column values. Concrete implementations encode different physical layouts:

  • ColumnVector<T> stores fixed-width values contiguously;
  • ColumnString separates character bytes from offsets;
  • ColumnNullable pairs a nested column with a null map;
  • ColumnArray pairs nested values with array offsets;
  • ColumnLowCardinality represents repeated values with dictionary-style indirection.

The logical SQL type is a separate concern under src/DataTypes/. This separation lets the same execution machinery operate on column containers while data types control serialization, casts, default values, and related semantics.

Contiguous fixed-width columns make tight loops, SIMD-friendly operations, and compression possible. That does not mean every expression is automatically vectorized by hardware. It means ClickHouse evaluates many operations over a batch rather than paying virtual-call, branch, and allocation costs per row.

Two containers appear repeatedly:

  • Block associates columns with names and data types. It is also used as a header describing a stream's schema.
  • Chunk is the lighter runtime payload passed between processors: columns, a row count, and optional chunk metadata. Names and types are known from the connected port headers.

That distinction is worth remembering while tracing a query. Planning and schema negotiation often talk in Block headers; the hot execution path moves Chunk objects.

2. MergeTree turns inserts into immutable parts

Most production ClickHouse tables use an engine in the MergeTree family. The relevant code begins under src/Storages/MergeTree/.

An insert creates a data part whose rows are ordered by the table's sorting key. Parts are immutable after they are committed. Background work merges compatible parts into larger parts, and mutations rewrite affected data rather than editing values in place.

Immutability simplifies concurrent reads: a query can hold references to a stable set of parts while background merges create replacements. The trade-off is write amplification, temporary disk usage, and operational pressure when small parts arrive faster than the merge pool can consolidate them.

A useful mental model is:

incoming rows
          |
          v
      sort by table key
          |
          v
      write immutable part
          |
          +-------------------+
          |                   |
          v                   v
      foreground reads   background merges
                              |
                              v
                       larger immutable parts

The storage engine is therefore not an in-place B-tree. It is a lifecycle of immutable part versions plus metadata that decides which parts form a consistent table view.

3. A part is a collection of column streams and metadata

A MergeTree part contains the physical column data plus metadata needed to avoid reading all of it.

Depending on format, data type, codecs, and settings, the exact files vary. The important concepts are stable:

  • compressed column streams;
  • marks that locate granules in those streams;
  • primary-key index entries sampled at granule boundaries;
  • optional skip-index data;
  • checksums and part metadata;
  • information about row counts, min/max values, and sorting-key ranges.

Reading ClickHouse source becomes easier once you stop imagining a part as one row-oriented file. A logical column can itself serialize into multiple streams. Nullable values, arrays, strings, and nested structures all have different physical components.

The read path reconstructs the requested columns from those streams only for the ranges the planner and storage engine decide are necessary.

4. The primary index is sparse by design

ClickHouse's MergeTree primary index is not a dense pointer per row. It stores sorting-key values at granule boundaries.

Conceptually:

part ordered by primary/sorting key
      
      rows 0 ............ 8191      mark 0: key ~= A
      rows 8192 ......... 16383     mark 1: key ~= D
      rows 16384 ........ 24575     mark 2: key ~= M
      ...

The exact granule size is controlled by settings and can vary with adaptive granularity. The important property is that the index narrows the candidate ranges of marks rather than identifying one row directly.

For a predicate compatible with the sorting key, the engine can use key conditions to exclude large mark ranges before decompression.

That explains both the strength and the limitation of the index:

  • it is tiny enough to keep and search cheaply;
  • it is extremely effective when query predicates align with ordering;
  • it is not a general secondary index that can locate arbitrary values with point-lookup precision.

Schema design and ordering are therefore part of query performance.

5. Marks connect logical ranges to physical reads

Once the engine has selected mark ranges, it still needs to find the corresponding compressed data.

Mark files record positions into column streams. A reader can seek near the compressed block associated with a granule rather than scanning the whole part.

The path is roughly:

predicate
         |
         v
      key condition / skip indexes
         |
         v
      selected mark ranges
         |
         v
      seek into requested column streams
         |
         v
      decompress blocks
         |
         v
      produce column batches

This is why “ClickHouse only reads the columns you select” is incomplete. It tries to reduce work along two axes:

  1. projection: which column streams are needed;
  2. pruning: which ranges within those streams are needed.

Good performance comes from both.

6. Query execution becomes a plan and then a processor pipeline

A SQL query does not jump directly from parser to table reads.

The exact analyzer/planner implementation evolves, but a useful source-reading model is:

SQL text
         |
         v
      parse syntax
         |
         v
      analyze names, types, expressions
         |
         v
      build / optimize query plan
         |
         v
      turn plan steps into processors
         |
         v
      execute a connected pipeline of Chunks

Plan steps describe logical/physical transformations such as reading, filtering, expression evaluation, aggregation, sorting, joins, unions, limits, and exchanges.

The processor layer implements the runtime dataflow. Processors expose input and output ports; an executor schedules processors when inputs, outputs, and asynchronous work make them ready.

This is different from a simple iterator model in which one operator calls next() on its child recursively. The pipeline can express multiple parallel streams, fan-in, fan-out, resizing, and asynchronous sources.

7. Chunk keeps the hot path lighter than Block

The Block/Chunk split makes more sense in the processor pipeline.

A connected port already knows its schema from a header. Carrying names and types with every runtime batch would repeat metadata that rarely changes.

So execution can move something closer to:

Chunk {
          columns,
          num_rows,
          metadata
      }

between processors while headers describe the shape.

That is a small example of a broader ClickHouse pattern: keep rich metadata where it helps planning and validation, and keep the data path as compact as possible.

8. Aggregation is stateful and parallel

For a query such as:

SELECT country, sum(revenue), count()
      FROM events
      GROUP BY country

each execution stream can build aggregate states for the groups it observes. Those states can later be merged.

Conceptually:

stream 1 -> local aggregate states --+
                                            |
      stream 2 -> local aggregate states --+--> merge --> finalize
                                            |
      stream 3 -> local aggregate states --+

The engine's aggregate-function interface separates adding input rows, merging states, serializing states, and inserting final results. That separation is exactly what makes parallel and distributed aggregation possible.

Memory behavior depends heavily on group cardinality and key representation. A low-cardinality group-by may stay compact; an unexpectedly high-cardinality key can grow large hash tables and hit query memory limits.

ClickHouse contains multiple aggregation strategies and specialized hash-table paths. The right way to understand which one a workload uses is to trace the plan, settings, key types, and runtime profile rather than assume one universal implementation.

9. Distributed queries extend the same partial-state model

A Distributed table or distributed query adds another boundary. The initiating server sends work to shards, shards execute local plans, and results return to the initiator for further processing.

For aggregation, the efficient path is usually to send partial aggregate states rather than raw rows when semantics allow it:

initiator
         |
         +----------+----------+
         |          |          |
         v          v          v
       shard A    shard B    shard C
       partial    partial    partial
       states     states     states
         |          |          |
         +----------+----------+
                    |
                    v
             initiator merge/finalize

This reduces network volume dramatically when many input rows collapse into relatively few groups.

The source-reading lesson is to trace both sides of the distributed query. The initiator's profile may show remote waiting or final merge cost, while shard-level logs reveal reads, decompression, filtering, and local aggregation. Network serialization and compression are also part of the execution budget.

10. Compression is integrated with column layout

ClickHouse compression works well because columns tend to contain values with similar statistical structure, and because sorting can cluster related values.

Different streams can use codecs suited to their representation. But compressed bytes are not directly useful to most expressions; selected blocks are decompressed into column objects before computation.

That means a selective query's cost can include:

read compressed bytes
          +
      decompress selected blocks
          +
      materialize column representation
          +
      evaluate expressions

A query can be I/O-light but CPU-heavy because decompression or expression evaluation dominates. Looking only at bytes read can therefore misdiagnose performance.

11. Background merges compete with foreground work

MergeTree performance depends on background maintenance keeping up with ingestion.

Too many small parts create several costs:

  • more metadata and file handles;
  • more ranges to consider during reads;
  • more merge work;
  • greater temporary disk usage;
  • possible insert throttling or rejection when part counts cross configured limits.

Merges consume CPU and disk bandwidth that foreground queries may also need. A production investigation should correlate query latency with merge activity, part counts, and disk saturation rather than treating them as independent systems.

system.parts and system.merges are useful starting points for connecting the storage lifecycle to live behavior.

12. Memory tracking and caches are observable policy

ClickHouse attaches allocations to query and thread memory trackers, enforcing per-query or server-level limits. A limit failure is not necessarily a leak: it may reflect hash-table growth, sorting, joins, decompressed working sets, or too many concurrent queries.

Caches reduce repeated work but are not correctness mechanisms. Uncompressed data, marks, query results, and filesystem pages have different lifetimes and eviction behavior. A benchmark that runs once measures a different system from a warmed repeated query.

Use system tables and profile events to connect source concepts to evidence:

  • system.query_log and system.query_thread_log for timings and resource counters;
  • system.parts for part counts, sizes, and lifecycle;
  • system.merges for active background merges;
  • system.processes for running queries;
  • EXPLAIN PIPELINE for processor topology.

The exact available columns and events depend on the release and configuration.

13. A disciplined way to read the repository

The repository is too large for productive top-down reading. A better method is:

  1. pin the ClickHouse commit and build configuration;
  2. choose one small query with a known table definition;
  3. record EXPLAIN, profile events, selected parts, and relevant settings;
  4. locate the corresponding plan steps;
  5. follow the pipeline builder into concrete processors;
  6. follow the read source into MergeTree range selection, marks, and column readers;
  7. change one predicate, grouping key, or setting and compare the plan and counters.

Search by class and include path rather than assuming directory names. Add temporary logging or use a debugger only after the observable plan narrows the path. Many apparent source mysteries are actually alternate execution paths selected by a setting, part format, analyzer version, or data type.

Putting the layers together

For a selective aggregate query, the path is approximately:

SQL
        -> parse and analyze
        -> optimize a query plan
        -> choose MergeTree parts and mark ranges
        -> read and decompress selected column streams
        -> emit Chunks through processor ports
        -> update per-stream aggregate states
        -> merge states locally and, if distributed, at the initiator
        -> finalize and format the result

Every arrow is a boundary where work can be avoided or introduced. Ordering and sparse indexes reduce read ranges. Projection reduces column streams. Batches amortize execution cost. Specialized aggregate methods control state updates. Parallel pipelines use available cores. Partial states reduce network traffic.

The rigorous explanation of ClickHouse performance is therefore conditional: a particular schema and query allow these layers to cooperate. The source code is valuable because it shows the conditions—and the fallback paths—behind the headline.