Distributed Systems · Chapter 07

Type a word, find everything

Databases store data; they don't search it well. This chapter builds a content-search service on Elasticsearch — why a search engine beats a database, how an inverted index turns text into instant lookups, how a query fans out across a cluster with no single point of failure, and the immutable-segment trick that makes near-real-time search possible.

🧭

How to read this

The chapter builds one idea in layers. First why a dedicated search engine (Elasticsearch) beats querying your database directly. Then how it works, from the outside in: the pipeline that feeds it, the inverted index at its core, how a query scatters across a cluster, and finally the counterintuitive choice — immutable indexes — that unlocks near-real-time search. Anything marked [+] is context I added; everything else is rewritten from the six teaching lessons.

Why a search engine, not the database

The goal: let users search videos, articles, or both by keyword — in natural language, with minimal latency, in near real-time (content is searchable as soon as it's persisted). Video data lives in the VideoMetadata table; articles live in the content graph.

The no-brainer architecture is to search those two databases directly — hit VideoMetadata for videos, the content graph for articles, run both in parallel for a mixed query and merge. It works for basic search, but fails at personalized, relevant results: databases are built to store data, not to run complex natural-language queries, and they lack the features search engines have.

🔍

Elasticsearch

Developers originally used Apache Lucene (open-source, Java) for search. Elasticsearch — a document-oriented store built on Lucene — became the industry default for application search and log analytics. It brings natural-language and multilingual search, autocompletion, faceted search, geo-aware search, ML-based ranking of similar content, synonyms, spell check, and more. Its inverted indexing searches massive data with minimal latency.

🏨

The hotel-search example

A user wants a room under $100. The DB has none — but there's one at $102. A relational query returns nothing (it asked for <$100 exactly), and the sale is lost. Elasticsearch's range feature can return results within a business-defined window (say +$10), surfacing the $102 room — and the user might well book it. A game changer, and faster too: no joins at runtime. Website B (Elasticsearch) beats Website A (relational) for search.

The indexing pipeline: keeping search in sync

Both primary databases stream to Elasticsearch in real time via Change Data Capture — and Elasticsearch becomes the source of truth for site search (a derived database). Since data flows continuously, keeping the primary DBs and the derived store consistent is essential — and that's a Kafka queue's job.

🔗

Why Kafka sits in the middle

Suppose a transaction adds a video to VideoMetadata and the Elasticsearch index must update too — but Elasticsearch is down. Without a buffer you'd have to abort the transaction or block waiting. A Kafka queue holds the record until Elasticsearch comes back and updates its index — decoupling the two stores and preserving consistency.

Figure 1 — The indexing pipeline: CDC streams both databases through Kafka into Elasticsearch; a search service queries it and prepares the response. Original diagram after Figures 7.5–7.7.

On the read side, a user's search hits a backend search service that holds the business logic to build Elasticsearch queries, fetches data, then prepares custom objects for display the way the business wants before returning them.

The inverted index: text → instant lookup

Elasticsearch's vocabulary maps to the relational world: document = row, field = column, index = table (a field can hold a list of same-type values; queries run against an index).

An inverted index stores, for each unique word, the list of documents it appears in — which is what makes full-text search fast. Take three video titles:

Lionel Messi Opens the New Season with a Bicycle-Kick Goal
Football World Reacts to the Messi 's Bicycle-Kick
Messi Looking to Bounce Back With PSG and Peak for World Cup

Break each title into terms (tokens), build a sorted list of unique terms, and record which documents each appears in:

messi → [1, 2, 3]
bicycle-kick → [1, 2]
world → [2, 3]
psg → [3]
goal → [1]...

Now searching "Messi" instantly returns documents 1, 2, and 3.

Figure 2 — An inverted index maps each term to the documents containing it, so a keyword lookup is instant. Original diagram after Figure 7.8.

Analyzers, synonyms & querying

Normalization — some terms are uppercase, some lowercase; normalize to a standard form for case-insensitive search.
Synonyms — index "leaped/jumped/pounced" as one term (jump ), shrinking the index and making results more relevant (a search for "jump" also surfaces its synonyms).
Analyzers — built-in ones remove punctuation, standardize case, split on whitespace, detect language, and index accordingly. You can specify custom analyzers, but Elasticsearch is detailed with a steep learning curve — you typically index, then play with the API to see how it analyzes text and computes relevance.
Querying — a Query API runs queries with endless combinations. Elasticsearch computes a relevance score per matched document and returns results in descending order of score by default.

🎓

Interview note [+]

The author notes from experience (building product search at HPE) that interviewers won't expect Elasticsearch minutiae — exact field mapping, which analyzer to pick — if you've no hands-on experience. The details are here to understand the service, not to memorize.

Distributed search behind the scenes

Concept	Meaning
Index	A collection of documents grouped by logic (e.g. a basketball index, a cricket index). You can group unrelated data, but it's not advisable.
Shard	Indexes are stored in shards; an index can span one or more shards.
Type	Further groups data within an index (e.g. a Baseball index with types per tournament). A document is uniquely identified by index + type + id.
Node / cluster	A running instance is a node; nodes form a cluster. The cluster elects a master node for cluster-wide operations (create/delete indices, add/remove nodes).

📡

No single point of failure on reads

The master node isn't involved in document searches. Every node knows where each document lives and can forward the request — so the master never becomes a search-time bottleneck or SPOF. Whichever node the request hits owns gathering the cluster's response and returning it.

Scatter-gather: how a query fans out

A request hits Node 3; the matching documents are spread across all three nodes.
Node 3 broadcasts the request to the other nodes.
Every node runs the search in parallel, keeping a local priority queue ordered by relevance score.
All nodes return document IDs to Node 3, which merges them with its own priority queue and returns the final results to the user.

Figure 3 — Scatter-gather search: the node that receives the request coordinates a parallel search across the cluster and merges relevance-ranked results. Original diagram after Figure 7.9.

Immutable indexes and the segment trick

Here's the counterintuitive core: the inverted index written to disk is immutable — it never changes. Updates don't modify the original; they add new supplementary index segments holding new and updated data. Why give up in-place updates?

No locks Immutable indexes need no locking when many processes read them simultaneously — a big read-performance win.

Warm caches Kernel file cache and system caches don't rebuild on every change.

Cheap compression Large immutable indexes compress easily, cutting disk I/O and the RAM needed to cache them.

The cost You can't add to an index — one naive fix is rebuilding the whole index per write, which is resource-heavy and kills update frequency.

Updating dynamically with segments (from Lucene)

Instead of rebuilding, Elasticsearch appends new segments of recent data to the original, like a log. A query scans the whole log of segments and combines the results. In Lucene, an index is a collection of segments plus a commit point (a file listing all segments). New data flows in-memory buffer → filesystem cache → disk; on a write, pending filesystem-cache writes are flushed to disk to guarantee they're physically written, then the in-memory buffer clears for new data.

🗑️

Deletes and updates without mutation

Since indexes are immutable, old field values can't be overwritten. To discard them during search, the commit point includes a .del file listing data to ignore. Deletions use the same mechanism — mark, don't erase.

Making search near real-time

Segment-based search shrinks the gap between indexing a document and seeing it in results to within minutes. But there's a bottleneck: writing a new segment to disk needs an fsync to guarantee it's physically persisted, and fsync is costly — every one is a performance hit.

⚡

The trick: visible before committed

In Lucene, a file sitting in the filesystem cache can be opened, read, and made visible to search without a commit. Elasticsearch exploits this — segments become searchable without running a commit, slashing the performance cost and enabling near-real-time search. The segments still get fsync 'd to disk eventually (writes don't survive a reboot until fsync'd), but at regular intervals, less often than segment writes — balancing performance against durability.

When the system is write-heavy: merging

Frequent writes → proportionally more segments → more CPU cycles to process them and larger scans per query, slowing search down. Elasticsearch fixes this by auto-merging segments in the background: small segments merge into bigger ones, which merge into even bigger ones — and data marked deleted in .del files is excluded from the new merged segments (so deletes are physically reclaimed here).

🧠

Why learn these internals

Two reasons the chapter gives: (1) knowing how scalable products work helps you optimize your services that build on them; (2) you can borrow their design ideas — the same append-only, sequential, cache-first patterns that make Kafka and Elasticsearch blazing fast can inform your own architecture.

Recap

Search engine >database Databases store, they don't search well. Elasticsearch (on Lucene) adds NLP search, synonyms, ranking, and the range feature (the $102 hotel room) — no runtime joins.

Indexing pipeline CDC streams both databases through a Kafka buffer into Elasticsearch, the search source of truth. Kafka preserves consistency if ES is down.

Inverted index Term → list of documents. Tokenize, normalize (case-insensitive), collapse synonyms; analyzers clean text; queries return by relevance score.

Distributed reads Index → shards; node + cluster + master (not on the read path). Scatter-gather: coordinator broadcasts, nodes search in parallel, merge by score.

Immutable indexes Never mutated — no locks, warm caches, easy compression. Updates append segments; a commit point lists them; a .del file marks removals.

Near real-time Segments visible from filesystem cache without a commit; fsync at intervals; background segment merging keeps write-heavy search fast.

💡

The single idea underneath it all

Search is fast because it refuses to do two expensive things at read time. It won't compute matches — an inverted index pre-answers "which documents contain this word?" before anyone asks. And it won't coordinate writes into reads — indexes are immutable, so readers never wait on a lock and caches never go cold. Every hard part (updates, deletes, freshness, write-heavy load) is then handled by adding immutable segments and merging them in the background, never by mutating what's already there. Do the work ahead of time, and never make the reader pay for the writer.