System Design · Chapter 12

Turning a firehose of data into insight

IoT sensors, social apps, and machines emit data endlessly. Stream processing is the pipeline that collects that flood, standardizes it, analyzes it, and shows it — deciding along the way whether to answer fast or answer accurately.

🧭

How to read this

The chapter follows the life of a data point: why it's generated (data-driven systems) → how it's collected and standardized (ingestion and its layers) → the fundamental fork of real-time vs. batch and why ingestion is hard → where it's used → the pipelines and ETL that move it → the distributed engines that crunch it → and finally the two architectures (Lambda, Kappa) that decide how fast vs. how accurate the answer is. Anything marked [+] is context I added; everything else is rewritten from the eight teaching lessons.

Everything now emits data — constantly

The world is increasingly data-driven, and the Internet of Things (IoT) has given machines a degree of self-awareness — they generate and transmit data online at an unprecedented rate, even making decisions without human intervention. The main sources: industrial and smart-city sensors, drones, smartwatches, phones, and wearable health sensors.

Businesses depend on this flood. Customer data drives plans and projections; understanding user needs and behavior lets companies build better products, make smarter decisions, run sharper ad campaigns, recommend products, and read the market — which loops back into more customer-centric products and loyalty. A second use is tracking service health: the steady "everything is okay" signal from millions of deployed IoT devices tells a business its products are up and functioning. To handle all of it, you need a backend sophisticated enough to gather the meaningful data and archive or purge the rest — and the more data you study, the better your systems evolve.

Ingestion: collecting the flood and making it usable

Data ingestion is the whole process of collecting data streaming in from many sources and getting it ready for the system to process. Data is pulled from IoT devices and other sources, routed through pipelines to different layers, run through algorithms, and eventually archived. A processing setup has several layers:

Figure 1 — The layers of a data-processing setup: collection/standardization → processing → analytics → visualization, with storage and security spanning the flow (plus a query layer). Original diagram after Illustration 1.82.

Collection & standardization. Data arrives heterogeneous — different speeds, sizes, and semantics from web services, social networks, IoT, and industrial machines. It's collected and converted to a uniform, standardized format in the collection/preparation layer to avoid downstream processing issues.
Processing. The standardized data is classified into different flows and routed to different destinations per business requirements.
Analytics. Models run on the data — predictive modeling, statistical analytics, text analytics.
Visualization. The resulting intel is presented to stakeholders, usually in a web dashboard. Kibana is a widely used example.
Storage & security. The storage layer persists data; the security layer keeps it safe as it moves (moving data is highly vulnerable to breaches).

Real-time or batch — and why ingestion is hard

There are two ways to ingest data — in real-time, or in batches at regular intervals — and the business need decides which.

Approach	Best for	Trade-off
Real-time	When time, lives, or money are on the line — medical data from wearables (heartbeat, blood pressure), stock-market events.	Analytics runs on only a limited slice of data, so results are less accurate/holistic.
Batch	Trends over time where instant data isn't critical — e.g. a sport's popularity in a region over a period.	Takes longer, but the entire dataset is considered, so results are more accurate and comprehensive.

⚖️

The tension that drives the whole chapter

Fast vs. accurate. Real-time gives quick insight on partial data; batch gives comprehensive insight but slowly. This exact trade-off is what the Lambda and Kappa architectures (Section 07) are built to resolve.

Why ingesting massive data is genuinely painful

It's slow. Data from each source has a different format, syntax, and metadata — it's heterogeneous, and must be transformed into a standard format (like JSON) the analytics system can understand. That conversion is tedious and resource-heavy; data is staged, processed, authenticated, and verified at every step. Traditional cleansing (legacy ETL ) could take weeks or months and couldn't even do real-time.
It's complex and expensive. The whole flow is resource-intensive and needs a dedicated team. Off-the-shelf tools often fall short, forcing custom builds — LinkedIn had 15 ingestion pipelines and wrote its own tool, Gobblin, in-house to manage them. And because external data sources aren't under your control, their semantics can change and break your backend code.
Moving data is risky. Every hop through a staging area is a breach opportunity, so teams spend extra effort meeting security standards throughout.

🎯

"Real-time" ≠ perfectly accurate

Real-time results aren't as accurate or holistic as batch, because they run continuously over a limited data slice. Batch, by taking the whole dataset, is more accurate — the more time spent studying the data, the better the results.

Where streaming actually gets used

Moving Big Data into Hadoop The most popular use — Big Data from IoT, social apps, and other sources flows through pipelines into Hadoop for analysis.

DB → Elasticsearch Streaming data to Elasticsearch (the de-facto search framework) for indexing; new data delivered async in real time to keep search current.

Log processing Large services on clusters emit massive logs; stream them to a central place (e.g. the ELK stack — Elastic, Logstash, Kibana) to study behavior and track errors.

Real-time event engines LIVE systems like sports — ingest, analyze, and push updates to fans fast, using Kafka + compute frameworks (Storm, Nifi, Spark, Samza, Kinesis).

Data pipelines — and how they beat old ETL

Data pipelines are the core of a data-processing infrastructure: they move data efficiently from point to point and let developers apply filters and business logic to the stream in real time. They run on rules predefined by the engineering team, so the entire flow — extraction, transformation, combination, validation, and the convergence of multiple streams into one — is automated with no manual intervention. What they give you:

Smooth data flow, with no bottlenecks or redundancy.
Filters and business logic applied to streaming data.
Parallel processing by managing multiple streams at once.
Protection against data corruption.

ETL, and why pipelines surpass it

ETL = Extract, Transform, Load. Extract pulls data from one or many sources; Transform converts that heterogeneous data into a standardized format per business rules; Load moves the transformed data into a warehouse or storage for further processing. The ETL flow mirrors data ingestion — the key difference is that ETL moves data in batches, whereas pipelines can stream it in real time, which legacy ETL couldn't do.

Figure 2 — The ETL flow — Extract → Transform → Load — mirrors ingestion but moves data in batches, where modern pipelines stream it in real time. [+] Diagram of the lesson's ETL definition.

Real-time doesn't make batch obsolete — companies use both to get the best of each (the Lambda/Kappa payoff ahead). The popular tools — Apache Flink, Storm, Spark, Kafka — all share one trait: they process data in a cluster, in a distributed environment, via pipelines.

Distributed data processing: divide and conquer

Distributed data processing diverges large amounts of data to many nodes running in a cluster for parallel processing. The nodes execute their allotted tasks in parallel, coordinated by a node-coordinator — Apache ZooKeeper is the common example. Because work is parallel and distributed, the setup is scalable (horizontally and vertically) and highly available, with data replicated across the cluster to prevent loss. The speed win is structural: a distributed system splits tasks across nodes, whereas a centralized system queues them to be processed one by one.

Figure 3 — Distributed processing: a coordinator splits data across parallel worker nodes, then merges their partial results. Original diagram after Illustration 1.83.

Technology	What it is & best use
MapReduce / Hadoop	A programming model for distributed processing: map sorts data by a parameter, reduce summarizes it. Apache Hadoop is its popular open-source implementation — used by Twitter for analytics and Facebook for storing big data. Best for batch.
Apache Spark	Open-source cluster-computing framework; high performance for both batch and real-time in-stream processing. Has a cluster manager (node communication) and distributed storage; integrates with Cassandra, HDFS, MapReduce FS, Amazon S3.
Apache Storm	Distributed stream -processing framework for massive streaming data — real-time analytics, ML, distributed RPC.
Apache Kafka	Open-source distributed stream-processing and messaging platform (Java/Scala, built by LinkedIn). Its storage layer is a distributed, scalable pub-sub message queue. Used for notifications, stream management, activity/metrics monitoring, messaging, log aggregation.

The rule of thumb: Hadoop for batch; Spark, Kafka, and Storm for real-time streaming.

Lambda & Kappa: resolving fast-vs-accurate

Lambda — run both, then merge

Lambda is a distributed data-processing architecture that uses both batch and real-time streaming, then joins their results — beating the latency of batch-only while keeping its accuracy. Batch is slow but comprehensive and accurate; real-time is fast but runs on a small slice, so less accurate. Lambda takes the best of each via three layers:

Batch layer — results from batch processing (accurate, comprehensive).
Speed layer — results from real-time streaming (fast).
Serving layer — combines the batch and speed results before presenting them.

Figure 4 — Lambda: batch and speed layers run in parallel and the serving layer merges them. Original diagram after Illustration 1.84.

Kappa — one pipeline for everything

In Kappa, all data flows through a single streaming pipeline — unlike Lambda's separate layers that converge. Both real-time and batch processing run through that one pipeline, cutting the complexity of managing separate layers. Kappa has just two layers: Speed (the stream-processing layer) and Serving (the final layer).

Figure 5 — Kappa: a single streaming pipeline feeds the serving layer — no separate batch path. Original diagram after Illustration 1.85.

🔑

Kappa isn't a replacement for Lambda

Both have their use cases. Pick Kappa when the batch and streaming analytics results are fairly identical (one pipeline suffices). Pick Lambda when they differ and you need both a fast approximate answer and a slow accurate one.

Recap

Data-driven by default IoT and apps emit endless data; businesses mine it for products, decisions, and service-health signals.

Ingestion has layers Collect/standardize → process → analytics → visualize, with storage + security spanning. Heterogeneous data → one format.

Real-time vs. batch Fast-but-partial vs. slow-but-complete. Ingestion is slow, costly, and risky to move.

Pipelines >legacy ETL Pipelines automate flow and stream in real time; ETL (Extract-Transform-Load) moves data in batches.

Distributed processing Split work across parallel nodes (ZooKeeper coordinates). Hadoop=batch; Spark/Kafka/Storm=real-time.

Lambda vs. Kappa Lambda merges batch + speed layers; Kappa runs one streaming pipeline. Choose by whether results diverge.

💡

The single idea underneath it all

Stream processing is one long negotiation between speed and accuracy. Every choice in the chapter — real-time vs. batch ingestion, pipelines vs. ETL, Lambda vs. Kappa — is really answering the same question: how fresh does the answer need to be, and how much accuracy will you trade to get it that fast? Lambda's refusal to choose (run both, merge) is the tell: when you can afford it, you buy speed and accuracy by paying for two paths.