Episode 2 — Data, Model, Code: The Three Pipelines

Episode 1 established that an AI system changes along three axes. Those axes are not abstractions: each one is a physical pipeline with its own stages, its own tooling and its own failure modes. This episode walks all three, and ends with the decision that most often gets made by accident — how the model is actually served.

Learning objectives

Describe the data engineering pipeline and its four stages, with the best practices attached to each.
Describe the ML pipeline and its four stages, ending in a packaged, portable model.
Classify any ML workload into one of the four architectural patterns derived from two dimensions.
Choose a model serving pattern among the five, and justify it against a requirement.
Explain what a serialization format buys you and why ONNX matters.

Why it matters

"We built a model" is not a deliverable. A model becomes a product only when three pipelines exist, are automated, and are versioned. Architects are usually hired at the moment a team discovers that its notebook cannot be operated. This episode is the anatomy of what has to be built instead.

1. The map

Note the two dotted arrows. They are the whole point. A classical delivery pipeline is a line; an ML delivery pipeline is a cycle, and the return path carries two distinct signals: "performance dropped, retrain" and "here is new data, re-ingest".

Golden rule from the source: document everything you have learned at each step of the whole pipeline. In ML, undocumented decisions are unrecoverable — you cannot re-derive them by reading the artifact.

2. The data pipeline — where the time actually goes

"Garbage In, Garbage Out" means, in ML terms, that the model is only as good as the data, and that the training data indirectly determines the performance of the whole production system. Data engineering is reported as the most time-consuming part of an ML project.

2.1 Ingestion

Collecting data from internal/external databases, data marts, OLAP cubes, warehouses, OLTP systems, Spark, HDFS — possibly including synthetic data generation or enrichment. Best practices, all to be maximally automated:

Data source identification — find the data and document its origin (provenance).
Space estimation and workspace location — know the volume before you start.
Obtaining data — convert to a manipulable format without changing the data itself.
Back up data — always work on a copy, keep the original untouched.
Privacy compliance — delete or anonymise sensitive information (GDPR).
Metadata catalog — record size, format, aliases, last modified time, access control lists.
Test data — sample a test set, put it aside and never look at it, to avoid data-snooping bias. You have fallen for it if you selected a model class using the test set: the selection is over-optimistic and will not survive production.

2.2 Exploration and validation

Profiling produces metadata (min, max, avg); validation runs user-defined error-detection routines over the dataset (are the address components consistent? is the postal code right? are there missing values in relevant attributes?).

Per-attribute profile to document: name, number of records, data type, numerical measures, missing-value ratio, distribution type. Then identify the label attribute, visualise distributions, and compute attribute correlations.

2.3 Wrangling (cleaning)

Programmatic re-formatting and re-structuring: transformations, outliers, missing values, dropping irrelevant attributes, restructuring (reordering fields, extracting values, combining fields, filtering records, changing granularity through aggregations and pivots).

Non-negotiable: write scripts or functions for all data transformations, so they can be re-applied to future data. A transformation performed by hand in a notebook is a defect waiting for the next data batch.

2.4 Splitting

Split into training (≈80 %), validation and test datasets.

3. The ML pipeline — from data to a portable artifact

3.1 Training

Includes feature engineering and hyperparameter tuning:

Discretise continuous features; decompose features (categorical, date/time); add transformations (log, sqrt, x²); aggregate into new features; scale — standardise or normalise.
New features should move from idea to production quickly — cycle time on features is a first-class metric.

And model engineering, an iterative workflow worth memorising as a recipe:

Every model specification goes through code review and is versioned.
Train many models from different categories with standard parameters.
Compare them using N-fold cross-validation, reporting mean and standard deviation.
Error analysis — what kinds of mistakes does each make?
Revisit feature selection and engineering.
Keep the top three to five, preferring models that make different types of errors.
Tune hyperparameters by cross-validation. Data-transformation choices are hyperparameters too. Random search is preferred over grid search.
Consider ensembles — majority vote, bagging, boosting, stacking.

Step 6 is the one people skip. Preferring diverse errors is what makes step 8 pay off.

3.2 Evaluation, testing, packaging

Evaluation — validate against the original business objectives before serving.
Testing — final model acceptance test on the hold-back test set, to estimate the generalisation error.
Packaging — export to a format that the business application can consume.

4. Four architectural patterns from two dimensions

Classify any ML workload on two axes:

Training: offline (batch/static — the model stays constant until retrained, and therefore decays) vs online (dynamic — retrained as data arrives, typical for time-series such as sensors or trading).
Prediction: batch (predictions computed over historical input) vs real-time / on-demand (generated at request time).

	Batch prediction	Real-time prediction
Offline training	Forecast	Web-Service
Online training	(rarely useful)	Online Learning → AutoML

Forecast — train, then run on historical data. Common in academia and Kaggle; not common in industry production systems.
Web-Service — the most commonly described deployment. Trained offline on historical data, but predicts near real-time on live data, one record at a time. The model is constant until retrained and redeployed.
Online Learning (better named incremental learning) — a continuous stream of data points or mini-batches; the model is incrementally retrained and instantly available as a service. Fits the lambda architecture. Its big danger: if bad data enters the system, model and system performance decline continuously.
AutoML — instead of updating a model, an entire training pipeline runs in production and produces new models on the fly. The user supplies data; algorithm selection and configuration are automatic. Caveat: AutoML models still have to reach the accuracy required for real-world success.

Design consequence. Moving from Web-Service to Online Learning is not a tuning decision — it changes your failure model. A poisoned or broken input in a Web-Service produces one bad prediction; in Online Learning it produces a permanently degraded model. Episode 15 revisits this as an attack surface.

5. Serialization: how far the model can travel

To be distributable, the model must be present and executable as an independent asset, outside the training environment (e.g. a scikit-learn model used from a Spark job).

Language-agnostic exchange formats

Format	Idea	Note
Amalgamation	Model + all code to run it bundled as a single compilable source file	Portable and compact for simple algorithms; model code and parameters must be managed together
PMML	XML description of model and pipeline, standardised by the DMG	Does not support all algorithms; limited open-source uptake
PFA	JSON "scoring engine" with control structures and a function library; DMG's intended PMML replacement	Requires a PFA-enabled runtime
ONNX	Framework-independent format so any tool can share one model format	Supported by many large vendors; most deep-learning tools support it

Vendor-specific formats: scikit-learn .pkl, H2O POJO/MOJO, SparkML via MLeap .jar/.zip, TensorFlow .pb, PyTorch TorchScript .pt, Keras .h5 (HDF), Apple .mlmodel.

Of that whole list, ONNX is the one that became the de facto interchange standard, alongside each cloud's native format. When an architect asks "can we change inference runtime without retraining?", the answer is a function of this choice.

6. The code pipeline: five serving patterns

Serving needs three things: a model, an interpreter to execute it, and input data. Deploying an ML system means two things at once: deploying the pipeline for automated retraining and model deployment, and providing the prediction API.

Pattern	Mechanism	Use it when
Model-as-Service	Model + interpreter wrapped in a dedicated web service, consumed via REST or gRPC	You need independent release cycles and multiple consumers. Works for Forecast, Web-Service and Online Learning workflows
Model-as-Dependency	The packaged model is a library dependency of the application; you call a prediction method	Simplest packaging; mostly used for the Forecast pattern; ties model releases to application releases
Precompute	Predictions are computed in advance for a batch and persisted in a database; requests become queries	Input space is enumerable and latency must be near zero
Model-on-Demand	The model has its own release cycle, and prediction requests flow through a message broker (input queue → event processor holding the model runtime → output queue)	Throughput smoothing, asynchronous consumers, bursty load
Hybrid-Serving (Federated Learning)	One model per user on-device plus a general server model; devices send model updates, never personal data; server aggregates and redistributes the initial model	Personal data must not leave the device. Updates run when the device is idle, on Wi-Fi and charging

Federated learning's trade-off is explicit in the source and worth quoting in a design review: normal ML assumes homogeneous, large datasets on powerful, always-available hardware; federated learning has less powerful devices, data spread over millions of them, and intermittent availability.

Cross-check with CD4ML

CD4ML names three patterns with slightly different boundaries. Being able to translate between the two vocabularies is a useful architect reflex:

CD4ML	Equivalent here	Consequence
Embedded model	Model-as-Dependency	Application and model versions are treated together
Model as separate service	Model-as-Service	Decoupled updates, but inference latency is introduced
Model as data	Model-on-Demand / Precompute	Model published independently and ingested at runtime — enables blue-green and canary releases

Deployment strategies

Because inference is stateless, lightweight and idempotent, containerisation is the de-facto delivery standard: package the ML stack and the prediction code into a container, orchestrate with Kubernetes (or an equivalent), expose prediction over a REST API. The alternative is a serverless function — code and dependencies zipped behind a single entry point — where the constraint to watch is the artifact size limit.

Numbers & names to memorize

4 data stages: ingestion → exploration/validation → wrangling → splitting. Train ≈ 80 %.
4 ML stages: training → evaluation → testing → packaging.
3 code stages: serving → performance monitoring → logging.
2 dimensions → 4 patterns: Forecast, Web-Service, Online Learning, AutoML.
5 serving patterns: Model-as-Service, Model-as-Dependency, Precompute, Model-on-Demand, Hybrid (Federated).
ONNX = the surviving language-agnostic interchange format.
Data snooping = choosing a model using the test set.
Random search > grid search for hyperparameters.
Prefer keeping 3–5 candidate models that make different errors.

Key takeaways

Three axes mean three pipelines, and the delivery path is a cycle, not a line.
Every transformation must be code. Manual data work does not survive the next batch.
Offline vs online training and batch vs real-time prediction are the two questions that determine your architecture; answer them before choosing tools.
Serving pattern is an architectural decision with release-cycle consequences, not a deployment detail.
Serialization determines portability — and portability is what protects you from being frozen to one framework (the glue code debt of Episode 1).

Exercises

Exercise 1 — Pick the pattern (scenario)

For each case, name the architectural pattern (Forecast / Web-Service / Online Learning / AutoML) and the serving pattern, in one line of justification each.

(a) A bank scores every incoming card transaction in under 50 ms; the model is retrained monthly on labelled fraud cases.
(b) A telecom computes churn propensity for its 12 million subscribers every Sunday night; the CRM reads the score during agent calls.
(c) A mobile keyboard personalises next-word prediction per user; word-usage data must never leave the handset.
(d) An industrial sensor platform adapts a vibration-anomaly model continuously as new readings stream in from a plant.

Exercise 2 — The notebook that cannot be operated (scenario)

A data scientist hands over: one notebook, a model_final_v3.pkl, and a CSV extracted from the warehouse three months ago "with a couple of manual fixes in Excel". Accuracy on their held-out split is 0.94.

List, stage by stage across the three pipelines, what is missing before this can be operated — and identify the single fact in the description that most undermines the 0.94 figure.

Exercise 3 — MCQ

Which statement about Precompute serving is correct?

A. It is the only pattern compatible with online learning.
B. Predictions are computed in advance for a batch and persisted, so a request becomes a database query.
C. It requires a message broker with input and output queues.
D. It couples the model release cycle to the application release cycle.

Exercise 4 — MCQ

Your team must be able to swap the inference runtime (from a Python service to an embedded C++ runtime) without retraining. Which decision matters most?

A. Choosing Kubernetes over serverless functions.
B. Choosing an open, language-agnostic serialization format such as ONNX rather than a vendor-specific one.
C. Increasing the training set size.
D. Moving from Model-as-Service to Model-as-Dependency.

Answers

Exercise 1.

(a) Web-Service (offline training, real-time prediction) served as Model-as-Service — independent release cycle, sub-50 ms request/response.
(b) Forecast (offline training, batch prediction) served by Precompute — the scores are persisted and the CRM reads them; near-zero read latency, no inference at call time.
(c) Hybrid-Serving / Federated Learning — the requirement "data never leaves the handset" is exactly the property federated learning exists to provide; expect the device-availability constraint (idle, Wi-Fi, charging).
(d) Online Learning (incremental) served as Model-as-Service on a stream. Flag the risk explicitly: bad data entering the stream degrades the model permanently, so input validation and an action limit are mandatory.

Exercise 2. Missing, by pipeline:

Data — no documented provenance, no versioned dataset or snapshot, no schema/validation routines, no metadata catalog, no reproducible split, no privacy assessment.
Model — no versioned training code, no experiment tracking (parameters/metrics), no cross-validation report, no comparison against a simple baseline, no portable packaging (a .pkl is a vendor-specific Python pickle).
Code — no serving pattern chosen, no API contract, no monitoring or prediction logging, no retraining trigger.

The most damaging fact: "a couple of manual fixes in Excel." Those transformations exist nowhere in code, so the pipeline cannot be re-run on new data — and, worse, they may have touched rows that later landed in the held-out split. The 0.94 is unreproducible and possibly contaminated. (A close second: a three-month-old extract, which makes the figure a statement about the past, not about production.)

Exercise 3. B. C describes Model-on-Demand; D describes Model-as-Dependency; A is false — Precompute is tied to the Forecast workflow.

Exercise 4. B. Portability is a property of the serialization format. A is an orchestration choice, C is unrelated, and D would make the coupling worse.