# Scikit-Learn Pipelines in 2026: Feature Engineering and Interview Questions

> Master scikit-learn pipelines with ColumnTransformer for feature engineering. Learn preprocessing best practices, avoid data leakage, and prepare for machine learning interviews with practical code examples.

- Published: 2026-09-02
- Updated: 2026-09-02
- Author: Anthony Fillion-Maillet
- Tags: scikit-learn, pipeline, feature-engineering, machine-learning, python, interview
- Reading time: 12 min

---

Scikit-learn pipelines transform chaotic preprocessing code into reproducible, production-ready workflows. With [version 1.9](https://scikit-learn.org/stable/whats_new/v1.9.html) released in June 2026, `Pipeline` and `ColumnTransformer` remain the foundation of any serious machine learning project, now with callback monitoring, enhanced HTML visualization, and Array API support.

> **Interview Insight**
>
> Interviewers often ask: "How do you prevent data leakage during cross-validation?" The answer is pipelines. Fitting preprocessing steps inside the CV loop ensures test data never influences scaling or encoding decisions.

## Why Pipelines Eliminate Data Leakage

Data leakage occurs when information from the test set influences preprocessing. A common mistake: fitting `StandardScaler` on the entire dataset before splitting. The scaler learns mean and variance from test samples, inflating cross-validation scores.

Pipelines fix this by chaining transformers and estimators into a single object that fits all steps together:

```python
# pipeline_basics.py
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score

# Create a pipeline that scales features and trains a classifier
pipeline = Pipeline([
    ('scaler', StandardScaler()),  # Step 1: Normalize features
    ('classifier', LogisticRegression(max_iter=1000))  # Step 2: Train model
])

# cross_val_score fits the scaler separately for each fold
# No data leakage: test fold is never seen during scaler.fit()
scores = cross_val_score(pipeline, X, y, cv=5, scoring='accuracy')
print(f"Mean accuracy: {scores.mean():.3f} (+/- {scores.std():.3f})")
```

The `Pipeline` object implements `fit`, `predict`, and `score`, making it interchangeable with any scikit-learn estimator. This means GridSearchCV, RandomizedSearchCV, and all cross-validation utilities work without modification.

## ColumnTransformer for Mixed Data Types

Real datasets contain numeric columns (age, salary) and categorical columns (country, product_type). `ColumnTransformer` applies different preprocessing to different column subsets, then concatenates the results:

```python
# column_transformer_example.py
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
import pandas as pd

# Sample dataset with mixed types
df = pd.DataFrame({
    'age': [25, 30, None, 45],
    'salary': [50000, 60000, 75000, None],
    'country': ['US', 'UK', 'DE', 'US'],
    'education': ['Bachelor', 'Master', 'PhD', 'Bachelor']
})

# Define numeric preprocessing: impute missing values, then scale
numeric_transformer = Pipeline([
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler', StandardScaler())
])

# Define categorical preprocessing: impute with mode, then one-hot encode
categorical_transformer = Pipeline([
    ('imputer', SimpleImputer(strategy='most_frequent')),
    ('encoder', OneHotEncoder(handle_unknown='ignore', sparse_output=False))
])

# Combine transformers with ColumnTransformer
preprocessor = ColumnTransformer(
    transformers=[
        ('num', numeric_transformer, ['age', 'salary']),
        ('cat', categorical_transformer, ['country', 'education'])
    ],
    verbose_feature_names_out=True  # Prefix output names with transformer name
)

# Fit and inspect output shape
X_transformed = preprocessor.fit_transform(df)
print(f"Transformed shape: {X_transformed.shape}")
print(f"Feature names: {preprocessor.get_feature_names_out()}")
```

The `verbose_feature_names_out` parameter (enhanced in 1.6 to accept strings and callables) controls output feature naming. Setting it to `True` prefixes each feature with the transformer name, preventing name collisions when multiple transformers generate similar column names.

## Automating Column Selection with make_column_selector

Hardcoding column names breaks when datasets change. `make_column_selector` selects columns by dtype automatically:

```python
# automatic_column_selection.py
from sklearn.compose import ColumnTransformer, make_column_selector
from sklearn.preprocessing import StandardScaler, OneHotEncoder

# Select columns by dtype: no hardcoded column names
preprocessor = ColumnTransformer(
    transformers=[
        ('num', StandardScaler(), make_column_selector(dtype_include='number')),
        ('cat', OneHotEncoder(handle_unknown='ignore'),
         make_column_selector(dtype_include='object'))
    ],
    remainder='passthrough'  # Keep unprocessed columns as-is
)

# Works with any DataFrame that has numeric and object columns
X_processed = preprocessor.fit_transform(df)
```

The `remainder` parameter controls what happens to columns not matched by any transformer. Options include `'drop'` (default), `'passthrough'` (keep unchanged), or a transformer to apply to remaining columns.

## Feature Engineering Inside Pipelines

Pipelines extend beyond preprocessing to include feature engineering steps. Custom transformers inherit from `BaseEstimator` and `TransformerMixin`:

```python
# custom_transformer.py
from sklearn.base import BaseEstimator, TransformerMixin
import numpy as np

class DateFeatureExtractor(BaseEstimator, TransformerMixin):
    """Extract year, month, day, and weekday from datetime columns."""
    
    def __init__(self, date_column='date'):
        self.date_column = date_column
    
    def fit(self, X, y=None):
        # No fitting required for date extraction
        return self
    
    def transform(self, X):
        X = X.copy()
        dates = pd.to_datetime(X[self.date_column])
        
        # Extract temporal features
        X['year'] = dates.dt.year
        X['month'] = dates.dt.month
        X['day'] = dates.dt.day
        X['weekday'] = dates.dt.weekday
        X['is_weekend'] = (dates.dt.weekday >= 5).astype(int)
        
        # Drop original date column
        return X.drop(columns=[self.date_column])
    
    def get_feature_names_out(self, input_features=None):
        # Required for Pipeline.get_feature_names_out() to work
        return ['year', 'month', 'day', 'weekday', 'is_weekend']

# Integrate into a full pipeline
full_pipeline = Pipeline([
    ('date_features', DateFeatureExtractor(date_column='signup_date')),
    ('preprocessor', preprocessor),
    ('model', LogisticRegression())
])
```

Implementing `get_feature_names_out` enables the new HTML representation in scikit-learn 1.9 to display output feature names, making debugging and documentation easier.

## Hyperparameter Tuning Across Pipeline Steps

GridSearchCV tunes parameters across all pipeline steps using the `step__parameter` naming convention:

```python
# pipeline_gridsearch.py
from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestClassifier

# Build pipeline with tunable components
pipeline = Pipeline([
    ('preprocessor', preprocessor),
    ('classifier', RandomForestClassifier(random_state=42))
])

# Define parameter grid: note the double underscore syntax
param_grid = {
    'preprocessor__num__imputer__strategy': ['mean', 'median'],
    'classifier__n_estimators': [100, 200],
    'classifier__max_depth': [10, 20, None]
}

# GridSearchCV handles all combinations automatically
grid_search = GridSearchCV(
    pipeline,
    param_grid,
    cv=5,
    scoring='f1_weighted',
    n_jobs=-1
)

grid_search.fit(X_train, y_train)
print(f"Best params: {grid_search.best_params_}")
print(f"Best score: {grid_search.best_score_:.3f}")
```

The double underscore (`__`) accesses nested parameters. For `ColumnTransformer`, the path includes the transformer name: `preprocessor__num__imputer__strategy` reaches the imputer's strategy parameter inside the numeric transformer.

## Caching Transformers with memory Parameter

Complex preprocessing on large datasets can be slow. The `memory` parameter caches fitted transformers to disk:

```python
# cached_pipeline.py
from sklearn.pipeline import Pipeline
from tempfile import mkdtemp
import joblib

# Create a cache directory
cachedir = mkdtemp()

# Pipeline caches intermediate steps
cached_pipeline = Pipeline(
    [
        ('preprocessor', preprocessor),  # Cached after first fit
        ('classifier', RandomForestClassifier())
    ],
    memory=cachedir  # Or use joblib.Memory for more control
)

# First fit: slow (computes all transformations)
cached_pipeline.fit(X_train, y_train)

# Second fit with same X_train: fast (loads cached transformers)
cached_pipeline.fit(X_train, y_train)  # Skips preprocessor fitting
```

Caching is particularly useful during hyperparameter search, where the same preprocessing applies to multiple model configurations. The final estimator (classifier or regressor) is never cached, only intermediate transformers.

## Monitoring Training with Callbacks (scikit-learn 1.9)

The callback API in [version 1.9](https://scikit-learn.org/stable/whats_new/v1.9.html) adds progress monitoring to pipelines:

```python
# callback_monitoring.py
from sklearn.callback import ProgressBar, ScoringMonitor
from sklearn.linear_model import LogisticRegression

# Display progress during GridSearchCV
with ProgressBar():
    grid_search = GridSearchCV(
        pipeline,
        param_grid,
        cv=5,
        n_jobs=1  # Progress bars require single-threaded execution
    )
    grid_search.fit(X_train, y_train)

# Monitor scoring metrics per iteration
with ScoringMonitor(X_val, y_val, scoring='accuracy') as monitor:
    logreg = LogisticRegression(solver='lbfgs', max_iter=500)
    logreg.fit(X_train, y_train)
    print(f"Scores per iteration: {monitor.scores_}")
```

Callbacks work with `Pipeline`, `StandardScaler`, `LogisticRegression` (LBFGS solver), and all search CV classes. This experimental feature provides visibility into long-running fits without custom logging code.

## FeatureUnion for Parallel Feature Extraction

`FeatureUnion` runs multiple transformers in parallel and concatenates their output horizontally:

```python
# feature_union_example.py
from sklearn.pipeline import FeatureUnion, Pipeline
from sklearn.decomposition import PCA
from sklearn.feature_selection import SelectKBest, f_classif

# Combine PCA and univariate selection
feature_union = FeatureUnion([
    ('pca', PCA(n_components=5)),
    ('select', SelectKBest(f_classif, k=10))
])

# Pipeline: preprocess, combine features, classify
pipeline = Pipeline([
    ('preprocessor', preprocessor),
    ('features', feature_union),
    ('classifier', LogisticRegression())
])

# Output contains 5 PCA components + 10 selected features = 15 features
pipeline.fit(X_train, y_train)
```

Version 1.7.2 added validation ensuring all transformers return 2D outputs. This catches errors early when a custom transformer accidentally returns a 1D array.

## Serialization and Deployment

Pipelines serialize with `joblib`, preserving all fitted parameters:

```python
# model_deployment.py
import joblib

# Save the entire pipeline (preprocessing + model)
joblib.dump(pipeline, 'churn_predictor.pkl')

# Load in production
loaded_pipeline = joblib.load('churn_predictor.pkl')

# Predict on new data (same preprocessing automatically applied)
new_data = pd.DataFrame({'age': [32], 'salary': [65000], ...})
predictions = loaded_pipeline.predict(new_data)
```

The serialized file contains everything: column selector logic, fitted scaler means and variances, encoder categories, and model weights. Deployment requires only scikit-learn and joblib, no custom preprocessing code.

> **Version Compatibility**
>
> Pipelines serialized with scikit-learn 1.8 may not load correctly on 1.9 if they use deprecated parameters. Always pin your scikit-learn version in production and test upgrades before deploying.

## Interview Questions on Scikit-Learn Pipelines

**Q: What is data leakage and how do pipelines prevent it?**

Data leakage occurs when information from the validation or test set influences model training. The classic example: fitting a `StandardScaler` on the entire dataset before splitting. The scaler's mean and standard deviation include test samples, making cross-validation scores artificially high.

Pipelines prevent this by encapsulating preprocessing inside the cross-validation loop. When `cross_val_score` calls `pipeline.fit()`, each fold fits the scaler only on training data. Test fold statistics never leak into the preprocessing.

**Q: How do you access and modify parameters of nested pipeline steps?**

Use the double underscore syntax: `pipeline.set_params(classifier__n_estimators=200)`. For `ColumnTransformer`, include the transformer name: `pipeline.set_params(preprocessor__num__scaler__with_mean=False)`. The `get_params()` method returns all nested parameters as a flat dictionary, useful for inspection and logging.

**Q: When should you use FeatureUnion vs ColumnTransformer?**

`ColumnTransformer` applies different transformers to different columns of the same input. `FeatureUnion` applies different transformers to the entire input and concatenates horizontally. Use `ColumnTransformer` for heterogeneous column preprocessing (numeric vs categorical). Use `FeatureUnion` for feature augmentation, where multiple extraction methods (PCA, polynomial features, domain-specific extractors) produce complementary features from the same data.

**Q: How do you debug a failing pipeline?**

Set `verbose=True` on the pipeline to see step timings. Access intermediate results with `pipeline.named_steps['step_name'].transform(X)` for a fitted pipeline. In scikit-learn 1.9, the HTML representation displays fitted attributes and output feature names, simplifying debugging in Jupyter notebooks. For deeper inspection, use `pipeline[:-1].fit_transform(X, y)` to get the output just before the final estimator.

## Production Checklist for ML Pipelines

- Pin scikit-learn version in requirements to avoid serialization issues
- Use `make_column_selector` instead of hardcoded column names when datasets may change
- Set `handle_unknown='ignore'` on `OneHotEncoder` to handle new categories in production
- Cache expensive preprocessing with the `memory` parameter during development
- Validate pipeline output shape and feature names in tests before deployment
- Store `get_feature_names_out()` alongside the model for feature importance analysis
- Use `Pipeline` inside `GridSearchCV`, never the reverse, to ensure correct CV behavior

---

Source: SharpSkill (https://sharpskill.dev), tech interview preparation for your real stack.
HTML version of this page: https://sharpskill.dev/en/blog/data-science/scikit-learn-pipelines-feature-engineering-interview-2026