Preprocessing#

Back to User Guide

macroforecast.preprocessing turns a canonical pandas panel from macroforecast.data into a processed panel plus metadata. The preferred input is a DataBundle or DataSpec; the output is a PreprocessedData object. The default reprocess() path follows the public McCracken-Ng FRED-MD Matlab workflow for FRED-MD/FRED-QD style panels.

For use inside a POOS runner, preprocess_spec stores the preprocessing choices without executing them. The runner then applies the spec at each origin, refitting stateful steps (outlier thresholds, EM factors, standardization scale) only on the estimation-window rows available at that origin. This is the leak-free path.

Key Callables#

mf.preprocessing.reprocess applies the full preprocessing sequence to a panel immediately (full-sample; for exploration and single-shot use).

mf.preprocessing.preprocess_spec stores preprocessing choices for runner-fitted execution. Pass the returned PreprocessSpec to forecasting.run or to an Arm in pipeline_spec.

import macroforecast as mf

# Full-sample preprocessing (for exploration).
processed = mf.preprocessing.reprocess(
    data_spec,
    transform="official",   # apply McCracken-Ng t-codes
    outliers="iqr",         # IQR-based outlier replacement
    impute="em_factor",     # EM algorithm factor imputation
    standardize="zscore",
)

# Deferred preprocessing spec for runner-fitted execution (leak-free).
prep_spec = mf.preprocessing.preprocess_spec(
    transform="official",
    outliers="iqr",
    impute="em_factor",
    standardize="zscore",
)
# Pass prep_spec to Arm(..., preprocessing=prep_spec) or forecasting.run(...).

Executed walkthrough#

Running the full-sample path on the loaded data_spec applies the t-code transforms, flags outliers, imputes by EM factor, and standardizes:

processed = mf.preprocessing.reprocess(
    data_spec,
    transform="official", outliers="iqr", impute="em_factor", standardize="zscore",
)
print(type(processed).__name__, processed.panel.shape)
print("NaN before:", int(bundle.panel.isna().sum().sum()),
      "| NaN after:", int(processed.panel.isna().sum().sum()))
print(processed.panel.iloc[:3, :4])

PreprocessedData (694, 128)
NaN before: 942 | NaN after: 0
                 RPI   W875RX1  DPCERA3M086SBEA  CMRMTSPLx
date
1960-03-01 -0.166574 -0.325648         2.178270  -2.831788
1960-04-01  0.132535  0.205311         2.425007   0.704389
1960-05-01 -0.026582  0.006684        -4.410584  -3.134124

The output panel is stationary and standardized. The row count drops from 708 to 694 because the official transforms difference the early observations away, and the EM-factor step fills the 942 missing entries, leaving none remaining. This is the full-sample path for exploration; inside a runner, preprocess_spec refits these same steps on each origin’s estimation rows only.

Custom Steps And Caches#

Custom preprocessing steps run after the built-in transform/outlier/impute/ standardize/frame sequence. In the normal origin_available runner path, fitted standardization is applied before custom steps at transform time, matching the fit-time order, so scale-sensitive custom steps see the same units on train and test rows.

Disk-backed preprocessing caches use content-derived identities. If a preprocess_spec(custom_steps=...) contains a custom callable, set a stable func.__mf_digest__ string before using preprocessing_cache_dir or a PreprocessorStore; update the digest when the callable behavior changes. Named custom callables without a digest still run, but disk get/put is skipped and the runner recomputes instead of risking stale reuse. Anonymous lambda custom steps without __mf_digest__ are rejected because they collide by qualified name.

Under policy="fit_window", built-in outlier/imputation/standardization state is fitted on the training window and applied to later rows, but custom steps are re-executed on the apply window. Keep those custom steps row-local/stateless; a custom step that computes statistics from all rows it receives can read post-origin rows and leak future information.

Reference#