Running#

Back to User Guide

macroforecast.forecasting.run is the atomic forecasting function. It accepts one model, one data source, and a WindowSpec, then iterates over test origins and returns a ForecastResult. For each origin it: applies preprocessing to the estimation-window panel; builds features and targets from those rows; selects hyperparameters via the validation window; fits the model; and generates predictions for the test horizon.

macroforecast.pipeline.run_pipeline wraps run into a full POOS evaluation. It enumerates every (arm, target, horizon) cell, calls run for each, collects the master forecast frame, evaluates every contender against the benchmark with relative MSE, DM/CW, and the Model Confidence Set, and returns a PipelineReport.

Forecast policies#

The forecast_policy argument to run (and the policy resolved from a TargetSpec t-code in run_pipeline) controls how h-step forecasts are built:

  • direct (forecast_policy="direct"): fit one model for each horizon h separately, using y[t+h] as the target. The simplest and most common choice.

  • direct_average (forecast_policy="direct_average"): the forecast object is the h-period cumulation (average) of the stationary transform, not the raw single-period value. This is the standard convention for growth-rate series (t-codes 2, 3, 5, 6, 7 in FRED-MD/QD) and matches how practitioners report average inflation or average growth over the horizon.

  • path_average (forecast_policy="path_average"): fit h step-specific models, where step s forecasts the one-period object realized at t+s from information available at the origin, then average the h step forecasts. This is a direct multi-step design, not an iterated one; iterating a single model forward instead is the separate recursive policy.

  • recursive (forecast_policy="recursive", code alias "iterated"): fit one one-step-ahead model, then roll it forward h times, feeding each step’s own prediction back in as the next step’s lagged input (the textbook “iterated” multi-step forecast). Unlike path_average, a later step’s forecast depends on an earlier step’s prediction rather than only on origin-available data. See future_feature_policy for how exogenous (non-target) predictors are rolled forward under this policy.

At horizon 1, direct_average and path_average are the same forecast by construction (averaging over a single step is that step), so the two policies produce identical predictions there. They diverge only for h greater than 1, where the h-period-average target and the averaged one-step path are genuinely different objects. This holds across every model, including the information-criterion autoregressions (ar, far), whose order is selected by BIC/AIC on the same sample under both policies.

Textbook mapping#

macroforecast’s policy names do not always match the vocabulary of a forecasting textbook. The table below lines them up:

Textbook term

forecast_policy

What it does

direct

"direct"

One model per horizon h, fit directly on the h-period-ahead value.

iterated / recursive

"recursive" (code alias "iterated")

One one-step model, rolled forward h times, each step’s prediction feeding the next.

direct, h-period-average object

"direct_average"

The direct idea, but the forecast object is the h-period average of the stationary transform.

h-average of h one-step models

"path_average"

h independent step-specific one-step-ahead models (never iterated), averaged.

The *_average variants (direct_average, path_average) are both h-average forecast objects; they differ in whether one model is fit on the h-period-average target directly (direct_average) or h one-step models are averaged after the fact (path_average). Neither feeds a prediction back into the next step’s inputs – only recursive does that.

The t-code to policy mapping is documented in the Pipeline reference.

Recursive custom models need extra care with exogenous predictors. A custom supervised model with non-target features under forecast_policy="recursive" must only use predictors that are genuinely available at each recursive step (for example, lagged values). pipeline_spec() warns when it can see a custom supervised model, a recursive target policy, and exogenous features in the same arm.

Parallel custom code#

pipeline_spec(..., n_jobs>1) runs cells in worker processes. Any custom model, feature, preprocessing, policy, or model-selection object carried by an arm must therefore be pickleable. Define custom callables at module scope, or keep n_jobs=1 for notebook-local closures. The runner preflights these arm objects before dispatch and raises an actionable ValueError instead of a raw PicklingError from a worker pool.

Key Callables#

mf.forecasting.run executes one (model, data, window) cell and returns a ForecastResult.

mf.pipeline.run_pipeline executes a full PipelineSpec and returns a PipelineReport.

import macroforecast as mf
from macroforecast.pipeline import pipeline_spec, run_pipeline, Arm, EvalSpec, TargetSpec

# Low-level: run one model for one target.
result = mf.forecasting.run(
    data_spec,
    model="ar",
    window=mf.window.from_cutoffs(test_start="1985-01-01", horizon=1),
    forecast_policy="direct",
    target="INDPRO",
    horizon=1,
)
forecasts_df = result.to_frame()

# High-level: run the full pipeline with multiple arms and automatic evaluation.
spec = pipeline_spec(
    data=bundle,
    targets=[TargetSpec(name="INDPRO")],
    horizons=[1, 3, 6, 12],
    window=mf.window.from_cutoffs(test_start="1985-01-01"),
    arms=[
        Arm(name="AR", model="ar", is_benchmark=True),
        Arm(name="RF", model="random_forest",
            preprocessing=mf.preprocessing.preprocess_spec(transform="official"),
            features=mf.feature_engineering.feature_spec(
                target="INDPRO",
                predictors="all",
                lags=None,
                feature_steps=[mf.feature_engineering.marx_step(name="MARX_X", max_lag=12)],
            )),
    ],
    evaluation=EvalSpec(benchmark="AR"),
)
report = run_pipeline(spec)

For a runnable end-to-end example, see the single-forecast and full-study snippets in Getting Started and the step-by-step pipeline in the Replication Gallery.

Seeds, parallelism, and model storage#

pipeline_spec(..., seed=...) is the run-level reproducibility knob for the pipeline. During run_pipeline(spec), it temporarily becomes the active mf.configure(random_seed=...) value, so model-owned random search and parallel workers see the same seed. Stochastic model fits derive a stable per-arm random_state from (seed, arm name) only when the arm did not explicitly pass random_state in params; explicit model params win. The report records these effective values under report.provenance["effective_seeds"].

n_jobs="auto" resolves to a concrete cell-worker count plus a per-worker model thread budget. On Linux it honors CPU affinity; on macOS and Windows it falls back to os.cpu_count(). Parallel workers cap common BLAS/OpenMP thread environment variables before running forecast cells and receive the data payload once per worker rather than once per cell.

Pipeline model persistence is opt-in. pipeline_spec(...) defaults to save_models=False; pass save_models=True and model_store=... only when you need fitted-model pickles. Large projected stores warn before execution. Remove old model fits with mf.pipeline.purge_model_store(...). The lower-level mf.forecasting.run(...) keeps its historical save_models=True default.

Incremental horse races#

Long paper projects often grow a model comparison over many months. Use pipeline_spec(..., result_store="results/cells", preprocessing_cache_dir="results/prep") when you expect to add arms later:

spec = pipeline_spec(
    data=bundle,
    targets=[TargetSpec("INDPRO")],
    horizons=[1, 3, 12],
    window=window,
    arms=[
        Arm("AR", model="ar", features=features),
        Arm("RF", model="random_forest", features=features),
    ],
    evaluation=EvalSpec(benchmark="AR"),
    result_store="cache/result_cells",
    preprocessing_cache_dir="cache/preprocessing",
)
first = run_pipeline(spec)

later = pipeline_spec(
    data=bundle,
    targets=[TargetSpec("INDPRO")],
    horizons=[1, 3, 12],
    window=window,
    arms=[
        Arm("AR", model="ar", features=features),
        Arm("RF", model="random_forest", features=features),
        Arm("GBM", model="gradient_boosting", features=features),
    ],
    evaluation=EvalSpec(benchmark="AR"),
    result_store="cache/result_cells",
    preprocessing_cache_dir="cache/preprocessing",
)
second = run_pipeline(later)

The second run reuses the stored (target, horizon, arm) cells for AR and RF and computes only GBM. The shared preprocessing_cache_dir also reuses the prepared per-origin preprocessing base when the preprocessing spec is unchanged. Result-store identities include the data content fingerprint, the effective selection seed, arm/model/features/preprocessing choices, and the backend package versions that own the arm’s numerical fit. Vintage-aware specs additionally hash the enumerable vintage labels, reference calendar, and a bounded latest-vintage panel fingerprint. Stores created before this identity hardening will miss and recompute cells once, then reuse normally under the new digest.

For custom code, reuse is opt-in. A custom model function, feature step, preprocessing step, metric, or loss must carry a stable __mf_digest__ string to be stored. Without it, the cell is recomputed every run. If you edit the callable, do not trust old results unless you also update __mf_digest__ and force a miss. The store is intended for a single writer; inspect it with mf.pipeline.result_store_summary(...) and delete cells with mf.pipeline.purge_result_store(...).

For custom models, prefer passing the digest through the constructor:

model = mf.models.custom_model("my_model", fit_model, mf_digest="my-model-v1")

When a custom object lacks a digest, run_pipeline() emits a warning for that undigestible cell and includes the reason from the result-store identity layer.

Checkpoint rescoring also verifies identity for new checkpoints. Each completed cell writes a small manifest next to its h<h>/origin_*.parquet files. rescore() refuses manifest-bearing cells whose stored digest no longer matches the current spec/data identity; pass allow_stale=True only when intentionally scoring stale forecasts. Older checkpoint directories that lack manifests still rescore, but emit a warning because they can only be matched by directory name.

Reference#