Evaluation#

Back to User Guide

macroforecast.pipeline performs automatic evaluation when run_pipeline completes. It computes accuracy metrics, runs forecast-comparison tests, and identifies the Model Confidence Set for every (target, horizon) combination. Raw metric functions live in macroforecast.metrics, forecast-comparison statistical tests live in macroforecast.tests, and macroforecast.evaluation provides multi-slice evaluation reports combining both.

Accuracy metrics#

The pipeline reports several metrics per (contender, target, horizon) cell. The two most important for model comparison are:

  • RMSE ("rmse"): root mean squared forecast error over the test origins.

  • relative MSE ("relative_mse"): the ratio of contender MSE to benchmark MSE. A value below 1.0 means the contender beats the benchmark. This is the standard metric in the macro forecasting literature (not relative RMSE, which would be the square root of this ratio).

relative_mse and relative RMSE are related by relative_mse = (relative_RMSE)^2, but they are not the same quantity. The EvalSpec default uses "relative_mse"; to report the square-root version, add a post-processing step or use mf.metrics.relative_rmse directly.

Evaluation sample#

Per-contender metrics (RMSE, relative MSE, OOS-R2) are scored on each contender’s pairwise common sample with the benchmark, meaning the origins where both that contender and the benchmark have a forecast and the realised target is observed. A contender whose feature block starts late (for example a raw lag of a series that only begins mid-sample) is therefore scored on its own shorter window without truncating the other contenders. n_common in the accuracy table is per-contender, and the pipeline emits a RuntimeWarning when coverage is ragged so the heterogeneity is visible rather than silent.

The Model Confidence Set is different. It needs a single joint sample where every candidate is observed, so it uses the listwise-common sample across all contenders. The accuracy table and the MCS therefore rest on different samples by design: a pairwise relative metric uses all the data each pair shares, while the joint MCS needs origins common to every candidate.

Forecast comparison tests#

The pipeline runs statistical forecast comparison tests across all contenders:

  • Diebold-Mariano (DM): tests whether contender and benchmark have equal predictive accuracy. Valid for any pair of forecasts (nested or non-nested). The default applies the Harvey-Leybourne-Newbold small-sample correction and uses a Student-t reference with df=n_obs-1, matching the package’s forecast::dm.test parity contract. Use test_options={"dm": {"small_sample": False}} only when a replication design needs the plain Diebold-Mariano (1995) statistic and asymptotic standard-normal p-value, such as MATLAB oracles that report the uncorrected DM statistic.

  • Clark-West (CW): adjusts the DM test for the finite-sample upward bias of a larger nested model. Valid only when the benchmark is nested within the contender (declare nested_in_benchmark=True on the arm). The pipeline emits CW only for arms that declare nesting; CW is silently invalid otherwise.

  • Additional pairwise tests: opt in with "gw" (Giacomini-White conditional predictive ability), "gr" (Giacomini-Rossi fluctuation), "enc_new" / "enc_t" (nested encompassing), "mz" (Mincer-Zarnowitz forecast-rationality regression), or "pt" / "hm" / "ag" (directional accuracy). Directional tests evaluate the contender’s own sign skill on the same benchmark-aligned origins. Degenerate directional forecasts are reported with status="degenerate" rows rather than aborting evaluation; ENC-NEW/ENC-T rows without a p-value or configured critical value are marked status="inconclusive".

  • Joint multi-horizon tests: "uspa" and "aspa" run Quaedvlieg-style uniform and average SPA jointly across all horizons for each target/contender/benchmark triple. They require at least two horizons and land in report.significance with horizon="joint".

  • Model Confidence Set (MCS): identifies the set of models that cannot be statistically distinguished from the best model at a given significance level (mcs_alpha). Uses the iterative elimination algorithm by default.

  • Full-set benchmark tests: "spa", "rc", and "stepm" compare the full contender set against the benchmark and land in report.mcs alongside MCS. They require the arch extra (pip install "macroforecast[arch]") and carry a dependent-loss size caveat; prefer model_confidence_set or uspa/aspa when serial dependence in losses is central to the inference.

Tests that estimate a HAC or lag-truncated long-run variance accept fixed lag overrides through test_options. Use hac_lags when a replication design pins a Newey-West bandwidth rather than deriving it from the forecast horizon:

evaluation = mf.pipeline.EvalSpec(
    benchmark="AR",
    tests=("dm", "cw", "gw", "enc_t", "gr", "mz"),
    test_options={
        "dm": {"hac_lags": 4},
        "cw": {"hac_lags": 4},
        "gw": {"hac_lags": 4},
        "enc_t": {"hac_lags": 4},
        "gr": {"hac_lags": 4},
        "mz": {"hac_lags": 4},
    },
)

hac_lags must be an integer greater than or equal to zero and is validated when pipeline_spec is built. For "gr", hac_lags is the paper-facing alias for the legacy lag_truncate option and takes precedence if both are supplied.

Choosing the benchmark#

The relative metrics (relative_mse, r2_oos) and the comparison tests score every contender against one benchmark, named by EvalSpec(benchmark=...). The benchmark is itself an arm, so it is fit and forecast like any other contender, and you can point it at whichever arm you want. The relative metrics divide by that arm’s realised forecast error.

evaluation = mf.pipeline.EvalSpec(benchmark="AR")   # any arm name

Because an arm is just a model with its preprocessing and features, the benchmark can be the same model as the contenders under a different configuration. A common design is a base model as the benchmark and enhanced variants as the contenders, for example a plain random forest on the base features scored against random forests that add feature blocks (MARX, factors). User-defined models built with mf.custom_model are arms too, so a custom model works as a contender and as the benchmark.

arms = [
    mf.pipeline.Arm("RF_base", model="random_forest", features=base_features, is_benchmark=True),
    mf.pipeline.Arm("RF_MARX", model="random_forest", features=marx_features),
    mf.pipeline.Arm("RF_factors", model="random_forest", features=factor_features),
    mf.pipeline.Arm("my_model", model=mf.custom_model("my_model", my_fit_func)),
]
evaluation = mf.pipeline.EvalSpec(benchmark="RF_base")   # every arm scored vs base RF

The benchmark is matched by contender name within each (target, horizon) cell, which is enough when the benchmark shares the forecast policy of the contenders.

A benchmark from another policy (or any fixed benchmark)#

Sometimes the benchmark you want is produced under a different forecast policy than the contenders. The GCLS (2021) appendix, for instance, scores both its direct and its path-average tables against a single FM benchmark, the direct FM. run_pipeline accepts several policies for one target in a single spec, so run them together and score with evaluate_cross_policy, which makes each (arm, forecast_policy) its own contender and scores all of them against the one benchmark policy you name:

report = mf.pipeline.run_pipeline(mf.pipeline.pipeline_spec(
    data=bundle,
    targets=[
        mf.pipeline.TargetSpec("Y", transform="value", policy="direct_average"),
        mf.pipeline.TargetSpec("Y", transform="value", policy="path_average"),
    ],
    arms=[fm_arm, rf_arm, ar_arm],
    horizons=[1, 3, 6, 12, 24],
    window=window,
    evaluation=mf.pipeline.EvalSpec(benchmark="FM"),
))

# every contender, direct and path, scored against the DIRECT FM
acc = mf.pipeline.evaluate_cross_policy(
    report.forecasts, benchmark="FM", benchmark_policy="direct_average",
)

The returned table has one row per (target, horizon, arm, forecast_policy) with relative_mse / r2_oos / rmse computed against the fixed benchmark, and keeps arm and forecast_policy as their own columns.

This is also the safety note for multi-policy specs. accuracy_table keys the relative metrics on contender name within a (target, horizon) cell and does not split on policy. If you run more than one policy for a target in a single spec and score with the plain accuracy table, the two policies’ rows for the same arm are pooled and the relative metrics mix them. evaluate_cross_policy qualifies the contender by forecast_policy for you and is the recommended path.

Key Callable#

EvalSpec declares the benchmark arm, which metrics and tests to compute, per-test options, MCS alpha, and optional evaluation-window subsamples. Pass it to pipeline_spec.

from macroforecast.pipeline import EvalSpec, SubsampleWindow

evaluation = EvalSpec(
    benchmark="AR",
    metrics=("rmse", "relative_mse", "r2_oos"),
    tests=("dm", "cw", "mcs", "spa", "uspa", "mz"),
    test_options={"spa": {"n_boot": 999, "block_length": 5},
                  "uspa": {"n_boot": 999, "block_length": 3},
                  "dm": {"hac_lags": 4}},
    cw_for_nested=True,    # compute CW only for arms with nested_in_benchmark=True
    mcs_alpha=0.10,
    subsamples={
        "full": SubsampleWindow(),
        "ex_covid": SubsampleWindow(exclude=(("2020-03-01", "2021-12-31"),)),
        "post_gfc": SubsampleWindow(start="2010-01-01"),
        "nber_recession": SubsampleWindow(mask="nber_recession"),
        "nber_expansion": SubsampleWindow(mask="nber_expansion"),
    },
)

The accuracy table, significance tests, and Model Confidence Set are produced by run_pipeline. Subsamples filter the already-produced forecast frame by target date before scoring; they do not refit models. SubsampleWindow(mask=...) intersects the date window with a boolean state series. Pass a date-indexed boolean Series, a {date: bool} mapping, or the named masks "nber_recession" / "nber_expansion". The NBER masks fetch USREC for month-start targets and USRECQ for quarter-start targets through the raw FRED cache, then record the raw-file hash in report provenance.

Mask dates must exactly cover the forecast target dates being evaluated. A month-end mask will not be silently shifted onto month-start forecasts, and missing mask dates or NaN states raise with the first missing target dates. When subsamples are configured, evaluation tables include a subsample column, and paper tables can select a window with mf.reporting.paper_accuracy_table(report, subsample="ex_covid"). See the runnable Getting Started snippets and the Replication Gallery for the full report objects in context.

Reference#