✓ Survived Rank 1 of 18 proposedIndie Data Oct 20, 2026

Leakage-proof backtest audits for time-series models

Independent contamination and leakage audits so energy and finance teams can trust zero-shot time-series foundation model backtests.

Open the discussion ↗

A ready-to-paste brief for Claude Code, Cursor or any coding agent.

I want to build a small, bootstrappable product. Help me plan and start the MVP.

Product: Leakage-proof backtest audits for time-series models
What it is: Independent contamination and leakage audits so energy and finance teams can trust zero-shot time-series foundation model backtests.
Who pays: Heads of forecasting / quant researchers at energy trading firms, utilities, and macro pods adopting TSFMs — $10–50k per audit engagement or $1–5k/mo for a continuous evaluation subscription, from the model-validation budget.
Closest alternatives: GIFT-Eval / TIME / fev-bench public leaderboards (static, self-reported, can't r, In-house backtesting scripts built by each quant team, Nixtla utilsforecast / Darts evaluation utilities (metrics only, no contaminatio, Model vendors' own published benchmark reports
The gap: Leaderboards are one-off, public, and vendor-submitted; none can prove a foundation model's pretraining corpus did not contain the buyer's evaluation period, and none offer signal-to-noise-stratified evaluation on a customer's proprietary series. Each firm currently rebuilds this validation alone with no way to resolve the leakage question.
Why now: TSFMs trained on opaque web-scale corpora (Timestral, TimesFM-3, Chronos-2) only reached production-grade adoption in the last year, and comments show finance and energy teams migrating now — the train/test contamination risk is a new problem created by this model class.
Moat to build toward: Continuously generated post-cutoff evaluation series plus canary/honeypot series planted in public data sources compound over time; audit reputation and a growing corpus of verified-clean benchmarks are hard to replicate quickly.
Biggest risk: Model vendors preempt the market by publishing training-data manifests or timestamped clean-eval guarantees, making third-party audits seem unnecessary.

First three steps:
1. Build a contamination-detection harness: canary series probes and membership-inference-style tests run against Timestral, Chronos-2, TimesFM and TimeGPT via their public APIs/weights.
2. Assemble rolling, post-cutoff evaluation datasets for three verticals (power prices, macro series, retail demand) with signal-to-noise stratified test suites.
3. Publish free audit reports on the major TSFMs to build credibility, then sell paid audits on customers' proprietary series plus a subscription for re-validation on each model update.

Start by asking me up to five questions about constraints (stack, budget, timeline), then propose the smallest version a paying customer would use, and scaffold it.

Overall6.0/10

Pain6

Willingness to pay6

Timing7

Gap6

Moat5

Feasibility6

For

Who pays

Heads of forecasting / quant researchers at energy trading firms, utilities, and macro pods adopting TSFMs — $10–50k per audit engagement or $1–5k/mo for a continuous evaluation subscription, from the model-validation budget.

What people said · 4

A power-price forecaster says they cannot know whether Mistral's training data leaks into their backtest period, so out-of-sample validation of a TSFM is impossible for their domain. Read the comment ↗
Argues forecasting models should be tested on data-generating processes with known, varying signal-to-noise ratios, otherwise every model looks like SOTA and real skill is unmeasurable. Read the comment ↗
Points out that with only one observed universe, forecast models are inevitably overfit to it, which is why nobody ultimately trusts forecasting model evaluations. Read the comment ↗
A head of macro says their pod is migrating to this model, showing real money is already flowing into adopting models whose backtests can't be trusted. Read the comment ↗

Paraphrased from the discussion. Follow the links for the original comments.

Competitors and gap

GIFT-Eval / TIME / fev-bench public leaderboards (static, self-reported, can't rIn-house backtesting scripts built by each quant teamNixtla utilsforecast / Darts evaluation utilities (metrics only, no contaminatioModel vendors' own published benchmark reports

Leaderboards are one-off, public, and vendor-submitted; none can prove a foundation model's pretraining corpus did not contain the buyer's evaluation period, and none offer signal-to-noise-stratified evaluation on a customer's proprietary series. Each firm currently rebuilds this validation alone with no way to resolve the leakage question.

Why now

TSFMs trained on opaque web-scale corpora (Timestral, TimesFM-3, Chronos-2) only reached production-grade adoption in the last year, and comments show finance and energy teams migrating now — the train/test contamination risk is a new problem created by this model class.

Moat · 5/10

Continuously generated post-cutoff evaluation series plus canary/honeypot series planted in public data sources compound over time; audit reputation and a growing corpus of verified-clean benchmarks are hard to replicate quickly.

Against

Red team: The contamination problem is genuinely new, unresolvable by individual firms, and model-validation budgets at trading firms easily clear $10-50k per engagement — the main risk is it's a boutique consultancy, not a scalable product.
What would kill it: Model vendors preempt the market by publishing training-data manifests or timestamped clean-eval guarantees, making third-party audits seem unnecessary.

Start

First three steps: Build a contamination-detection harness: canary series probes and membership-inference-style tests run against Timestral, Chronos-2, TimesFM and TimeGPT via their public APIs/weights.
Assemble rolling, post-cutoff evaluation datasets for three verticals (power prices, macro series, retail demand) with signal-to-noise stratified test suites.
Publish free audit reports on the major TSFMs to build credibility, then sell paid audits on customers' proprietary series plus a subscription for re-validation on each model update.
Source discussion: Timestral: Mistral's time-series foundation models ↗ 613 points118 commentsFrom the commentsLinked article