From a moving brief to 20 years of trusted macro data

In March 2026, a client engagement began with three documents: a macro framework, an indicator guide and a weekly scoring worksheet. The client scored the economy each week on 16 macroeconomic indicators (inflation, money supply, interest rates, the oil price, credit spreads, bitcoin's share of the crypto market and others) to decide which "regime" it was in: reflation, stagflation, goldilocks or recession. The point of knowing the regime is to know which assets, such as gold, silver, bitcoin or stocks, tend to do best in it. The 16 indicators themselves had come out of the client's conversations with Claude.

Before the engagement, the client had tried to collect the data with Claude Cowork, Anthropic's desktop agent. Out of the box it had no connections, permissions or API access to trusted data sources, so where it could not fetch a number it produced a plausible one. When the client asked it to build dashboards on its own, one run gave the oil price as US$64 when the client's own source showed about US$99. A regime score built on numbers like that is not a score.

Code: github.com/s0yabean/macro-regime-data-pipeline is a rebuilt, open-source version of the pipeline, runnable offline on a synthetic sample. Research tooling, not investment advice.

The pressure: a brief that kept moving

The engagement was paid advisory work, and the requirements moved during it. In the first weeks the ask went from an AI assistant that could discuss the data, to no AI at all and an algorithm that predicts the best-performing assets, to a scoring system covering 30 years. By the end of March it had become four tasks:

1. prove that the 16 indicators are the best predictors of the regime;
2. show which assets did better in each regime over the past 50 years;
3. turn that into an allocation framework, and project how different mixes of assets would perform;
4. find the signals that warn a regime is about to change, because that is when money gets moved.

None of these asks was unreasonable, and the client's instinct behind them was sound: a confident answer with no proof behind it is worth nothing. But each task hid a bigger question:

• "The best 16" compared with what? Proving they are the best means testing them against every other series in the world, and that search has no end.
• There was no answer key. Nobody had labelled which regime each past month belonged to, so there was nothing to test a model's calls against.
• 50 years did not exist for every asset. Bitcoin has about 15 years of price history, and USDT about 11.
• The scoring rules had gaps. The worksheet scored an indicator as rising or falling, but did not always say against what.

The model: requirements first, then sources, then time

The first delivery: a skill that only fetches data

The failure before the engagement was not a weak model. It was an agent with no fixed sources, free to fill gaps. So the first deliverable, inside the first week, was a Claude skill that removed that freedom. For each of the 16 indicators it:

• names the source. FRED, the St. Louis Fed's public economic database, is the first choice wherever it has the series: 11 of the 16 indicators. Gold and silver come from Yahoo Finance, the two crypto indicators from a crypto data provider, and two from monthly publications;
• names the backup. Five indicators that both FRED and Yahoo Finance carry (the dollar index, copper, oil, the yield curve and the VIX volatility index) switch to Yahoo Finance when FRED fails, and the output says so under "Backup plan triggered";
• labels the coverage. Every value is marked exact (the indicator itself), proxy (a close substitute, such as gold futures for the London gold price) or not_available, so a gap is reported as a gap;
• shows the date of every number, because a monthly figure read on 25 March may describe February, and reports where every scraped number came from, with the URL.

A skill like this makes the AI's job narrow: run the named scripts, read the labelled output, and say which numbers are proxies. It no longer decides where a number comes from.

The handover was a 12-minute recorded walkthrough. One command, about a minute of running, and a table of real numbers with dates. The demo run returned 15 of the 16 until Claude was given permission to install the Yahoo Finance library. The sources were free ones on purpose, or a subscription the client already paid for, because the first job was to test accuracy and reliability before spending on data. And the limit was stated on the recording: the skill was purely for obtaining data, and working out the regime was a later layer.

Where the chat window ran out: the 30 March call

The client spent a weekend pushing the skill towards the real goal, and came to the next call with a list of problems. The AI had recommended allocating to bitcoin in 1994. It had quoted a 65% chance of stagflation with no historical base rate behind it. It produced impressive dashboards, and the client could only spot the obvious errors, which raised the question of how many others were sitting in there unseen.

The diagnosis was that the skill worked and the tool had reached its ceiling. A chat has a limited context, and decades of data and correlations across 16 series go well past it. That kind of high-precision work suits scripts run against a database, with AI writing the scripts, not a data file dropped into a conversation. This changed what the client had to sign up for: someone would have to run those scripts, so either the client or a person they assigned. The client agreed to learn. Fetching current numbers stayed with the skill, and the project's centre moved to a database.

The same 35-minute call settled most of the open requirements:

Question raised on the call	What it surfaced	What was decided
Are these 16 the best?	The 16 came from Claude, so the client had no independent reason to trust them. They wanted fewer, not more: an investment professional had advised looking for redundancy first, and Claude had offered a list of nine	Keep all 16 for the first phase, because the sources were already working. Measure redundancy later on real data, where a few indicators may well turn out to do most of the work
Which regime was each month of last year in?	The client could not say, and there is no outside authority to ask, because the framework is their own	Regimes have to be inferred from the 16 indicators. The test of a regime is the client's: if the framework says an asset does well in that regime, the back-tested prices must show it doing generally well
What does "falling" mean for the dollar index: against last week, or last month?	CPI had a clear rule (3-month rate above 12-month rate and rising). The dollar index did not, and "easing or tightening" was hard to quantify at all	The rule has to come from the client. A rule invented during the build could quietly change what the framework measures
How much data, exactly?	All 16 indicators for 50 years, and also the price history of the assets being tested, about five per regime	The client would list the assets. That list became the April data specification: 49 series, the oldest starting in 1934

A wrong number that was not wrong

Midway through that call, the client noticed the skill showing the US dollar index at about 120 when their own screen showed about 100. After a weekend of hallucinations it looked like one more. So the call stopped and checked it against FRED, and FRED showed 120 too. The skill had returned the correct value of the nominal broad dollar index, a different measure from the market DXY that traders quote at about 100. Both numbers were right.

The way I put it on the call: "that's a domain question, not a technical question." A builder can fetch whichever series is asked for, but in finance one name can cover several series, and only the person who designed the framework can say which one they meant. The client preferred the market index, so it is now the primary source in the catalog, with FRED's broad index as the fallback. It collects cleanly and needs no key: 5,243 daily rows from January 2006, no gap longer than five days, and all nine health checks pass. The client was also asked to check the other 15 the same way. The catalog records a display name, a provider code and units for every series, so it is clear which one a number came from.

Fixing the rest of the scope

The database was built in the order those decisions implied. Task 2 needs regimes to be defined first, task 3 needs task 2, task 4 needs a history of regimes to move between, and all of them need the same data. So regime scoring, tuning the algorithm and a full backtesting system became a separate phase, and three more things were agreed with the client:

Question	What was fixed
How is the data organised?	A database design agreed with the client before any data was pulled
How far back?	2006 to April 2026, about 20 years: the window most sources could fill. The client agreed bitcoin data would start in 2013
How is it checked?	A small pilot first (two years, at most 100 rows per series) for the client to inspect, then the full backfill

When the sources pushed back

Agreeing the scope did not make the data cooperate. Each source came with its own limit, and each limit forced a tradeoff that is now written into the catalog (docs/data-sources.md):

Problem	Tradeoff made
The client expected one charting platform to hold all the data, but its API does not readily serve market data	One source per series, each chosen for its own API
ISM's manufacturing index has no free API, and ISM's site blocks automated access	The skill tried ISM's monthly report pages, then the previous month's, then a third-party chart, and took pasted report text when a CAPTCHA blocked it. The rebuild stopped scraping and loads a hand-copied monthly file
The link for central-bank gold purchases in the data specification was dead, and the World Gold Council publishes those figures about two months late	A monthly file with a 60-day release lag
FRED removed the London gold price in 2022	Gold futures as a stated proxy, marked proxy in the catalog
Mid-project, the crypto data plan in use was cut to two years of history, and even the paid plans start in 2013	Coinranking for history back to 2013, agreed with the client. Bitcoin dominance is only weekly that far back, and 69 of 2,753 days (2.5%) of stablecoin data were missing USDT, USDC or both
The Fed reports the reverse repo in billions and its balance sheet in millions	Units normalised on the way in, before the three are combined

The two monthly files are the one place a person types numbers in. The loader rejects any row whose "available from" date is earlier than the release lag allows, so a hand-copied number cannot appear before it was published.

The hardest part: when could each number have been known?

A regime model's inputs update on different schedules. Prices are daily. The Fed's balance sheet is weekly. CPI, M2, the fed funds rate and Treasury yields are monthly. And most of them are published well after the period they describe.

FRED labels US consumer prices for September 2025 with the date 2025-09-01, the first day of the month the number describes. The US Bureau of Labor Statistics publishes that number in mid-October, about six weeks later. A model that lines macro data up with market prices by the label date can "know" September inflation before September has ended, and nothing raises an error. Using data before it was published is called look-ahead bias. Any backtest built on it is graded on information no investor had.

The delivered version guarded against this. Every row had an observation_date (what the value describes) and an available_at (the first date a strategy may use it), and nine series carried a release lag. When the pipeline was rebuilt and open-sourced in October 2026, two processing bugs turned up in how those dates and series were built.

Bug 1: lags counted from the label date. The ingest helper computed availability as the label date plus the lag:

# delivered version
return (obs + dt.timedelta(days=release_lag_days)).isoformat()

For CPI, with a 14-day lag, September 2025 (label 2025-09-01) became available on 2025-09-15. The lag was meant to mean "about two weeks after the month ends", but it was added to the first day of the month. Daily and weekly series were unaffected, because their labels already sit at the end of the period. All six monthly series were affected: CPI, M2, the fed funds rate, 10-year and 3-month Treasury yields (14 to 28-day lags) and copper (30 days). Replaying both rules for every month from January 2019 to December 2025:

Measure (6 monthly series, 84 months each)	Result
Prints affected	504 of 504
Average days early per print	29.4 (one day less than the month's length)
Prints visible on or before the last day of the month they describe	464 of 504
Days from 2019-03-01 to 2025-12-31 on which a point-in-time read returned a print that was not yet published	2,416 of 2,498 (97%), for every series

On almost every day of a seven-year test, the inflation, money-supply and rate inputs would have been one month ahead of what anyone could have had. The rebuild counts availability from the end of the period:

# rebuilt: available_at = end of period + release lag
end = period_end(observation_date, frequency)   # 2025-09-01 monthly -> 2025-09-30
return (end + dt.timedelta(days=release_lag_days)).isoformat()   # -> 2025-10-14

A unit test pins it: as of 2025-10-10, the point-in-time query returns August CPI (available 2025-09-14), not September CPI (available 2025-10-14).

Bug 2: zero-filled stablecoin supply. Total stablecoin supply is USDT plus USDC. The CoinGecko ingester and the Coinranking export took every date in either series and filled the missing side with zero:

# delivered version
all_dates = set(usdt_map) | set(usdc_map)
combined[d] = usdt_map.get(d, 0.0) + usdc_map.get(d, 0.0)

The production series was loaded with a strict rule written in SQL, so its values were right. A rerun from either script would have brought the gaps back. The risk is in the mix: across the last 365 days of CoinGecko data, the median USDC share of the combined supply was 28.8%. One missing USDC day would read as a 29% one-day drop in stablecoin supply, and one missing USDT day as a 71% drop. In a model that reads stablecoin supply as a liquidity signal, that is a fake crisis created by a data gap. The long history had 69 incomplete days, each a chance for one. The rebuild keeps a day only when every input exists on it:

def strict_sum(*maps):
    shared = set(maps[0]).intersection(*maps[1:])
    return {d: sum(m[d] for m in maps) for d in shared}

A missing day now shows up as coverage below 100%, which a health check can see, not as a number that looks real.

The structure: a catalog, one time rule and health checks

The fixes are small. The structure around them is what keeps them fixed.

1. The catalog is the contract. Each series in config/series_catalog.yaml declares its provider, units, frequency, release lag and exact or proxy coverage, with the reason in the docs (CPI 14 days after month end, M2 28, central-bank gold 60). Changing a source or a lag is a reviewed one-line edit.
2. One function owns the time rule. compute_available_at(observation_date, release_lag_days, frequency) is called by every ingester: FRED, Yahoo Finance, the crypto providers and the monthly files.
3. The database refuses impossible rows. raw_observations has CHECK (available_at >= observation_date).
4. Nine SQL health checks run after every ingest: dates well-formed, no value available before its period, catalog lag respected, monthly and quarterly lags counted from the end of the period, observed frequency matches the catalog, every active series has rows, nothing stale, no non-positive prices, and the last run per provider succeeded. Each check returns its failing rows, so a failure names the series and dates.
5. Readers only get point-in-time queries. The query tool and the agent skill filter on available_at <= as_of, and the skill makes an agent run the health check first and cite both dates in every answer.

The client was set up to query the pilot database through an AI coding assistant, in an in-person session. That met the original wish for an AI that could discuss the data, from a source it could not invent.

Which checks would have caught the bugs

For bug 1, the original checks would not. Rebuilding the synthetic sample with the old rule, label date plus lag, on all 191 monthly rows, all eight health checks in the repo at the time passed. The lag_respected check measures from observation_date, the same label date the bug used, so a label-date rule satisfies it exactly. It catches a lag of zero, not a lag counted from the wrong day.

A check that measures from the period end does catch it. This is its monthly part:

-- monthly rows available before end of period + catalog lag
SELECT ro.series_id, ro.observation_date, ro.available_at
FROM raw_observations ro
JOIN series_catalog sc USING (series_id)
WHERE ro.frequency_observed = 'monthly'
  AND ro.available_at < date(ro.observation_date, 'start of month',
        '+1 month', '-1 day', '+' || sc.release_lag_days || ' days');

On the sample built with the old rule it returns 191 rows; on the correct sample, 0. It is now the ninth check, lag_from_period_end in sql/health_checks.sql, with a unit test that it fails on label-date data and passes on correct data.

The pattern: a check written in the same terms as the code it tests tends to share the code's blind spot. The checks that caught these bugs were written from the outside, from a known publication date. The same idea runs through the article quality gate on this site: a gate is only useful once it has been seen to fail on the problem it is meant to stop.

Results

What	Result
Scope	The client's 16 indicators plus the asset prices they listed, with regime scoring and backtesting set aside as a separate phase
Production database (May 2026)	192,971 rows, 53 active series, 2006 to April 2026, no active series empty; walked through with the client in person before the engagement closed
Stablecoin supply	2,684 of 2,753 days kept (97.5%) under the strict rule; the 69 incomplete days are dropped, not zero-filled
Monthly look-ahead (rebuild)	504 of 504 prints from 2019 to 2025 now appear after their period ends, instead of 29.4 days earlier on average
Rebuilt repo	15 unit tests pass offline; synthetic sample of 53 series and 22,887 rows, 9 of 9 health checks pass; keyless real-data run, 9 of 9 pass

What comes next

• Write the known-date test before the ingester. One assertion ("as of 10 October, CPI is August's") would have caught bug 1 on the first run. Each source's publication schedule is a test fixture, not a comment.
• Show every check failing once. Building the old rule into a copy of the sample took minutes and found the gap.
• Default to strict joins. A series built from several inputs drops a day when an input is missing, and reports coverage. Filling with zero needs a written reason.
• First-release values and real calendars. Store first-print values from ALFRED, FRED's archive of past versions, so backtests see numbers as first published rather than revised. Then replace fixed day counts with release calendars from BLS, the Fed and ISM.
• Then the modelling phase: regime scoring on the point-in-time view, asset behaviour by regime, and transition signals, each tested against this database.

Research tooling, not investment advice.