The Modern Shrine's AI agent on the Global Fortune Teller Competition benchmark: 36.7%

At The Modern Shrine, the production chat agent reads a person's Chinese birth chart and answers questions about career, money and relationships. Every AI product in this field says its readings are accurate. Almost none can say how accurate, and The Modern Shrine was no exception: the agent had been tested by reading its answers, which is not a number. Nobody could tell whether a change to the prompt or the model made it better or worse.

A team in Hong Kong had published a way to measure it. Their agent, Tianfu, was tested on real questions from the Global Fortune Teller Competition. It scored 50%. The top 20 human practitioners scored 53.5%. Claude Opus 4.6 on its own scored 40%. The thread had 1.6 million views, and the team published the benchmark.

The Modern Shrine's agent went through the same test.

The benchmark

MingLi-Bench (MIT licence) is built from the annual Global Fortune Teller Competition (2022–2025). It contains:

• 32 real people, each with a birth date, time, gender and country
• 160 multiple-choice questions, five per person, about things that actually happened to them
• One correct answer per question, from the competition's official answer key

This is a test set, not training data. The agent was never trained or tuned on it, and no prompt was changed before the run.

Measuring this field is hard, and the benchmark is honest about it. Open-ended readings cannot be checked. Multiple-choice questions about real events can. The price is that the test says nothing about tone, empathy or follow-up questions, which matter in a real consultation.

What a question looks like

Here is one person from the dataset, exactly as the agent receives them:

男命：1974年4月28日下午4:40分 出生地点：usa
(Male, born 28 April 1974, 4:40 pm, USA)

And three of the five questions about him:

#	Question	Options	Answer
1	What happened in 1996?	A. Severe depression · B. Returned to Hong Kong, met his wife · C. Car accident, unhurt · D. Unexpected money	A
3	What happened in 2008?	A. Lost a lot on stocks · B. Made money in a partnership · C. Grandmother died; he fell ill and needed long-term medication · D. Accident; found a hereditary illness	C
4	In which year did his father die?	A. 2009 · B. 2013 · C. 2018 · D. 2020	C

Each wrong option is plausible. Option 3A is a nice trap: 2008 was the financial crisis, so a model that reasons from world events instead of the chart will pick it.

What the agent sees

The agent never calculates the chart itself. A chart service computes it, and the agent receives structured data. For this person it looks like this (trimmed):

{
  "八字": "甲寅 戊辰 己亥 壬申",
  "日主": "己",
  "年柱": { "天干": { "天干": "甲", "十神": "正官" }, "地支": { "地支": "寅" } },
  "时柱": { "天干": { "天干": "壬", "十神": "正财" }, "地支": { "地支": "申" } },
  "大运": [
    { "干支": "辛未", "开始年份": 1996, "结束": 2005, "天干十神": "食神" },
    { "干支": "壬申", "开始年份": 2006, "结束": 2015, "天干十神": "正财" },
    { "干支": "癸酉", "开始年份": 2016, "结束": 2025, "天干十神": "偏财" }
  ],
  "刑冲合会": { "年": { "地支": { "冲": [{ "柱": "时", "知识点": "寅申相冲" }] } } }
}

In plain terms: the four pillars, the day master (己 Yin Earth), the "ten gods" relationships, ten-year luck periods with their dates, and the clashes inside the chart, such as the Tiger–Monkey clash between the year and hour pillars. The agent also gets retrieved passages from a curated knowledge base, and it can call tools for calendar data and date arithmetic.

How the test was run

The production chat route does a lot that has nothing to do with answer quality: login, bot checks, rate limits, saving messages, generating a chat title. I did not want to fake any of the parts that do affect the answer, so I wrote a runner that reuses them directly:

Same as production	Skipped
Chart service and chart snapshot	Login and bot checks
Full system prompt	Rate limits
Knowledge-base retrieval	Saving chats and messages
The same three tools and 8-step limit	Title generation
Same model (Kimi K2.5) and provider order	Analytics

From the benchmark I kept the exact Chinese question prompt ("analyse step by step, then answer in the form 答案：X") and its answer-extraction rules, ported to TypeScript. Then I added the scoring the Tianfu report used:

• 5 independent runs per 2025 question
• A question counts as correct only if at least 3 of 5 runs got it right
• Trimmed mean: accuracy per person, dropping the best and worst person, then averaged

I also ran all 160 questions once, for a breakdown by year and topic.

Was it the same chart a real user would get?

A test like this is only fair if the agent sees what a real customer would see. In production, a user's birth details go through one function that calls the chart service, and the result is saved to their profile. The runner calls that same function, with no time-zone or location adjustment, just as production does.

Two details needed checking:

• Time windows. 17 of the 32 people only have a two-hour window (for example 巳时, 9–11 am). The dataset records the start of the window. A real user would pick the window in onboarding, and TMS would use its midpoint. I built both charts for all 17. They were identical every time.
• The benchmark's own charts. The product's charts matched the reference charts the benchmark gives other models for 29 of 32 people. Two of the differences are people born just before midnight, where schools disagree on whether the day changes at 11 pm or at midnight. The third is a woman born on 15 February 1988: the reference chart gives her the previous year, 丁卯, by using Chinese New Year (17 February) as the boundary. The chart system changes year at 立春 (4 February), so her year is 戊辰, as the product's chart says. The published baselines were given the wrong year pillar for her.

The study used 360 agent runs: 4.3 million input tokens and 325,000 output tokens, about $3.55 at list price. Each answer cost about one cent.

The results

2025 competition (8 people, 40 questions)	Trimmed mean
Top 20 human practitioners (of 3,069)	53.5%
Tianfu Agent	50.0%
Claude Opus 4.6, no tools	40.0%
The Modern Shrine agent	36.7%
Random guessing	25.0%

Across all four years, the agent got 60 of 160 right (37.5%). The single run and the five-run majority vote gave the same 2025 score, so the number is stable. Four runs came back without an answer (more on that below). Retrying just those four moved the all-years score to 61 of 160 and left the 2025 score unchanged.

By topic, the pattern was clear:

Stronger	Weaker
Family: 59% (22 questions)	Marriage: 30% (44 questions)
Personality: 57% (14)	Money: 23% (13)
Career: 48% (25)	Education: 18% (11)

It answered in a median of 10 seconds. Tianfu, in my own use, takes three to five minutes.

Where it was right

On the 2008 question, the agent picked C and reasoned like a practitioner. 2008 sits in the 壬申 luck period. The Monkey (申) clashes with the Tiger (寅) in the year pillar, which stands for elders. Heavy water from the 子 year overwhelms the Earth day master, and Earth governs digestion, so long-term medication fits. It even named the financial crisis as the reason option A was tempting, and rejected it.

On the father question, it picked 2018. Its reasoning: the father star is the 甲 Wood in the year stem, and 2018 (戊戌) is the year 甲 Wood enters its "tomb", under heavy Earth. That is textbook timing work.

Where it broke

It held one reading too confidently. On the 1996 question it chose "unexpected money". It read the year's 丙 Fire as protective, and on that basis ruled out depression, the correct answer. A second practitioner, or a second school of thought, might read the same Fire as overheating the chart. The agent has nothing that makes it weigh one reading against another.

It changed its mind. In the five-run test, the agent gave the same answer on all five runs for only 12 of 40 questions. On the other 28, it picked different answers on different runs. That is not randomness in the chart. It is the model choosing a different reading each time, because nothing in the system decides between them.

It sometimes stopped before answering. In 4 of 360 runs there was no answer at all. Two ended on a line like "Now let me get the Chinese calendar data for 2020:": the model described a tool call but never made it, so the agent loop ended. The other two came back empty. Real users would see the same cut-off or blank reply. The benchmark found a production bug that manual testing had missed.

What the numbers say

The Modern Shrine agent is roughly as good as a strong general model given a chart: within one or two questions of Opus 4.6 alone, and about 13 points behind Tianfu.

The gap sits exactly where Tianfu's design differs from The Modern Shrine's. Their write-up describes:

1. 200+ small tools, so every calculation is exact instead of remembered
2. Expert rules written as tools the model calls when they apply, instead of rules buried in the prompt
3. Sub-agents with their own tools and context
4. Confidence scores, plus a step that reconciles conclusions from different chart systems (Zi Wei, Qi Men and others)

The agent's weakest topics, marriage and money, are timing questions that need several steps. Its unstable answers are the problem point 4 exists to solve.

Limits of this study

• 40 questions is small. One question moves the 2025 score by 2.5 points. The gap to Opus is within noise; the gap to Tianfu is not.
• Different charts. The published baselines were given charts from a different library (iztro) that also include Zi Wei. The Modern Shrine agent uses its own chart, without Zi Wei. The four pillars match for 29 of 32 people, but this is a test of the product, not a pure model-versus-model comparison.
• Multiple choice is not a consultation. The test cannot see whether a reading is kind, clear or useful.
• The comparison uses published numbers; Tianfu was not re-run for this study.

What gets built next

Each change gets scored on the same 160 questions, so its effect is measured, not assumed:

1. Fix the cut-off and blank replies. Detect an answer that is empty or ends in narration, and ask the model to continue.
2. Compare models. The runner switches models with one flag.
3. Add timing-rule tools for marriage and money, the biggest losses.
4. Add a reconcile step. Run several readings and make a final step choose between them with explicit reasons. Success shows up as more questions with the same answer on all five runs, not just a higher score.

The same three questions apply to any agent: what is it measured against, how often does it give the same answer twice, and what did the last change do to that number? If nobody knows, the agent has not been improved yet. It has only been changed.