The Unreliable Narrators

Chapters of the quest

A quarter million field reports, unread

A guild of healers' suppliers sends scouts to thousands of shops. After every visit, the scout writes a short field report: who they saw, what was said, what seemed to be coming.

The guild's treasurer forecasts next season's sales from past sales and a few tallies. The reports just sit in the archive.

Brother Knowledge

Most managers think their forecasts could be better, yet only 15% of companies use these reports. Our task: find out whether the scouts' words can sharpen the forecast, and when.

In the paper's terms: CRM activity logs from a U.S. medical device maker, read with large language models, used to forecast customer sales.

The shopkeeper never writes the report

When people predict from customer reviews, the customer wrote the words. Here, the shopkeeper whose purchases we're predicting never writes anything. A scout tells their story.

That story passes through three hands, and each can bend it:

1. The scout reads the shop. They notice some cues and miss or misread others.
2. The scout writes it down. Later, briefly, and in their own style.
3. The model reads the report. It can only find what made it onto the page.

The paper's narrator and protagonist framework, from information processing theory. The message is an illustration.

Does this report point up or down?

A language model read every report and judged whether it pointed to rising sales, falling sales, or neither. It read each one twenty times and only called a direction when all twenty readings agreed.

About 19% of reports pointed up and 15% pointed down. The rest gave no clear signal.

Try it on a few made-up reports:

Tap a report to see how it would be read.

On average, an upbeat report was worth about a tenth of an extra sales visit, and a downbeat report about a fifth. Real reports are confidential; these are illustrations.

A few scouts carry the signal

Averill pools every scout together, and the reports look only mildly useful. Split them up, one scout at a time:

Only some scouts are worth reading. For about 27% of scouts, upbeat reports reliably predicted rising sales, and for about 32%, downbeat reports predicted falling sales. Among those scouts the effect was 17 times (upbeat) and 6 times (downbeat) the pooled average. About 70% showed no consistent signal at all.

If every visit is wonderful, none is

Some scouts call every visit promising. Others call every visit grim. Either way, their words stop carrying news.

How cheerful is this scout about their other shops?

Table 3. The more upbeat a scout's other reports, the less an upbeat report predicts; the same holds for gloomy scouts and downbeat reports. The bar is an illustration of that direction.

Add the reports, and the forecast gets worse

Three forecasts of each shop's coming sales, tested on weeks they had never seen. Lower error is better:

Past sales alone do worst. Adding the guild's tallies helps a lot. Adding the reports for everyone at once makes the forecast slightly worse.

Section 2.7, outside sales agents. Mean absolute error in the held-out weeks.

The scouts worth reading

Either way, the scouts whose reports helped wrote differently. The useful signal lives in how a report is written, not only in what it's about.

Section 2.7.3–2.7.4 and Table 6.

Train the narrators

Respondent 4,217 writes the same report every time. Defeat him by checking off what the paper recommends:

Treat reports as valuable, never as automatically reliable. Find which scouts' words carry signal, use those, and help the rest write better.

Section 4.2, implications for practice.

Relics claimed

Characters and art from Dungeons & Datasets. The villains are fictional; the findings are not.