The Oracle's Quill
Brother Knowledge
“The quill writes; the oracle decides. One idea per page.”
Chapters of the quest
A quill that never stops writing
The guild learned which runes made its thirty-four scrolls work. Then it found a magic quill. Ask it for new scrolls and it writes them by the hundred, each one fluent and convincing.
The quill has never seen a single sale. It can write a scroll, but it can't know which one will work for this guild and these townsfolk. Our task: use what the old scrolls taught us to judge new ones before they're sent.
In the paper's terms: causal prediction for novel, unstructured treatments, here email subject lines written by generative AI.
Lorelei Linkbait
“Clicks are love. And everyone loves me.”
Fluent isn't the same as effective
Lorelei is the quill left to judge its own work. Ask her for her five best scrolls and she picks the ones that sound best.
Testing every new scroll on real customers isn't an option either. Each test spends customers on scrolls that may flop, and the quill writes far more scrolls than any guild could test.
We need a judge that has seen the sales: something that has learned, from the thirty-four real scrolls, how meaning turns into money.
Lorelei falls in Chapter VII, when her picks meet the oracle's.
Glurb
“Glurb absorbs every segment into one. Glurb is the segment.”
Same words, different deal
The old way to read a scroll was to list its parts: a checkbox for discount, one for free shipping, or a count of word pairs. Glurb swallows every scroll into that bag of parts.
Read these two subject lines:
- Free Shipping + 50% OffFree shipping, and half off the product
- Free + 50% Off ShippingThe product is free, and shipping is half off
A word-by-word reading sees the same parts in both lines, so it calls them twins.
The paper's example of why the encoder must respect context and word order. Bars are an illustration.
The Diviner
“Close on my map means close in meaning, not in spelling.”
A map where nearby means similar
A language model turns each subject line into 1,536 numbers that capture its meaning. Squeezed down to 28 directions, those numbers still keep over 99% of the information.
On this map, lines that mean similar things sit together, however they're worded: “Extra 50% off and Free Shipping” lands right beside “50% Price Cut and Free Delivery.”
The guild's thirty-four scrolls (illustration)
Tap a dot. Positions are drawn for illustration; the paper's map has 28 dimensions.
Most scrolls sit in tight neighborhoods. One sits far from all the others.
Trendwell
“My line goes up forever. I checked with my telescope.”
Know where the map runs out
The oracle predicts a new scroll by learning from its neighbors. Far from any neighbor, it's guessing, and Trendwell will happily extend his line off the edge of the world.
The paper tested this by hiding each old scroll in turn and predicting it from the rest. Typical neighbors sat 0.2 to 0.4 apart. The Happy Birthday scroll sat 0.68 from its nearest neighbor, and its prediction missed by 15 times the typical error.
How far is the new scroll from its nearest neighbor?
The cutoff sits at the second most isolated scroll. Inside it, use the prediction. Beyond it, run a real test.
Leave-one-out exercise and the novelty threshold. For 32 of 33 scrolls, uncertainty grew steadily with distance; only Happy Birthday broke the pattern.
Respondent 4,217
“Strongly agree. Strongly agree. Strongly agree.”
Can a prediction replace a live test?
The guild's usual choice is test-and-roll: try the new scroll on some customers, then send the winner to everyone. It works, but every customer in the test pays for the lesson.
The paper staged 992 duels: every new scroll against every old one, deciding which to send. Pick a way of reading scrolls and see whether it beats a live test on average:
Figure 7. Only contextual embeddings deliver positive average value over test-and-roll; every coarser representation fails to.
Duchess Bargaina
“A sale for everyone! The average says it works, darling.”
Let the quill write, let the oracle choose
- The quill writes. Generative AI proposes a batch of new subject lines using the guild's own ingredients.
- The map checks. Any line too far from the old scrolls is set aside for a real test.
- The oracle predicts. Every remaining line gets a predicted effect on spending.
- Keep the best five.
Across 30 fresh batches, the quill's own top five gave negligible or even negative predicted lift. The oracle's picks were higher in every batch, for both kinds of customer.
Rejection sampling with ChatGPT 3.5, Figure 8. Kept lines are real examples from Table W2; the rejected lines are illustrative.
Write for who's reading
The same oracle can predict for each kind of customer. Asked for each clan's best scrolls, it chose very different words.
Table W2, one representative batch. Segments come from clustering customers on spending and recency.
Relics claimed
The Map of Meaning
Read scrolls by what they mean, not by their parts. Only the language-model map beat a live test.
The Edge Marker
Trust predictions only near scrolls you've already sent. Past the edge, run the test.
The Sieve
Let generative AI write freely, then let evidence from your own customers decide what goes out.
The Clan Banner
The same oracle can pick different words for different customers.
The sequel to The Thirty-Four Scrolls. Characters and art from Dungeons & Datasets. The villains are fictional; the findings are not.