The Thirty-Four Scrolls

Chapters of the quest

Thirty-four scrolls, one question

An apparel guild sent 34 different promotional scrolls to more than 1.3 million townsfolk. On each seal it carved a bundle of runes: a discount here, free shipping there, the word Exclusive, an exclamation point for good measure.

Brother Knowledge

The guild wants to know which runes earn their keep. The trouble is that its own Seers decided who got which scroll, so the raw tallies lie.

The quest has two halves. First, find the true effect of each whole scroll, despite the Seers' meddling. Then break every scroll into its runes and learn what each rune does on its own, and for whom.

In the paper's terms: estimating heterogeneous treatment effects (who responds) and the effects of heterogeneous treatments (which subject-line components matter), from observational data.

Three gates of the Great Sales Funnel

A rune that opens the first gate can still fail at the last.

The Gate of Opening. Did the townsperson open the scroll? Across the 34 scrolls, open rates ran from 1.7% to 31.7%.

The Gate of Purchase. Did they buy anything traced to that scroll? Rarely. A typical rune's purchase rate sat near 0.1–0.2%.

BOFU, the Bottom of the Funnel. How much gold did they spend, counting a zero for everyone who didn't buy? This is the number the guild's coffers feel.

Every effect in this book is measured at all three gates. Each scroll carried a hidden mark that traced a recipient from open to purchase, so nothing had to be guessed.

Email also dodges the usual display-ad headaches: nobody has to be online to receive it, no ad auction sits in the middle, and the same message reaches every device.

Granny's rigged tally

Two scrolls, identical except for the size of the clearance discount.

Granny says: the smaller discount wins at every gate.

Look at the bottom rows. The 20% scroll went to loyal regulars. The 30% scroll went to people who hadn't bought in years. Granny is comparing her customers, not her discounts.

Table 8 and Section 6.1.

The two Guild Laws

We can see what the Seers saw. The guild chose recipients using 15 ledgers: how recently and how often someone bought, how much they spent, a few demographics. We hold every one of those ledgers. So for any two townsfolk with the same ledgers, which scroll they got was down to chance.

Everyone had a shot at everything. If the Seers followed a rule with no exceptions, some kinds of customers would never see some scrolls, and there'd be nobody to compare them with.

One bar stands tall: the birthday scroll, sent only to people born that month. It looks like a broken law, but only about 80% of those eligible actually got it. That leftover chance is exactly what a fair comparison needs.

Table 5. Bars below 0.20 count as small differences. Dropping the five most lopsided scrolls leaves the findings materially unchanged.

Two witnesses are better than one

The Mage guesses

For every townsperson, she predicts what they would have done under each scroll, using machine learning on their ledgers.

The Diviner corrects

She knows how surprising each assignment was. A lapsed customer who happened to get the loyal-customer scroll is rare, so his response speaks for many like him and counts for more.

Combine the two and you get the paper's doubly robust spell: the verdict holds if either witness is right. It's a lock with two keys.

Brother Knowledge then takes that cleaned-up signal for every recipient and sorts it by rune and by customer type. That step is what lets the book later name each rune's effect.

The tailor who only fits you

Fitwell measures each customer and cuts a suit to match. On the people he has measured, his suits are perfect. On a stranger, they hang like sacks.

Flexible models do the same. If the model that memorized a townsperson is also asked to judge that townsperson, it hands back their own random noise and calls it a pattern.

Fitwell fizzles. Every prediction now comes from forests that never met the person being judged, so memorized noise can't leak into the verdict.

The paper uses random forests of 1,000 trees, which predict each person out of sample. A LASSO version gives similar results.

Opening isn't buying

Every scroll was compared with Scroll 34, a widely sent 40%-off offer. Sir Spurious insists that a scroll which wins at the first gate will win at the coffers.

Ten scrolls beat the baseline at opening. Half of those ten lost to it at BOFU. Counting opens as a cheap stand-in for sales can point the guild the wrong way.

Four scrolls with the very same 40% discount, compared with Scroll 34:

Same discount, opposite fates. Whatever else is carved on the seal decides the outcome.

Figures 1–3 and Table 9.

The genie's one wish

Averill grants a single number for all the non-price favors: free gifts, free shipping, half-off shipping and free returns lumped together. His answer: they do almost nothing.

Two help, two hurt. Free gifts and free shipping raise spending. Half-off shipping and free returns lower it. Averaged together they cancel, and the genie's "nothing" hides four real effects.

Discounts split the same way. Tie a percentage off to one named product and it hurts. Frame the very same discount as a clearance event and it's the strongest rune in the book.

Table 10 and Section 6.3.

What each rune does

Each bar is the effect of carving that rune on the seal, holding the other runes fixed. Gold helps, red hurts, faded means the effect could be zero.

Tables 10, 12, 13. Bars use a square-root scale so small runes stay visible; the scale stays fixed when you switch groups.

The engaged feel everything

Sorting customers by their ledgers splits them into two clans: engaged regulars who bought recently and often, and lapsed ones who drifted away. The same rune lands very differently on each.

You might expect seasoned customers to see through the tricks. Instead their experience amplifies nearly every effect, good and bad. The least engaged are the least moved, much like Ascarza's (2018) finding that the customers most likely to leave are not the best ones to target.

Tables 11–13.

Swap the scrolls

Back to Granny's two clearance scrolls. The Seers sent the deeper 30% cut to lapsed customers, the ones who barely react to discount size. What if the engaged had gotten it instead?

For a clearance sale, revenue really is the goal, so for once Mammonworth is right. The party can price the swap by reweighting what already happened, without sending a single new scroll.

Section 6.5, doubly robust off-policy evaluation scaled to the full population.

Relics claimed

Open access under CC BY 4.0. Characters and art from Dungeons & Datasets by James C. Reeder III. The villains are fictional; the findings are not.