The Sorting Forest

Chapters of the quest

A forest that sorts, and a seal that opens vaults

Deep in the realm grows a forest that sorts. Send merchants through it and its trees split them, again and again, until each merchant stands among others who respond to a gift the same way. Scholars call it a causal forest.

The question: does holding the Crown's Seal, a credit rating that opens the public bond market, let a merchant house carry more debt? Faulkender and Petersen asked exactly this in 2006.

Brother Knowledge

The catch: sealed houses cluster in trades where everyone borrows a lot. Compare across trades and the Seal looks twice as powerful as it does within a trade. A regression fixes that with a switch for each trade. A forest has no such switch. Our task is to build one.

Some trades simply borrow more

Builders, car dealers and hotels borrow heavily against their land and fleets. If those trades also hold more Seals, a forest that ignores the trade will credit the Seal for what is really the trade.

In regression, the trusted fix is a fixed effect: one switch per trade. A forest has no counterpart. Its leading software won't even accept a trade's name; it must first be turned into numbers, and the software suggests one switch per trade or “any natural ordering.”

Brother Knowledge

A forest with a trade control rests on the same assumption as the regression: within a trade, and given the covariates, who holds a Seal is unrelated to how much they'd borrow anyway. The forest simply drops the straight-line assumption. It just has to be told about the trade properly.

The forest only sees the order

Each tree splits merchants with questions like “is this number above or below that one?” So if you swap a trade's numbers for any others with the same order, every split, every estimate and every standard error stays exactly the same. The paper proves this holds for the whole causal-forest procedure.

In simulation, swapping the paper's feature for its plain ranks reproduced the main estimates in 2,033 of 2,040 runs.

Two laws for a good ranking

1. Rank fidelity. Rank the trades the way their true hidden effects would rank them, so trees can pool trades that are alike.
2. Assignment independence. Keep treatment from leaking into the ranking, or conditioning on it rebuilds the very confounding it was meant to remove.

Pick a method to see which law it keeps:

Clustering changes the paperwork, not the comparison

Clustering is the one group option the forest software offers, so it's the natural thing to reach for.

But clustering changes how trees sample merchants and how standard errors are computed. It leaves untouched which merchants the forest compares with each other.

No better than nothing. A cluster-robust forest is as biased as one that ignores the groups, at every group size.

Four hundred switches crowd out the clues

Give the forest a yes-or-no switch for each trade and every split can peel off only one trade. With many trades, the switches also crowd out the real clues.

In the real data, the forest considers only 41 of 415 features at each split. With 404 trade switches in the mix, it rarely gets to use firm size, the strongest predictor of who holds a Seal. Its guess of who is sealed shrinks to a narrow band:

Don't rank trades by their results

A popular shortcut from prediction work ranks each trade by its average outcome: target encoding and its James–Stein cousin. It's the forest version of controlling for a group's average outcome, a practice finance has long warned against.

A trade's average debt already contains how many of its merchants hold Seals. Rank by it and treatment leaks straight back in. The smaller the trades, the worse it gets.

Leave-one-out encoding is stranger still: its feature flips each merchant's own outcome back into the data.

Worse than nothing. By 100 groups, target and James–Stein encoding are more biased than ignoring groups entirely, and at 500 small groups their bias is close to −18%. Leave-one-out misses by −56.75% at 10 large groups.

Give each trade one honest number

1. Run the regression you'd run anyway. Outcome on treatment, covariates, year effects and a switch for every trade.
2. Keep each trade's coefficient. Give every merchant the number estimated for their own trade.
3. Hand the forest that one column. It can now pool trades that are alike, and still find differences along the trade itself.

At 250 and 500 groups the fix misses by about 0.55%, while the next best method, one switch per trade, is roughly eight times worse. With small groups, use the cross-fitted version.

The authors call this “JPR encoding.” Code is on GitHub.

Making the comparison the authors made

The authors' regression compares sealed and unsealed firms within the same industry. Adding industry effects lowers it from 0.084 to 0.074. Which forest makes that same comparison?

Table 8. Effect of a rating on market leverage; the share is how much of each estimate comes from comparing firms in different industries.

The Seal matters most where it's rare

The authors found the Seal matters most for small firms, using a straight-line interaction with size. The forest, with no line imposed, agrees and adds shape:

Where other credit is easy, the Seal matters less. In homebuilding the forest without a trade control estimates 0.102; the fix says 0.052. Builders, car dealers, hotels and cable companies already borrow heavily against collateral.

Section 5.3. Tested on held-out firms, the forests' predicted differences were slightly too small; regressions with interactions predicted differences three to six times too large.

Removing the trade bias isn't removing all bias

A placebo: give the “Seal” to firms that will only be rated in the next three years. A future Seal can't raise today's debt, yet every method finds a gap of 0.014 to 0.023. Firms about to be rated already borrow more.

How strong would a hidden firm trait have to be to erase the effect?

Section 5.4, omitted-variable bounds (Chernozhukov et al. 2026). The appendix applies the same comparison to Yang (2025), Crépon et al. (2015) and Hollingsworth et al. (2024).

Relics claimed

Characters and art from Dungeons & Datasets. The villains are fictional; the findings are not.