The assignment: decide what a partner pilot established
A fictional software company invites twenty eligible accounts into a partner-assisted sales pilot and assigns twenty to a holdout. Ten invited accounts buy; six holdout accounts buy. The partner dashboard claims eight tagged wins. Your task is to reconcile those statements, calculate the pilot’s incremental contribution under explicit assumptions and decide what evidence should come next.
The exercise stipulates randomized invitation at the account level, equal sixty-day follow-up and no spillover between arms. The rows are constructed teaching material, not observations from an actual randomized trial. The assignment therefore teaches how to analyze such a design; it does not establish a real treatment effect or a market benchmark.
NIST’s discussion of randomized designs explains why assignment of experimental units matters. For this case, the experimental unit is an account, not a referral row or an individual contact. Keep every account in its assigned group even when it never receives a referral or never buys. Otherwise you would select the treatment group partly using outcomes.
Data and worked inputs
Sources[1]
Keep referrals, tagged wins and the group comparison separate
The referral file has thirteen rows but twelve unique referral IDs because R03 is retried. Twelve introductions do not equal twelve agreements. The account file shows ten wins in the invited arm, eight of which are partner-tagged under the exercise’s commercial rule. Two invited-arm wins have no partner tag.
The two arms contain twenty accounts each. Conversion is 50% in the invited group and 30% in the holdout, a difference of twenty percentage points. Multiplying that difference by twenty invited accounts gives a point estimate of four additional wins for this cohort under the stipulated design. The eight tagged wins answer an attribution question, not the counterfactual question of how many would have bought anyway.
Do not remove unconverted invited accounts from the denominator or compare referred accounts with all holdout accounts. Referral receipt can itself be affected by the pilot. Keeping original assignment is what preserves the intended comparison. In actual work, verify that the assignment and observation conditions really occurred instead of copying those assumptions from a template.
Charge costs according to the agreement, not according to the estimated lift
Each converted customer generates $10,000 of first-year revenue and has $4,000 of stipulated variable delivery cost. Contribution before program costs is $6,000 per customer. The partner receives 10% of revenue for the eight tagged wins, costing $8,000. The ten invited wins require $500 each of additional onboarding, costing $5,000. Setup costs another $4,000.
| Item | Amount |
|---|---|
| Incremental wins, point estimate | 4 |
| Contribution per win before program costs | $6,000 |
| Estimated added contribution | $24,000 |
| Commissions on eight tagged wins | −$8,000 |
| Onboarding for ten invited wins | −$5,000 |
| Fixed setup | −$4,000 |
| Net incremental contribution, point estimate | $7,000 |
Total incremental program spending is $17,000. The four-win point estimate contributes $24,000 before those costs, leaving $7,000. Do not charge commission on only four wins: the commercial agreement in the case pays eight tagged wins, regardless of your estimate of incrementality. Analytical credit and an actual payment obligation are not the same quantity.
The point-estimate contribution is not fully loaded company profit. It includes the costs specified in the case and a first-year horizon, but no unmodeled lifetime benefit, later renewal or residual value. Three additional wins would produce $1,000 above the stated costs; two would leave a $5,000 shortfall. The continuous break-even is about 2.83 additional wins.
The exact comparison does not make a small pilot conclusive
There are sixteen wins across forty accounts. Under the sharp no-effect comparison with twenty accounts per group, the reference calculates a fixed-margin hypergeometric distribution. It sums tables with probability no greater than the observed table’s probability, giving a two-sided Fisher/permutation p-value of approximately 0.3332.
This is not a 33.32% probability that the partner has no effect. Nor does the result prove that the program is ineffective. It says the observed split is not unusual enough under that particular no-effect comparison to provide strong evidence by itself. The reference includes the full enumeration so a reviewer can inspect the test rather than accept an unexplained significance label.
Your commercial conclusion should retain the positive point estimate and the weak precision. Do not manufacture certainty by reporting eight attributed wins as eight incremental wins. A bounded next pilot might be worthwhile if its cost is justified, assignment is maintained and the success criteria are specified before observing the next results. The decision to gather more evidence still has an opportunity cost.
A complete pilot review that does not oversell the result
The strongest memo does not choose between an enthusiastic sales story and an empty “insufficient data” conclusion. It gives the observed comparison, the economics and a concrete next decision.
Original fictional pilot review
Build the reconciliation and test its assumptions
Make an account-level table containing assigned arm, conversion outcome, revenue, contribution and partner-tag status. Link referral records by account ID without letting repeated introductions create additional accounts. Use separate columns for measured outcome, attributed credit and payable commission.
Reproduce the financial result before calculating the exact test. Then change the setup cost or per-customer contribution and report the break-even incremental wins. These changes alter the commercial decision without changing the conversion evidence. Conversely, a longer follow-up or a different eligibility rule changes the study and should not be silently combined with this cohort.
The advanced extension is to consider spillover: what if holdout accounts also receive partner help? Describe the evidence needed to detect that problem and the question the comparison would then answer. Do not simply reassign accounts after seeing which ones converted. The project rewards protecting the comparison, not obtaining the most favorable result.
Data and worked inputs
Use the project to demonstrate commercial and statistical judgment
A credible application description is: “Built a synthetic partner-pilot evaluation separating referral attribution, assigned-group conversion and contribution after actual program costs.” Add the numbers only when they help explain what you learned. Do not claim a real partner relationship or a measured increase in a company’s revenue.
Be ready for the question “Why pay commission on eight customers if only four are incremental?” Answer that the payment rule and the causal estimate serve different purposes. Another useful question is whether a larger p-value means the program cannot be profitable. It does not; uncertainty about the effect and the economics of possible effects must be considered together.
For a partnerships role, foreground the agreement and operating handoff. For an analytics role, foreground assignment, identity and estimation. The same work sample can serve both, but your explanation should respond to the responsibility in the actual posting rather than recite every calculation.