Churn Rate Analysison a 1,642-customer SaaS book

The client runs a SaaS product with both self-serve and managed accounts, and wanted a 90-day churn score for every customer on the book as of a 30 June 2026 snapshot — 1,642 customers to score, drawn from a larger 7,396-row historical file used for training.

The constraint shaped everything that followed: only interpretable model families were on the table — logistic regression and other linear models, a single decision tree capped at depth 5, naive Bayes, k-NN, and explicit business rules, alone or combined. No random forests, no gradient boosting, no neural nets, nothing an account manager couldn't be given a plain-language reason from. That rule is why the project turned into an exploration exercise rather than a modeling one — with an interpretability ceiling, the only way to raise performance is to understand the population well enough that the routing does the work the model can't.

Managed accounts churn 3.3x more often than self-serve — 38.5% against 11.6%, with Wilson 95% intervals that don't come close to touching (managed [36.7%, 40.4%], self-serve [10.7%, 12.6%]). A two-sample KS test separates the groups on 6 of 8 numeric columns at p < 0.0001. Pooling them into one model hides everything that follows.

More support tickets means a managed account in trouble — and a self-serve account that's engaged. Managed churn climbs from 20.4% to 50.7% as tickets rise (+30.3pp); self-serve falls from 23.0% to 6.7% (−16.2pp). Pooled, the two effects cancel into a mild upward slope of +0.105 — a sign that's wrong for all 4,202 self-serve accounts in the file. This is Simpson's paradox, and it's the finding the whole solution is built around: of ten candidate variables screened the same way, exactly one points the same direction in both groups.

Screening all columns rather than eyeballing them caught one that's bijective with the target and one that looks predictive but isn't:

The second one is the trap: a variable that would have looked like the best predictor in the file to anyone who ran a correlation matrix without segmenting first.

Inside the two-month renewal window, managed churn jumps to 66.5% (n=269) against 35.4% outside it (n=2,274) — a 31.2-point gap with disjoint confidence intervals. Self-serve shows no effect at all: 11.3% vs 13.1%, intervals overlapping. Pooled, this looks like a global 46.3% spike at month 2. It isn't — it's a managed-only rule, and reporting it globally would send a retention team after the wrong list.

The segment field itself was missing for 9% of rows, concentrated entirely in one source system. Contract length turned out to be deterministic at three of its four values — no managed account has ever had a 1-month contract, no self-serve account a 24- or 36-month one — which recovered 420 of 651 missing segments (64.5%) with zero counter-examples across 7,396 rows.

Each row is cleaned, the leak is dropped, and the row is routed — on account type if present, on contract length if not, otherwise into an honestly-scored "unknown" segment. That segment's own logistic regression, fit on that segment's own feature list, scores the row. No ensembling, no voting: one row, one model, one probability.

Regularization strength is itself a finding: managed lands on a strong penalty because its six features are informative and correlated; unknown, with only 185 rows, shrinks every coefficient below 0.016 — the model honestly saying it has nothing to work with rather than fitting noise. Runtime came in well inside a CPU-only, sub-millisecond budget: the full artifact is 2.4 KB and the pipeline rebuilds from raw data in under 20 seconds.

The retention team can make about 150 calls. So the metric that matters isn't AUC — it's: of those 150, how many really churn?

AUC is reported as a secondary number: it integrates over every threshold, including ones the team will never operate at. I also decomposed it honestly — knowing only which segment a customer is in gets 0.685 AUC on its own; the full routed model reaches 0.760. So the model adds roughly 0.075 AUC over segment membership alone, and that's the number I'd stand behind if asked what the modeling actually contributed.

This is the table the client actually needed. A single global odds ratio would have been actively wrong for one of the two segments on the single strongest signal in the file.

All five interpretable families were benchmarked on identical folds and features. Logistic regression led on both segments (0.726 / 0.698), but the margin over naive Bayes on managed was inside one standard deviation — AUC alone didn't settle it. Three things did: best calibration (Brier 0.139 vs. 0.143–0.147), the smallest artifact by two orders of magnitude (2.4 KB vs. k-NN's 546 KB, which stores the training set and grows with it), and the only family that reports a signed, per-variable direction — which is exactly what the effect-direction table above required. A depth-5 decision tree would have split on ticket count in both segments and never surfaced that the sign flips.

Who to call. The top 150 by predicted probability — in practice almost entirely managed accounts carrying open tickets and late payments, low NPS, many inside the two-month renewal window.

Who not to call. Two groups. The "unknown"-segment rows score at 0.535 AUC — close to random — so they go to engineering as a data-quality fix on the source system that drops the segment field, not a call list. And small self-serve accounts whose monthly fee sits below the cost of a retention call, regardless of predicted risk.

How to measure it. Not by comparing the called group's churn against the rest — that comparison is confounded by construction. Instead: take the top 300 by predicted probability, randomly assign 150 to be called and 150 held back, stratified by segment and risk decile, and measure the churn difference between arms.

What I wouldn't spend the two hours on: chasing AUC. Once the population structure is understood, the model class stops being where the remaining signal lives.

Handed a model-family constraint that most people treat as a handicap, I treated it as the spec: find the structure that makes a simple model correct, instead of reaching for a complex one that papers over not having found it. The population split and the direction reversal are the deliverable — the three logistic regressions are just where that understanding ends up living.