Study Overview
The business problem. Acquiring a new B2B customer costs five to seven times more than retaining an existing one. Customer churn costs U.S. businesses an estimated $136 billion annually. To make that concrete: a mid-sized B2B operation carrying a $55 million annual-revenue customer portfolio — the scale modeled in this study — can lose several million dollars a year to departures that were detectable months in advance, simply because no one was watching for the right signals. Yet most organizations either deploy no AI for departure detection, or spend years and millions of dollars cleaning and labeling data before any model runs. This study demonstrates a governance-first alternative: a supervised AI departure detector built without a single historical departure label — trained entirely on synthetic behavioral episodes generated from domain expertise. Behavioral surveillance and anomaly detection are the ideal entry point for organizations new to AI: low data requirements, interpretable outputs, measurable economic value, and no multi-year data preparation program required.
What was built. Five model architectures were evaluated against 175 synthetic B2B customers observed over 156 weeks. Every model was scored against the same locked ground truth using an economic framework — Economic Detection Potential, False Positive Penalty, and Net Economic Value — rather than statistical accuracy alone. The result: a corrected Sequential Fusion architecture that combines a Markov Switching Model (Stata), an LSTM trajectory challenger (MATLAB), and a Fourier amplitude detector (Python) to achieve +$3,323k net economic value with only $278k in false positive costs.
How it was built. This study was completed in less than one week of elapsed effort. Claude (Anthropic) wrote all data pipelines, feature engineering scripts, the scoring harness, the GAP branch detector, and the corrected fusion architecture. ChatGPT (OpenAI) implemented the MATLAB Hidden Markov Model and LSTM challenger. The human author served as quality manager throughout — defining architecture, demanding proof for every claim, cross-checking every number against source CSV files, and running OCC 5C governance reviews at each session transition. When AI systems made errors — fabricated results, logic inversions, double-counting — the author identified them through governance challenge and corrected them. The OCC framework is what made the speed possible without sacrificing accuracy. When AI writes the code, schedule risk nearly disappears — quality risk becomes the binding constraint. AI-era project management monitors evidence quality, not milestone dates. This is the paradigm shift.
Traditional machine learning demands clean predictor variables — imputed, normalized, de-duplicated, and validated across systems before a single model runs. Organizations routinely spend millions and years on data engineering before AI development begins. This study demonstrates a different path: the only input required was a clean weekly order history. No CRM integration, no labeled churn outcomes, no external enrichment. Domain expertise converted into synthetic behavioral episodes replaced years of historical label collection. The entire project was completed in less than one week of elapsed effort using AI-assisted development governed by the OCC quality framework.
A full working paper documenting the methodology, governance findings, and twenty lessons learned is available for download below.
Summary metrics
Best net economic value
+$3,323k
Sequential Fusion v2
Two-stage fusion net value
+$3,323k
MSM surveillance · LSTM filter
Best discrimination (AUC)
0.858
LSTM + Fourier hybrid
Lowest false positive penalty
$278k
Sequential Fusion v2
Lifecycle stage
POC Complete
Environment setup → system test → UAT → deployment
Validation coverage
175
All customers verified · zero discrepancy vs dashboard
Five-architecture comparison — all 175 customers · locked formula · CSV-verified
| Architecture |
AUC | TP | TN | FP | FN |
EDP | FPP | Net | Prec | Recall |
| Markov Switching ModelStage123 · max WoE v2 |
0.5260 | 15 | 44 | 113 | 3 |
$3,405k | $2,748k | +$656k | 0.098 | 0.833 |
| Hidden Markov ModelStage123 · state estimation |
0.5722 | 15 | 25 | 132 | 3 |
$3,571k | $3,690k | −$119k | 0.102 | 0.833 |
|
Isolation Forest best net
Stage123 · anomaly detection
|
0.4391 | 15 | 22 | 135 | 3 |
$3,866k | $3,680k | +$186k | 0.100 | 0.833 |
|
LSTM + Fourier Hybrid challenger
Trajectory learning · routed architecture
|
0.8581 | 16 | 138 | 18 | 2 |
$1,567k | $280k | +$1,288k | 0.471 | 0.889 |
|
Sequential Fusion v2 best precision
MSM watchlist · LSTM confirmation filter
|
— | 16 | 140 | 16 | 2 |
$3,601k | $278k | +$3,323k | 0.500 | 0.889 |
Visual analysis
Net economic value ($k) by architecture
Confusion matrix components by architecture
Precision vs recall — operating points
Economic Detection Potential vs False Positive Penalty ($k)
Sequential Fusion v2 — three-component architecture
Component 1 · Surveillance
MSM Stage123
Stata 18 · max post-burnin WoE · 2 dB threshold
Scores all 94 CONTINUOUS customers.
Broad detection net — all 94 enter the watchlist at 2 dB.
Sets detection timing and action intensity for confirmed customers.
Component 2 · Confirmation
LSTM+Fourier Hybrid
MATLAB R2025b + Python · trained on synthetic episodes
Filters the 94-customer watchlist.
Cumulative WoE ≥ 400 dB → confirm LEAVING (8 customers).
Below threshold → remove, reclassify STABLE (86 customers).
Component 3 · Independent
GAP Fourier Branch
Python 3.12 · amplitude dropout detection
Handles all 81 GAP customers independently.
80 PERIODIC: Fourier amplitude dropout scoring.
1 IDIOSYNCRATIC (C065): ABSTAIN · $0 cost · IOR monitoring only.
All three components run in sequence. CONTINUOUS customers pass through Components 1 and 2.
GAP customers go directly to Component 3. Ground truth was withheld from all components during construction.
Results verified against locked CSV decision files using the OCC CSV Auditor.
Routing and confirmation flow
Stage 1 — Broad surveillance net
All 175 customers→MSM (Stata) max post-burnin WoE scoring · 2 dB threshold
WoE ≥ threshold→94 CONTINUOUS customers enter watchlist (all flagged at 2 dB)
Not flagged→STABLE · no further processing
Stage 2 — Precision confirmation filter
CONTINUOUS watchlisted→LSTM cum WoE ≥ 400 dB → confirm LEAVING
CONTINUOUS watchlisted→LSTM cum WoE < 400 dB → remove · STABLE
GAP PERIODIC→Fourier amplitude branch · independent
GAP IDIOSYNCRATIC→IOR interval · ABSTAIN · $0 opportunity cost
Key findings
Architecture choice is an economic decision
Under the locked formula, net economic values range from −$119k to +$3,323k across five architectures. AUC rankings and economic rankings diverge sharply. The LSTM+Fourier Hybrid has the highest AUC (0.858) but not the highest net value (+$1,288k). Sequential Fusion v2 achieves the highest net value (+$3,323k) with a precision of 0.500 — catching the right customers and not wasting intervention budget on false alarms.
Filter, don't escalate — the fusion logic that matters
Fusion v1 used the LSTM to escalate Stage123 alarms — inheriting all 138 false positives. Fusion v2 uses it to filter: Stage123 alarms not confirmed by LSTM are removed from the watchlist. FPP drops from $1,953k to $278k. Net improves by $1.3M. The direction of the LSTM's role is the entire difference between a model that compounds errors and one that corrects them.
Governed acceleration — one week, five models
Claude wrote all pipelines and the scoring harness. ChatGPT implemented the MATLAB HMM and LSTM. The human architect provided direction and governance challenge. When AI systems made errors — fabricated results, double-counted costs, inverted logic — the author identified them through OCC 5C structured review. The OCC framework is what made the speed possible without sacrificing accuracy.
What makes fusion v2 work — exactly
MSM (Stata) flags all 94 CONTINUOUS customers at 2 dB — a broad net by design. LSTM evaluates each independently: 8 reach cumulative WoE ≥ 400 dB and are confirmed LEAVING; 86 fall below and are reclassified STABLE. The 400 dB threshold sits cleanly between the stable 95th percentile (119.7 dB) and the leaving minimum — a 280 dB separation gap. Early detection timing from the MSM is preserved, so EDP credit for catching customers early accrues correctly.
Project management paradigm shift
When AI writes the code, schedule risk nearly disappears. Quality risk does not — it becomes the binding constraint. Five architectures were built in under one week, but the majority of human effort went to quality management: demanding proof for every claim, verifying that numbers foot, documenting governance events. Evidence quality replaces schedule adherence as the primary project management challenge. The OCC 5C framework is the instrument of that quality management.
Validation coverage — 175 customers, zero discrepancy
The Customer Validation Workbook (178 tabs) provides step-by-step verification for every customer: registry facts, model decisions across all architectures, step-by-step economic calculation, and weekly score trace. The CROSS-VALIDATION tab confirms zero difference between bottom-up tab summations and locked dashboard values — EDP, FPP, Net, and all confusion matrix cells — across MSM, LSTM+Fourier Hybrid, and Fusion v2. No faith required: every number traces to a source CSV.
Project lifecycle: POC → environment setup → system test → UAT → deployment
POC designation. This study is designated a Proof of Concept. The models work, the governance works, the economics are verified, and every number traces to a source CSV. The POC qualifier applies to the environment — MATLAB desktop, Stata do-file, Python scripts running on a research workstation — not to the evidence quality. The validation workbook confirmed zero discrepancy across all 175 customers in the research environment. That is POC-level system testing: it certifies the arithmetic is correct in the environment in which the code was written. It does not certify the system is ready for operational use.
Environment setup is next. Before any formal system testing or UAT can occur, the software and data must move to appropriate environments. Platform, licensing, and infrastructure decisions for each model component will be evaluated in this phase in consultation with IT and the relevant platform vendors. Representative data — either a sanitized subset of real customer records or a recalibrated synthetic dataset — must be staged in the target environment. IT approval of the platform stack is a prerequisite, not a parallel track.
System testing in the target environment follows. Once the stack is deployed on approved infrastructure with representative data, formal system testing re-verifies that models produce correct outputs in the new environment — confirming the migration introduced no errors, the scoring harness operates correctly, and audit file generation is intact.
UAT follows successful system testing. User Acceptance Testing is operationally focused. It asks: Do account managers find the departure risk scores actionable? Does the action ladder align with how the organization actually responds? Are the scoring cadence and reporting format compatible with existing workflows? UAT requires real users in a representative operational context and cannot begin until system testing has passed.
Controlled deployment follows UAT. A limited production rollout — one business unit or customer segment — with defined success metrics, a rollback protocol, and ongoing OCC governance reviews.
Downstream deployment decisions by component. Platform, licensing, and environment alternatives for each model component — Stata MSM, MATLAB LSTM, and Python scoring harness — will be evaluated in consultation with IT and the relevant platform vendors during the environment setup phase. The fusion decision CSV is the natural integration point for any downstream system. OCC governance travels through all environments — it is not a development-only tool.
Hybrid cognition in practice — Bayes as the human-in-the-loop layer
Why this matters beyond the horse race
Discovery 20 in this study's governance log is the broadest finding of the whole exercise: effective decision support in complex environments rarely comes from one model, however accurate — it comes from structuring multiple evidence streams, including human judgment, so they can be combined and audited together. The mechanism that makes that combination rigorous, rather than a gut-feel override sitting on top of a model's number, is Bayesian Weight of Evidence (WoE).
Every signal in this study — whether produced by a model or stated by a person — is expressed in the same unit: decibans (dB), a log-odds scale measured relative to a fixed population baseline (10% prior departure rate). Because WoE is additive in log-odds space, a machine's forecast and a salesperson's field judgment can be summed directly into one auditable posterior, instead of one silently overriding the other. That additivity is the entire trick — it's what lets "hybrid cognition" mean something more precise than a person ignoring the model when they feel like it.
None of this matters if it doesn't change what someone does before the customer is actually gone. The entire economic argument in this study — Economic Detection Potential (EDP) weighed against False Positive Penalty (FPP) — rests on one assumption: the value an organization can still recover from a departing customer declines the longer detection takes, because the intervention window narrows as the relationship deteriorates. A forecast that arrives after the customer has effectively decided to leave has academic interest and close to zero economic value. The entire point of looking ahead — statistically, and now with the human layer added — is to surface the signal early enough that there is still a real decision to make and a real action that can change the outcome. Hybrid cognition exists to shorten that detection lag without manufacturing false confidence: the human layer fills exactly the gap where the model has a real but not-yet-confirmable signal, instead of forcing a choice between acting on nothing and waiting for statistical certainty that may arrive too late to act on.
Machine layer — statistical forecast
Sequential Fusion v2→Produces a statistical forecast probability for each customer, each week, from that customer's own behavioral history
confidence_score→Converted to decibans: WoE = 10 × log₁₀( odds(p) / prior odds ), prior = 10% population base rate
No human input→This number is fixed, fitted, and identical for every reader — it carries no field knowledge
Human layer — salesperson estimate
Calibrated evidence→Account tier, revenue concentration — Bayes factors derived empirically from the customer registry, not opinion
Judgment-only evidence→Call sentiment, competitor mention, champion turnover, budget signal — entered as dB, logged with a required typed rationale
No calibration claimed→These four nodes default to 0 dB until enough logged outcomes exist to calibrate them the same way tier and revenue already were
The combination, in one line
Posterior dB = Statistical Forecast dB + Salesperson Estimate dB → Posterior Probability. The human does not replace the model's number. The two are added, in the open, with every human contribution logged — customer, forecast week, evidence node, dB value, salesperson, and rationale. The model stays auditable. The human's judgment becomes auditable too, for the first time, instead of disappearing into a phone call.
How this works in practice
1 — The model flags, with a caveat attached
Illustrative example: a newly onboarded account is flagged by the surveillance layer at a moderate forecast probability in its first few weeks live. The trajectory-confirmation layer, which needs a longer observation window to confirm or dismiss the alarm, hasn't accumulated enough history yet to weigh in. The honest machine output at this point is "elevated, not yet confirmable" — not silence, and not a false confidence. Every week spent waiting for that confirmation is a week the intervention window is quietly narrowing.
2 — The salesperson adds what the model can't see
The account team already knows the customer is a Tier A account with concentrated revenue exposure (both empirically calibrated, both apply automatically), and that the customer's champion left the company this week — confirmed on a call, not a rumor. That gets entered as a judgment-only evidence node, at a strength the salesperson chooses, with a typed rationale required before it can be logged. This is the evidence that closes the gap the model's window can't close yet.
3 — The combined call, made while there's still time to act
The machine's moderate signal plus the calibrated registry facts plus the logged champion-turnover judgment sum to a posterior well above the model's number alone — enough to justify a retention call this week instead of waiting the additional weeks the trajectory layer would need to confirm on its own. By the time a model-only signal would have cleared its confirmation bar, much of the recoverable value may already be gone; the combined posterior is what makes acting now defensible instead of premature. Every input that produced that number is timestamped and on record, so the call can be reviewed later against what actually happened to that account.
Why log instead of just deciding
Four of the six evidence nodes in the tool — call sentiment, competitor mention, champion turnover, and budget cycle signal — are deliberately uncalibrated today. There isn't yet a dataset connecting "champion turnover, logged at this strength" to actual won/lost outcomes, so each is salesperson judgment only, entered as a dB value with a required typed rationale. Every logged override is the seed data for closing that gap. The other two nodes, account tier and revenue concentration, already went through this process: their dB values (+3.2 and +1.0 respectively) are empirical Bayes factors computed directly from the customer registry — how much more often Tier-A or revenue-concentrated accounts actually appear among departures versus stable accounts — not estimates. Once enough overrides accumulate against known outcomes, the four judgment-only nodes can be calibrated the same way: as empirical Bayes factors, not standing opinion. The governance discipline that audited the model — every number traces to a source, nothing is asserted without evidence — is the same discipline applied to the human contribution. That symmetry is the actual point of operationalizing hybrid cognition this way, rather than leaving human override as an unstructured, unrecorded judgment call layered on top of a model nobody can check.
Full Working Paper — Simulation, Governance & Evidence Fusion
Methodology · twenty governance discoveries · mathematical appendix · case studies · Appendix D results
↓ Download PDF