← Back to WorkProduct Analytics

B2B SaaS Onboarding Experiment

A project management company wanted to know whether a guided checklist would help new teams get to first value faster. It did overall, but the more useful finding was that the result was not consistent across customer segments.

RoleProduct Analyst
StackPython, pandas, SciPy, statsmodels, SQLite, Plotly
Timeline2026
CodeGitHub ↗
NotebookOpen ↗
2,000
Accounts in Test
+5%
More Teams Got Started
90
Day Experiment
$97K
Projected Best-Case Impact

Overview

The Problem

Too many new accounts were signing up, exploring briefly, and dropping off before they meaningfully adopted the product. The team wanted to know whether a more guided onboarding experience would improve that early activation window.

The Setup

The company ran a 90-day experiment across 2,000 new accounts. Half saw the existing blank-slate experience, and half saw a guided checklist. From there, I looked at activation, speed to first project, early usage behavior, and whether the effect held across different types of customers.

Key Findings

The Topline Was Positive

At the highest level, the experiment worked. Seven-day activation improved from 32.0% to 36.7%, which made the new onboarding experience look like a likely rollout candidate.

The Average Hid Two Different Outcomes

Once I broke the result down, the story changed. Smaller self-serve teams responded well to the checklist, while larger enterprise-oriented accounts tended to do worse with it.

Behavior Improved Faster Than Revenue

Accounts with the checklist got to their first project sooner and explored more in week one. But that improvement did not translate evenly into downstream business value, which made the segment differences more important than the headline lift alone.

The Best Decision Was a Targeted Rollout

The strongest recommendation was not to ship the checklist everywhere. It was to roll it out only where it clearly helped: smaller self-serve teams, while leaving higher-touch enterprise accounts on the existing flow.

Methodology

Setup
2K accounts
50/50 split
→
Check
Balance checks
Groups were comparable
→
Result
+4.7pp lift
32.0% to 36.7%
→
Detail
Segment split
SMB up, enterprise down
→
Decision
Targeted rollout
Self-serve under 20

Analysis Focus

1Making Sure the Test Was Worth Trusting
What I didBefore interpreting the outcome, I checked whether the experiment itself was trustworthy. I verified that the two groups were comparable and that nothing in the data distribution suggested the result was being driven by an uneven split.
Why it matteredIt made the observed lift worth taking seriously instead of dismissing it as noise or a flawed test.
2Looking Past the Average
What I didI broke the results down by company size, plan type, and whether the account was self-serve or sales-assisted. That is where the experiment became useful, because the same onboarding flow was clearly helping one group while getting in the way of another.
Why it matteredThat changed the recommendation from a broad rollout to a more selective one.
3Turning the Analysis Into a Real Decision
What I didI compared three rollout paths: ship it to everyone, ship it only to smaller self-serve accounts, or leave things as they were. The goal was to turn the experiment into a decision, not just a result.
Why it matteredThe targeted rollout projected roughly $97K more in first-year impact than shipping it broadly, and the strongest answer was also the most practical one.

Recommendation

Show the walkthrough only to small teams under 20 people. Leave larger companies and sales-led accounts on the old experience. That way the company keeps the upside where the checklist clearly helped, without forcing it onto the customers it slowed down.

Ship to
Self-Serve <20
Revisit Later
Mid-Market
Leave Alone
Enterprise