Funnel Diagnosis and Experiment Backlogs
A funnel shows progression through defined events under specific counting rules. It can locate a drop in recorded progression, but it does not explain the cause by itself. Check measurement first, gather relevant user evidence, and turn the combined evidence into a falsifiable improvement hypothesis.
An experiment backlog should preserve the problem, proposed mechanism, expected outcome, guardrails, evidence and next decision. Ranking ideas by a guessed score is less useful than knowing which assumption the next test will resolve. Start with the largest credible uncertainty, not automatically the most dramatic percentage drop.
TL;DR
- A funnel shows progression through defined events under specific counting rules; it locates drops but doesn't explain them.
- Check measurement first, then gather user evidence, and turn both into a falsifiable hypothesis.
- Know whether a funnel is open or closed, and whether it counts users or events, before comparing numbers.
- Record each backlog item's problem, mechanism, expected outcome, guardrails, evidence, and next decision.
- Prioritize the largest credible uncertainty, not automatically the most dramatic drop.
Quick Example
This Python example counts an explicitly defined closed, ordered toy funnel for four fictional users. Each user advances through view, start and done in that order; repeated earlier events do not add another user to a step. There is no time-window rule in this example.
The counts are three, two and one. A path beginning at start does not enter the closed funnel, and a user who skips start does not complete the last step. These are results of this own algorithm, not a reproduction of every analytics product's semantics. No real event collection, identity stitching, attribution, interview or experiment occurred.
Core Concepts
Counting rules precede interpretation
Google Analytics' funnel exploration documentation distinguishes open funnels, where users may enter at different steps, from closed funnels that require entry at the first step. Its examples also show how ordering affects progression. That is a useful reminder to read the actual tool definition before comparing reports. Funnel exploration
Define the unit, entry condition, event order, allowed gaps, conversion window and repetition rules. State whether the report counts users, sessions, attempts or events. A change from one unit to another can alter apparent conversion without any change in experience.
Verify time and identity boundaries. A delayed event may fall outside a window; a user who changes devices may appear as two identities; a restarted attempt may be counted differently from a continuous one. Do not diagnose user intent from a reporting artifact until the event path has been checked.
A drop is a symptom with competing explanations
Possible explanations include a real usability problem, a deliberate user choice, an unavailable dependency, an event that stopped firing, or a denominator containing people who never intended to continue. Form alternatives that can be investigated rather than naming every drop friction.
This is an editorial diagnosis matrix. It does not establish the cause of a real drop or authorize additional personal-data collection.
User evidence informs a mechanism
Use appropriate, authorized research to understand what people were trying to accomplish and where the proposed mechanism fails. Interviews, usability sessions, support themes and accessible interaction checks can offer context that event counts lack. Each has its own sample and observation limits.
Do not treat one vivid quote as proof that the same cause explains every recorded exit. Preserve who was represented, what was observed and which inference remains uncertain. Likewise, a statistical pattern does not automatically disprove a usability problem experienced by a smaller group.
A backlog item should be testable
Write a hypothesis connecting an observed problem to a proposed change and a measurable outcome. Specify which result would support the mechanism and which would cause the team to reject or revise it. Include guardrails for errors, task quality, accessibility or other consequences that matter to the product.
Separate a measurement repair from a product experiment. Fixing a broken event can change the reported conversion rate without changing user behavior. A necessary accessibility correction may also deserve implementation on its own merits rather than waiting for a conversion experiment to justify it. Choose the right decision process for the issue.
Best Practices
Keep a funnel definition and a small set of representative paths under review. Include entry at later steps, repeated events, skipped steps, delayed completion and identity transitions as appropriate to the real tool. Compare actual instrumentation with the intended definitions before interpreting a major change.
Attach evidence and confidence to backlog items. Record the observation, affected population, competing explanations, proposed mechanism, test method, primary outcome, guardrails and decision owner. A prioritization score should summarize these inputs, not replace them with a precise-looking number whose assumptions nobody can inspect.
Choose experiments that resolve an important uncertainty and are feasible under actual traffic and constraints. Define assignment, analysis and stopping rules before examining the outcome. Use an independent or appropriately designed assessment rather than selecting whichever metric looks favorable afterward. No sample-size or significance result is implied by the four-user example.
Handle research and analytics with the relevant privacy process. Minimize collected data, restrict access and retain only what the approved purpose requires. Session recording or surveys are not automatically necessary for every funnel question. Use the least intrusive evidence that can answer the unresolved issue.
Review outcomes against the original mechanism. A measured improvement can justify further investigation without proving the same effect will persist across populations or future releases. A null or adverse result is useful when it rules out an assumption and helps choose the next step. Keep that learning in the backlog instead of recording only winning variants.
Reverse an initial priority when the supposed drop is a tracking failure, the affected population is different from the target, or user evidence challenges the proposed explanation. The purpose of diagnosis is to change decisions when evidence changes, not to defend the first appealing redesign.
Common Mistakes
Comparing open and closed funnels
Counting events as users
Prioritizing the biggest drop by reflex
Calling a shipped change a validated experiment
FAQ
Does a funnel tell us why people leave?
It shows recorded progression under its definitions. Investigate competing causes using measurement checks and relevant user evidence before claiming an explanation.
Does this example match GA4 exactly?
No. It is a simplified closed, ordered funnel with no time window or identity stitching. Check your analytics product's documented funnel semantics before comparing its counts with this model.
What makes a useful next experiment?
A clear unresolved question, evidence-backed mechanism, defined population and outcome, appropriate guardrails and a decision rule. Its value comes from what the result can resolve, including a result that challenges the proposed change.