Myth Any before-and-after comparison counts as an A/B test.
What it means A controlled experiment that randomly splits users between versions at the same time and compares a goal metric, so the change is the only systematic difference.
Common mistakes
Stopping as soon as a result looks significant
Judging the test on micro-conversions such as clicks instead of the business outcome
Changing or ramping variants mid-test
Why it matters
Randomization is what lets you say the change caused the result. Without it, seasonality, campaigns and news get mixed into your number.
Myth Activation means signing up or creating an account.
What it means The behaviour threshold that shows a user is getting real value, such as Dropbox's one file stored in the first week. Users who cross it are likely to keep coming back.
Common mistakes
Picking a milestone that is easy to hit rather than one that predicts retention
Never checking that activated users actually stay longer
Why it matters
Activation sits before revenue. If the definition is just signup, onboarding work gets optimized for the wrong finish line.
What it means In GA4, the Users metric in standard reports is almost always Active users: people with an engaged session or specific engagement signals. Total users counts anyone who triggered an event.
Common mistakes
Comparing GA4 Active users with another tool's total visitors
Assuming New users plus Returning users should add up to the total
Why it matters
Mixing up Active and Total users makes audiences look smaller or larger than they are, and the numbers will never reconcile across tools.
Myth The aha moment is a feeling you can only learn about from interviews.
What it means The point where a user first experiences the product's core value. Product teams usually find it by looking for early actions that separate retained users from churned ones.
Common mistakes
Choosing an aha action because it correlates with retention, then assuming forcing it will cause retention
Why it matters
It focuses onboarding on one valuable action. Treat the correlation as a hypothesis and test it before rebuilding onboarding around it.
Myth Anonymous users are separate people from logged-in users.
What it means A visitor tracked under a device-generated ID before you know who they are. Calling identify links that anonymous history to a known user ID so it appears as one person.
Common mistakes
Never calling identify after login
Forgetting to reset on logout so shared devices merge people
Giving several people the same ID
Why it matters
Poor identity handling either splits one customer into many or merges many into one. Both distort conversion and retention.
Myth The attribution model shows which channel caused the sale.
What it means A rule or algorithm that assigns credit for a key event to the touchpoints before it. GA4's default is data-driven attribution, and the model you choose changes the credit each channel gets.
Common mistakes
Treating last-click or last-non-direct reports as the whole customer journey
Looking at paid channels only
Forgetting offline touchpoints
Why it matters
Attribution divides up credit. It does not prove a channel caused a sale, and budgets built only on it tend to overfund the last touch.
Myth Autocapture means you never need a tracking plan.
What it means Automatic collection of predefined events such as page views, clicks, form interactions, rage clicks and dead clicks, with very little setup.
Common mistakes
Relying on raw clicks and page views for key business events
Skipping the step of naming and curating captured events
Why it matters
Autocapture is great for discovery. Revenue, signups and other critical events still need deliberate, named tracking that won't break when the page changes.
Myth Bayesian tests let you stop whenever you like with no risk.
What it means An approach that starts with a prior belief and updates it as data arrives, reporting results as probabilities, such as the chance that B beats A.
Common mistakes
Assuming Bayesian methods make results valid sooner, or protect against small samples and weak design
Why it matters
The output is easier to explain, but bad design is still bad design. Priors, sample size and stopping rules still matter.
Myth BigQuery will match the GA4 interface exactly.
What it means A raw, unsampled export of GA4 event data into BigQuery for SQL analysis. Standard properties have a daily export limit of 1 million events.
Common mistakes
Expecting historical backfill
Assuming interface concepts such as sessions and attribution exist ready-made in the raw tables
Why it matters
It is the route to unsampled, ownable data, but you rebuild sessions, users and attribution yourself, so the totals won't exactly match the interface.
Myth Analytics tools filter all bots automatically.
What it means Automated traffic from bots, crawlers and AI agents. PostHog, for example, classifies it by user agent and lets you block it or exclude it at query time.
Common mistakes
Trusting built-in bot filtering as complete
Ignoring the growing share of AI crawlers and agents in web and experiment data
Why it matters
Bots inflate sessions, wreck conversion rates and can even skew A/B tests. Bot handling is a data-quality decision, not a default.
Myth GA4 bounce rate means the same thing as it did in Universal Analytics.
What it means In GA4, the percentage of sessions that were not engaged. An engaged session lasts over 10 seconds, has a key event, or has 2 or more page or screen views.
Common mistakes
Applying the old single-page definition
Expecting bounce rate in default GA4 reports
Reporting bounce and engagement rate side by side when one is the inverse of the other
Why it matters
A UA-era benchmark doesn't transfer. A 'drop' after migration may just be a definition change.
Myth (other) in a GA4 report is a bug or lost data.
What it means The number of unique values in a dimension. When high-cardinality data exceeds GA4's row limits, the excess is grouped into an '(other)' row.
Common mistakes
Sending unique values such as IDs or full URLs with query strings into custom dimensions
Confusing cardinality with thresholding
Why it matters
High-cardinality dimensions quietly collapse your detail. Design dimensions with this in mind, or analyze in BigQuery.
Myth A 95% interval means a 95% chance the true effect is inside this particular range.
What it means The range of effect sizes consistent with the data at a chosen confidence level. A 95% interval procedure captures the true effect about 95 times out of 100 experiments.
Common mistakes
Reading only whether the interval crosses zero and ignoring its width
Using it on very skewed data without checking assumptions
Why it matters
The interval tells you how big the effect could be. A significant but wide interval can still include effects too small to matter.
Myth Consent mode is a banner, or it makes tracking compliant on its own.
What it means A Google framework that changes how tags behave based on users' consent choices. Basic mode blocks tags until consent; advanced mode loads tags and sends cookieless pings while consent is denied.
Common mistakes
Setting default consent to granted
Tags firing before consent is known
CMP and custom templates sending conflicting updates
Expecting a consent update to fire tags by itself
Why it matters
Misconfigured consent either breaks measurement or collects data you shouldn't. It is plumbing that needs testing, not a checkbox.
What it means Conversions divided by an eligible base. In Google Ads that base is ad interactions; in GA4 it can be per session or per user (key event rate).
Common mistakes
Comparing per-session and per-user rates
Comparing GA4 key event rate with Google Ads conversion rate without checking definitions
Why it matters
The denominator is the definition. Agree on it before you compare channels, tools or periods.
Myth A funnel counts a conversion whenever it eventually happens.
What it means The time allowed for a user to complete the funnel or conversion after the first step or ad interaction. Mixpanel funnels and Google Ads both let you set it.
Common mistakes
Leaving default windows in place for long purchase cycles
Comparing funnels built with different windows
Why it matters
Too short a window undercounts considered purchases. Too long a window credits stale touches.
Myth If two metrics move together, one drives the other.
What it means Correlation means two variables move together. Causation means changing one actually changes the other, which usually takes an experiment to show.
Common mistakes
Treating users who use a feature and retain better as proof that the feature causes retention
Extrapolating short-term test results
Why it matters
Most expensive analytics mistakes start here. Feature-usage correlations often reflect who the users are, not what the feature does.
Myth If both sites have the same tag, users are tracked across them automatically.
What it means Configuration that lets GA4 treat a journey across two related domains, such as a store and a separate cart, as one user and one session.
Common mistakes
Missing domains in the configuration
Redirects dropping the _gl linker parameter
Forgetting to exclude the payment provider as a referral
Why it matters
Without it, one buyer becomes two users and checkout conversions get credited to your own domain as a referral.
Myth CUPED is a different statistical test that makes results more significant.
What it means A variance-reduction method that adjusts each user's metric using the same metric from before exposure. Statsig Cloud uses a 7-day pre-exposure window.
Common mistakes
Applying it to metrics with no pre-experiment history
Expecting it to fix SRM or bias
Why it matters
Lower variance means smaller tests or faster answers. It works best when past behaviour predicts future behaviour.
What it means An estimate of the total value a customer generates over their relationship with you, usually built from average order value, purchase frequency and lifespan.
Common mistakes
Using revenue instead of margin
Calculating one blended LTV across very different segments
Ignoring the time period assumed
Why it matters
LTV caps what you can safely spend to acquire a customer. Revenue-based or blended LTV leads to overspending.
What it means Limits GA4 applies to hide data in reports when user counts are low enough that individuals might be identified, often linked to Google Signals or demographics.
Common mistakes
Thinking thresholds can be adjusted
Assuming disabling Google Signals fixes historical data
Using tiny date ranges on low-traffic sites
Why it matters
A thresholded report can use 100% of data and still be incomplete. Low-traffic sites are hit most.
What it means GA4's default model, which uses machine learning to compare converting and non-converting paths and assigns credit specific to each advertiser and key event.
Common mistakes
Treating DDA credit as causal
Assuming it sees offline, cross-device or consent-blocked touchpoints
Why it matters
It is better than last click at dividing credit, but it still only sees what is tracked. It isn't an incrementality test.
Myth You can edit GA4's default channel group to fix your traffic buckets.
What it means The rule-based set of channels Google maintains to classify traffic sources. It can't be edited, but you can create custom channel groups.
Common mistakes
Using custom UTM mediums such as 'em' or 'pdf' that don't match the channel rules, then wondering why traffic is Unassigned
Why it matters
Channel reports are only as clean as your UTMs. Learn the rules, or build a custom group.
Myth Direct traffic means people typed in your URL or used a bookmark.
What it means Sessions where GA4 has no referrer or campaign information. That includes typed URLs and bookmarks, but also untagged links, PDFs, redirects and privacy tools.
Common mistakes
Untagged email, app and social links
Redirects that strip referrers or UTMs
Missing tags on landing pages
Malformed parameters
Why it matters
Direct is often a bin for tracking gaps. A big Direct share usually means misattributed marketing, not brand strength.
Myth Turning on ecommerce in GA4 gives you revenue reports automatically.
What it means A set of recommended events, such as view_item, add_to_cart, begin_checkout and purchase, with an items array that you implement to measure shopping behaviour.
Common mistakes
Missing the currency parameter or sending symbols instead of ISO codes
Duplicate events from creating an event in GA4 that GTM already sends
Item parameters sent outside the items array
Why it matters
Revenue in analytics is only as good as the implementation. Reconcile it with the order system regularly.
What it means Agreed rules for naming events and properties, such as lowercase verb_noun. In GA4, names are case-sensitive and must follow character rules.
Common mistakes
Inconsistent casing and tenses
Vague names
The same action named differently by different teams
Names that are too broad or too specific
Why it matters
Click_Button and button_click become two events forever. Naming is cheap to fix before launch and expensive after.
Myth Everyone assigned to an experiment should be counted in the analysis.
What it means The moment a user actually encounters the part of the product where an experiment runs. Only metric events after first exposure should count.
Common mistakes
Analyzing all assigned users, including those who never reached the changed screen
Starting analysis from the wrong point or with an incorrect trigger condition, which Lukas Vermeer lists as a cause of SRM
Why it matters
Counting unexposed users dilutes real effects toward zero and wastes sample size.
Myth A significant result is almost certainly real.
What it means A Type I error: concluding there is an effect when there isn't. At a 5% significance level, an A/A test still declares a 'winner' about 5% of the time.
Common mistakes
Testing many variants, metrics or segments without correction
Stopping when significance appears
Why it matters
False positives ship changes that do nothing and teach the team the wrong lessons.
What it means A remote switch in code that controls who sees a feature, without a redeploy. Flags separate deployment from release; on their own they don't measure impact.
Common mistakes
Leaving stale flags in code
Vague flag names
Assuming a flag always returns its expected value
Mixing flag checks into business logic
Why it matters
Flags enable safe rollouts. Experiments need randomization plus analysis on top.
What it means A sequence of steps toward a goal, showing how many users convert or drop off at each step. Options include order rules, conversion windows and totals vs uniques.
Common mistakes
Building funnels with steps in an order users don't follow
Ignoring the conversion window
Reporting incomplete funnel metrics that hide where users leak out
Why it matters
Funnel settings change the answer. Two analysts with different settings will report different drop-offs from the same data.
What it means Analysis at the level of a group such as a company or account rather than an individual user. Mixpanel and PostHog support it as a feature.
Common mistakes
Reporting user-level retention for a product bought by accounts
Not sending a company or account ID with events
Why it matters
In B2B, the account renews, not the user. User metrics can hide an account that is quietly going dark.
What it means A business-critical metric monitored for harm during a test. PostHog notes that with 10 guardrails at 0.05, an A/A test triggers a false alert about 40% of the time.
Common mistakes
Adding too many guardrails without accounting for false alarms
Ignoring guardrail regressions when the primary metric wins
Why it matters
Guardrails stop a 'winning' test from quietly hurting revenue or experience. Choose a few that matter.
What it means When a treatment's effect differs meaningfully across segments, such as new vs existing users. Statsig flags segments whose effect differs sharply from everyone else.
Common mistakes
Assuming movement is homogeneous
Slicing many segments without correction and calling the best one a finding
Why it matters
An average can hide a win in one segment and a loss in another. Check pre-planned segments, with corrections.
What it means A set of users kept out of shipped changes or experiments over the long term to measure cumulative impact. Amplitude recommends 1% to 10%.
Common mistakes
Changing the holdout percentage or experiments mid-stream
Holdouts too small to measure anything
Why it matters
Holdouts check whether many small wins actually add up. Individual test results often overstate the long-term effect.
Myth The tool will automatically know that two devices are the same person.
What it means The process of linking multiple identifiers, such as anonymous and logged-in IDs, to one user. Mixpanel's Simplified ID Merge has been the default for new organizations since April 2024.
Common mistakes
Choosing a merge approach without understanding it
Calling identify or alias inconsistently across platforms
Why it matters
Merge rules are hard to undo. Get them right before you have years of data.
What it means When one experiment's effect depends on another running at the same time. Statsig notes strong interactions are rare and usually predictable.
Common mistakes
Refusing to run overlapping tests at all, or ignoring obvious conflicts between tests on the same page
Why it matters
Over-isolating tests slows learning; ignoring real conflicts misleads. Manage the obvious overlaps.
Myth Key events are something new, unrelated to conversions.
What it means An event marked as especially important to the business. Google renamed GA4 'conversions' to 'key events' in 2024; 'conversion' now refers to what Google Ads uses.
Common mistakes
Marking too many events as key events
Expecting marking to be retroactive
Assuming key events automatically import to Google Ads as conversions
Why it matters
If everything is a key event, nothing is. Too many also muddies key event rate and attribution reports.
Myth Lifecycle analysis is just counting new users.
What it means Grouping users by lifecycle stage, such as new, current, resurrected and dormant, to see how the user base is growing or shrinking over time.
Common mistakes
Not defining stages consistently
Not tracking movement between stages such as new, retained, resurrected and dormant
Why it matters
It shows whether growth comes from new users or from keeping existing ones. Those call for very different fixes.
What it means The mean is the arithmetic average; the median is the middle value. With skewed data, a few big values pull the mean well above what most users experience.
Common mistakes
Reporting averages on skewed data such as revenue per user
Ignoring the distribution behind an average
Why it matters
Averages hide the shape of your data. Show the median or distribution for spending and time metrics.
What it means A way to send events directly to GA4 servers over HTTP, for server-side or offline interactions. Google says it supplements, not replaces, automatic collection.
Common mistakes
Sending events without the correct client_id and session_id
Timestamps outside allowed limits
Sending hits that can't be tied to a real user or session
Why it matters
Badly built Measurement Protocol hits show up as Unassigned or orphan sessions and add noise.
Myth Checking more metrics gives a more complete picture without any cost.
What it means Running several statistical tests increases the chance of at least one false positive. Statsig gives a 40% chance with 10 tests at a 5% alpha.
Common mistakes
Searching many segments or KPIs for a winner without correction
Testing many variants at once
Why it matters
If you look in enough places, something will be 'significant'. Pre-register metrics or correct for them.
What it means New users visited for the first time in the date range; returning users had visited before. The same person can be both within one range.
Common mistakes
Adding the two and expecting the total
Forgetting that one person on two devices may count as two new users
Why it matters
These buckets overlap, and cookie-based counting inflates new users.
Myth Early test results show the long-term effect.
What it means A short-term change in behaviour caused by something new, which fades as the newness wears off. It is most likely in high-frequency products.
Common mistakes
Calling a test in the first few days
Never checking effect by days since exposure
Why it matters
Early spikes can make a neutral change look like a win, or a good change look like a loss.
Myth Failing to reject the null proves there is no effect.
What it means The default assumption in a test that there is no difference between variants. Tests measure how surprising the data would be if it were true.
Common mistakes
Reporting 'no difference' from underpowered tests
Why it matters
'Not significant' often means 'not enough data', not 'no effect'.
Myth The p-value is the probability the variant is better.
What it means The probability of seeing a difference at least this large if there were truly no effect. Statsig marks results significant when p is below 0.05 by default.
Common mistakes
Claiming success with a borderline p-value
Treating p < 0.05 as a guarantee
Checking it repeatedly during a test
Why it matters
It measures surprise under 'no effect', not the chance you are right or how big the win is.
Myth Path reports show the typical customer journey.
What it means Analysis of the sequences of events or pages users take before or after a chosen point. GA4 path explorations show the top nodes at each step.
Common mistakes
Spending hours on spaghetti path charts without a question
Reading top paths as representative
Why it matters
Path reports suggest hypotheses; they rarely answer business questions alone.
What it means Repeatedly checking a fixed-horizon test and acting on interim significance. Evan Miller shows this can push false-positive rates well above the nominal 5%.
Common mistakes
Stopping as soon as significance appears
Using significance to decide whether to keep running
Why it matters
Peeking turns 1-in-20 false winners into far more. Use a fixed horizon or sequential methods.
Myth Power users are just the people who log in most often.
What it means A histogram of how many days in a period, often L7 or L30, users perform a core action. It shows whether usage is concentrated or spread out.
Common mistakes
Defining power users only by frequency
Assuming one type of power user
Over-building for them
Why it matters
The curve's shape shows engagement health, but the core action must reflect real value, not logins.
Myth You can pick the primary metric after seeing the results.
What it means The single metric, decided before the test, that determines whether the hypothesis succeeded. Secondary and guardrail metrics add context.
Common mistakes
Choosing the metric that moved
Having several 'primary' metrics
Why it matters
Choosing after the fact turns an experiment into a fishing trip.