How would you define and measure the success of a new feature — say, a 'Stories' feed added to a social app?

How to think about it

Direct answer

I would define success as a measurable improvement in the outcome Stories is supposed to create, not simply as “people tapped Stories.” I would choose one primary metric tied to the product goal, compare treatment and control users in a randomized experiment, and set guardrails so a local win does not damage the rest of the app.

Why usage is not the same as success

First, I would clarify the goal. Is Stories meant to increase retention, daily engagement, creator activity, ad revenue, or acquisition? The same feature can succeed on one objective and fail on another.

Suppose the goal is to improve retention by giving users a reason to return every day. Then the primary metric might be day-7 retention: the percentage of eligible users who are active again seven days after their first eligible day. If the goal is creator supply, I might instead measure the percentage of weekly active creators who publish at least one Story. If the goal is monetisation, I would look at incremental revenue per eligible user, not story views.

A view is an output. Retention or revenue is an outcome. Outputs are useful because they tell us whether users noticed and used the feature. They do not prove that the feature created lasting value.

A north-star metric is the single top-line measure that best represents the intended value of the feature. It should be specific enough to make a decision. “Engagement” is not a metric. “Meaningful sessions per eligible daily active user” can be one, provided the team defines what counts as meaningful.

I would also define the denominator. For example:

• Story-view rate: unique eligible daily active users who viewed at least one Story divided by eligible daily active users.
• Stories per user: total completed Story views divided by eligible daily active users.
• Day-7 retention: eligible users active on day 7 divided by eligible users in the starting cohort.

Those definitions matter. If autoplay creates a view as soon as a tile renders, the dashboard may report healthy usage even when nobody deliberately watches anything.

The measurement contract

Before launch, I would write down four things:

1. The product hypothesis.
2. One primary metric.
3. Guardrails and their thresholds.
4. The experiment and observation window.

A useful hypothesis might be:

Giving active users a lightweight way to share temporary updates will increase day-7 retention by at least 1 percentage point, without reducing core-feed engagement or increasing notification opt-outs.

A guardrail metric is a health measure that can veto an otherwise positive result. For Stories, I would use guardrails such as these:

Metric	What it protects
Core-feed meaningful interactions per user	Detects Stories stealing attention from the main feed
Day-7 retention for users who never use Stories	Detects harm to people who do not want the feature
Notification opt-out and mute rates	Detects over-promotion or notification fatigue
Report, hide, and unfollow rates	Detects content-quality or safety problems
Crash-free sessions and app launch success	Detects reliability regressions

I would predeclare thresholds rather than invent them after seeing the result. For example, the team might decide that a drop of more than 1 percent in core-feed interactions or an increase of more than 0.3 percentage points in notification opt-outs requires investigation. Those are product decisions, not universal statistical laws. They should reflect the value of the expected retention gain and the cost of the harm.

I would also set a minimum meaningful effect. That is the smallest improvement worth the engineering, infrastructure, moderation, and opportunity costs. A statistically detectable increase of 0.05 percentage points may be real but not worth shipping.

A concrete Stories experiment

Assume the product team wants a stronger daily habit. We assign 200,000 eligible users at random:

• 100,000 users receive Stories, the treatment group.
• 100,000 users do not receive Stories, the control group.

Random assignment makes the groups comparable on average. That is what lets us attribute a difference in outcomes to the feature rather than to the kinds of people who chose to use it.

After a two-week post-ramp period, suppose the results look like this:

Metric	Treatment	Control	Difference
Story-view rate	34.0%	0.0%	Expected by design
Day-7 retention	28.4%	27.9%	Plus 0.5 percentage points
Core-feed interactions per user	5.0	5.2	Down 3.8%
Notification opt-out rate	3.1%	2.7%	Up 0.4 percentage points
Crash-free sessions	99.30%	99.31%	Essentially unchanged

Stories clearly has reach: 34 percent of treatment users viewed at least one Story. But the primary outcome improved by only 0.5 percentage points, below the predeclared 1-point target. Meanwhile, core-feed interactions fell from 5.2 to 5.0 per user, and notification opt-outs rose by 0.4 points.

I would not call that a successful launch. I would investigate whether Stories is cannibalising the main feed, whether notifications are doing too much of the work, and whether the placement is turning the home screen into a crowded shop window. I would probably iterate and rerun the experiment rather than ship to everyone.

I would report both absolute and relative changes. The retention increase is 0.5 percentage points, or about 1.8 percent relative to the control rate of 27.9 percent. Those are different statements. Percentage points are usually clearer for rates.

I would also report uncertainty, usually with a confidence interval or an equivalent experiment estimate. A large sample can make a tiny effect look statistically distinguishable from zero. That still does not make it commercially or product-wise important.

The senior-level nuance

The primary metric depends on the job Stories is meant to do.

If Stories is an engagement surface, story views may be a reasonable leading metric, meaning an early signal that users are trying the feature. But I would still check total meaningful app engagement. A user watching ten Stories instead of reading ten useful feed posts may leave the company with the same number of taps and a worse product.

If Stories is a retention feature, I would prefer a retention outcome as the primary metric and treat adoption, views per user, replies, and shares as diagnostic metrics. Diagnostics explain the result; they should not quietly replace the agreed success measure.

If Stories relies on creators, I would measure both sides of the marketplace. Viewer adoption can look excellent while creator posting collapses because distribution is poor. I might track active creators posting weekly, creator retention, and the number of viewers per published Story as secondary measures.

The observation window must match the claim. A launch-day spike mostly measures novelty. A two-week window can show whether usage settles, but it cannot establish 30-day retention for users enrolled near the end of the test. For a day-30 retention claim, I would wait for complete day-30 outcomes or use a clearly specified sequential analysis plan. I would also cover full weekly cycles because a Friday launch can otherwise be compared with an unusually quiet Monday.

I would analyse important segments, such as new versus existing users, heavy versus light users, mobile platforms, and regions. But I would preselect those segments. If I search through 40 cuts after the experiment, one will often look exciting by chance.

Stories can also create network interference. A treatment user may publish a Story that a control user’s friend can see, which makes the control group less pure and usually pushes the measured treatment effect toward zero. If that spillover is substantial, I would consider randomising at a network, community, or other appropriate cluster level, accepting that this usually requires a larger sample.

Common trap. Do not compare people who watched Stories with people who did not and call the difference causal. People who choose to watch may already be more active, more social, or more likely to return. The correct default is to analyse users according to their assignment, including treatment users who never opened Stories.

A failure mode you would see in practice

The first symptom might be a dashboard showing story views up 60 percent while total app engagement is flat, core-feed interactions are down 5 percent, and notification opt-outs are climbing. That pattern often means the feature is being forced through autoplay or aggressive notifications rather than creating genuine value.

I would check event instrumentation before drawing a product conclusion. A view event might fire when a card renders, fire twice after a client retry, or use a different denominator after launch. A sudden step change in views immediately after a release is a measurement warning, not a user-behaviour insight. I would validate raw events, unique-user counts, assignment logs, and the exact exposure definition.

What they’ll ask next

“Why not use daily active users as the primary metric?”

Daily active users, or DAU, counts distinct users active on a given day. It is easy to understand, but it is broad and noisy. A notification can increase DAU for one day without improving retention or satisfaction. I would use DAU only if the product goal is genuinely to increase daily activity, and I would pair it with retention and quality guardrails.

“What if Story adoption rises but retention does not?”

Then adoption is a leading signal, not proof of success. I would check whether users are returning for Stories or merely sampling it once, whether the content is useful, and whether Stories is replacing another valuable surface. I would also inspect treatment effects by user segment. High adoption among heavy users can conceal no benefit for new or casual users.

“How would you decide whether to ship if the primary metric improves but a guardrail gets worse?”

I would apply the thresholds agreed before the test. A small, reversible movement in a soft guardrail might justify a staged rollout and more monitoring. A meaningful increase in crashes, reports, or notification opt-outs is a stop signal even if retention improves. The answer depends on severity, reversibility, confidence in the estimate, and the value of the primary gain.

Say this in the interview

“I would define Stories’ success from its product goal, choose one outcome metric and predeclared guardrails, and use a randomized test long enough to separate durable value from novelty while checking for cannibalisation and harm.”