AI Retention Reports: Compare Customers at the Same Stage

Imagine an AI-written shop report declaring that retention fell from 50% to 40%. The arithmetic checks out, but the comparison joins one customer group’s second follow-up month to another group’s first. The numbers describe different stages of the relationship. A convincing sentence has hidden the mismatch.

For creators and business owners, the useful task is to make the comparison inspectable. Establish who entered each group, what counts as returning, and how much time each group had to do so. The fictional print-shop example below shows how the conclusion changes when those decisions stay visible.

Define the starting group and the return

A cohort is a group linked by a shared starting condition. Shopify’s customer-cohort documentation, for example, describes its default grouping by the date customers placed their first order. Following that group through later periods lets the reader distinguish its subsequent purchases from purchases by newly arriving customers.

For this exercise, group buyers by their first-purchase calendar month. Call that starting month Month 0, the next calendar month Month 1, and the following calendar month Month 2. A return means placing at least one further order in the specified follow-up month. Count each returning customer once in that month, regardless of the number of orders.

Those are the rules for this worked example. State equivalent rules when commissioning an AI summary, rather than leaving a broad heading such as “retention” to carry the definition.

Work through two fictional cohorts

An imagined independent print shop closes its example dataset on 30 September 2026. There are 12 customers whose first order was in June and 10 whose first order was in July. The following counts are invented for illustration, not results from a real shop or an AI trial:

• June cohort: 3 of its 12 customers order again in July, giving Month 1 return of 25%. In August, 6 of the original 12 order, giving Month 2 return of 50%.
• July cohort: 4 of its 10 customers order again in August, giving Month 1 return of 40%. In September, 2 of the original 10 order, giving Month 2 return of 20%.

The denominator remains the original cohort size: 12 for June and 10 for July. Replacing it with the number who returned in the previous month would answer a different question. Keep the customer counts beside the percentages so this choice can be checked.

Each of these four results covers a calendar month that finished by the stated cutoff. That makes both Month 1 comparisons and both Month 2 comparisons available.

Compare the same stage before writing the headline

At Month 1, the July cohort’s 40% is higher than the June cohort’s 25%, a difference of 15 percentage points. At Month 2, July’s 20% is lower than June’s 50%, a difference of 30 percentage points. The newer cohort has a higher first follow-up result and a lower second follow-up result.

Comparing June’s Month 2 value of 50% with July’s Month 1 value of 40% loses that pattern. Ask the assistant to name the cohort, the actual observation month and the stage after acquisition beside every figure it compares. Then check those labels against the source.

The dates matter even when the calendar month matches. August is Month 2 for June buyers and Month 1 for July buyers. A report organised only around “August retention” still needs to explain which question it is answering.

Check whether returning means a period or a longer window

In this example, June’s rise from 25% to 50% describes orders during two separate months. It does not establish uninterrupted monthly purchasing. Because only three June customers ordered in July, at least three of the six August returners had not ordered in July.

Amplitude’s retention documentation illustrates another distinction: Return On counts users with a qualifying return event in a specified interval, while Return On or After includes users who return in that interval or later. These definitions answer different questions. A figure measuring activity during one month cannot be silently relabelled as everyone who returned at any later point.

The clock definition also matters. Amplitude distinguishes rolling time windows from strict calendar dates. Our shop example uses calendar months; Month 1 does not mean exactly 30 days after every individual’s first purchase. Record the time convention before comparing results from different reports.

Keep unfinished intervals visibly unfinished

Suppose a September cohort is added to the same report. At the 30 September cutoff, its first follow-up month, October, has not happened. Its Month 1 return is not yet observable. Entering zero would turn an unavailable future result into an apparent absence of returning customers.

Amplitude’s FAQ explains that it marks incomplete intervals with an asterisk and excludes incomplete values from relevant overall calculations. That is a documented application-specific treatment, not a promise about every reporting tool. Check how your own source identifies unfinished periods, and preserve that status in the AI summary.

Give the assistant a comparison brief

The following review structure is an editorial proposal for an AI-assisted reporting task. It is not a measured intervention or a claim that a particular analytics integration was tested:

• Starting condition: identify the event that admits a customer to a cohort and the dates used to group them.
• Return rule: define the qualifying action, whether customers or events are counted, and the observation window.
• Comparable stage: compare Month 1 with Month 1, then Month 2 with Month 2, retaining counts and percentages.
• Data cutoff: state when observation stops and flag intervals that have not finished.

Ask for a description of the observed pattern before recommendations. Verify the source counts and reproduce a small calculation yourself. In the July cohort of ten, one customer represents ten percentage points, so the count is useful context for interpreting a change.

Connect the measure to the business question

A repeat-purchase measure is a particular behavioural measure. It does not establish why a customer returned, how satisfied they felt or whether a campaign caused the purchase. The fictional counts contain no evidence to answer those questions.

A membership business also needs an appropriate definition of returning. MaryChuks’ guide to evaluating paid-community platforms considers active participation, payment status and outcome evidence alongside retention. Choose the behaviour relevant to the service before borrowing a shop’s measure.

Use the report to decide what to investigate next. Preserve the starting group, the return rule and the stage of observation in that discussion. A useful AI summary makes those choices easy to inspect, allowing the owner to act on the question the numbers actually answer.

Sources

Official documentation checked on 28 October 2026. These are maintained support pages; the date records access rather than their original publication. All shop figures are fictional.

• Shopify Help Center: Customers reports, Customer cohort analysis section.
• Amplitude Docs: Retention Analysis FAQ.
• Amplitude Docs: How time works in a retention analysis.