Google Play Closed Testing Service: Buyer Checklist

Buying a closed testing service is a procurement decision with a 14-day delivery window and no second chances inside a launch calendar. Google requires at least 12 testers opted in continuously for the 14 days preceding your application, so the thing you are procuring is continuity — not installs, and not a headcount.

This is the checklist we would want a buyer to run on OnTesters, so it is the one we publish. Some of it is generic procurement discipline, and some of it is specific to how this requirement fails.

Seat count: how many should you buy?

Start here, because it is the question vendors structure their pricing around and buyers most often get wrong.

So when a vendor offers tiers at 12, 16, and 20, the honest recommendation is 15-16. Roughly 40% of the rejections across the 1,500+ campaigns OnTesters has analyzed trace back to testers dropping below the minimum — a failure mode that a three-to-four-seat buffer almost entirely removes. That figure is our own campaign data.

If a vendor is still structuring its tiers around 20 as the requirement, that is a signal about how current their playbook is.

Continuity: the replacement question

Ask exactly this: if a tester opts out on day 8, what happens, who pays, and how quickly does the replacement opt in?

What you are listening for:

• A defined replacement SLA. "Same day" or "within 24 hours" is an answer. "We will look into it" is not.
• Who bears the cost. If replacement seats cost extra, continuity is not in the product.
• Whether replacements are counted correctly. A replacement's opt-in starts their own clock. They do not retroactively repair the preceding days, and a provider that implies otherwise does not understand the requirement.
• What happens if the cohort collapses. Several testers from one source going quiet at once is a real scenario. Ask what the provider does about it.

Devices: get specifics, not adjectives

"Real devices" is a claim, not a specification. Ask for the specifics that make it verifiable.

• Which Android device models, and how many unique models across your cohort?
• How do they verify testers are real people with their own Google accounts?
• What is their position on emulators and duplicate accounts?
• Are testers spread across Android versions?

Google's own best-practice guidance recommends recruiting a diverse group of testers who resemble your intended audience, specifically so you find bugs affecting particular device types and user groups. Device diversity is not a luxury add-on; it is how the testing produces useful signal. OnTesters averages 8-9 unique device models per campaign.

If a provider cannot name a device mix, they have not thought about this, and you will be paying for installs rather than testing.

Feedback: what you get, in what format

Google states that you must summarise your testing feedback when you apply for production access. Ask what arrives at the end of the 14 days.

Retesting and refund terms

Google documents that your app may be asked to continue testing, citing fewer than 12 opted-in testers and insufficient tester engagement. That is a real scenario, and it is the one that decides whether a package was good value.

Ask:

• Is retesting included, and if so, for how long and under what conditions?
• Is there a charge for running the window again?
• What exactly triggers a refund — non-delivery of seats, or the application outcome, or nothing at all?
• What evidence is required for a refund, and how long does it take?

Read the published refund policy rather than a sales message. "Full refund if Google rejects due to tester engagement" is a specific and checkable term; "satisfaction guaranteed" is not.

Things that should end the conversation

• A guaranteed approval. Google reviews applications. Google also states that meeting the criteria makes you eligible to apply — not approved. Any vendor promising the outcome is describing something outside the policy.
• Encouragement to add emulators or duplicate accounts to reach a count.
• Reluctance to let you verify the opted-in count in Play Console yourself.
• An approval-rate percentage with no defined denominator.
• "Emulators count." They do not produce genuine testing, and community product experts have said so repeatedly.
• Anything about needing 20 testers.

OnTesters' position under both our current and previous branding is the same: we sell testers, continuity, and retesting. We do not sell approval, and we will not tell you that a purchase makes the review outcome certain.

The checklist

1. Buy 15-16 seats, not 12 and not 20.
2. Get the replacement policy in writing, with a time commitment.
3. Ask what a collapse in the cohort triggers.
4. Get device model specifics and a unique-model count.
5. Confirm their stance on emulators and duplicate accounts.
6. Establish the feedback format and whether it is tied to app versions.
7. Confirm retesting terms and cost.
8. Read the actual refund policy, including triggers and evidence.
9. Confirm you can verify opt-ins in Play Console daily.
10. Ask what happens if the opt-in link has not propagated when your testers arrive.
11. Discount every claim about the approval outcome.

Seat sizing: the arithmetic behind the tier you pick

Vendors sell tiers, and the tiers are usually 12, 16 and 20. Working out which one you actually need takes about a minute of arithmetic.

So the honest answer to "should I buy 12 or more" is 15-16. Across OnTesters' own campaign data from 1,500+ analysed campaigns, testers dropping below the minimum accounts for roughly 40% of the rejections we see, and a three-to-four seat buffer removes most of that risk. Our own figures, from our own campaigns, not Google statistics.

The counter-argument to a bigger tier is coordination, and it is real. Twenty testers is twenty people to brief, verify daily and chase. That is why the recommendation is 15-16 rather than "as many as possible" — enough margin to absorb normal attrition, not so many that managing the cohort becomes the project.

If a vendor's smallest tier is 12 and their next is 20, the question worth asking is whether they will sell you 16. A provider building for continuity can usually accommodate it, because their operating cost is in replacements rather than in seats.

What to do when a service underdelivers

It happens, and the response matters more than avoiding it entirely. The order below is deliberate — escalating before you have evidence weakens your position.

1. Establish the facts before you complain

Check the opted-in count in Play Console and compare it against what you paid for. "My testers are not engaged" is a feeling; "the opted-in count has been 9 for four days" is a fact. Only the second one is worth sending.

2. Read the policy you agreed to

Refund terms in this category are usually tied to specific triggers — non-delivery of seats, or a rejection attributed to tester engagement. If your problem falls outside those triggers, you may be asking for a discretionary outcome, and knowing that before you write changes your approach.

3. Make one request, in writing, with a date

State what you need, by when, and what happens if it does not arrive. Vague escalation gets vague replies. A specific request with a deadline gets either the fix or a clear refusal, and both are useful information.

4. Decide whether to keep going

Two paths. If the count is recoverable and the window still has room, the fastest result usually comes from adding replacements from a different source yourself rather than negotiating. If the run is already broken, stop pouring effort into it and re-plan the dates. The restart-or-patch decision is in A Tester Opted Out: Closed Testing Next Steps.

5. Do not do the thing that feels satisfying

Padding the count with emulators or duplicate accounts to "salvage" the window is the one response with no upside. Google Play runs integrity checks aimed at exactly that pattern, and the production access review considers whether the testing was genuine. You would be trading a recoverable delay for a risk to the account itself, which is the argument set out in Do Emulators Count for Google Play Closed Testing?

The broader lesson is worth carrying into the next purchase: the reason to pay for continuity is that a dropout is a normal event rather than an exception, and the provider's job is to absorb it without you needing to escalate anything.

What to ask a vendor to send in writing

Verbal assurances are worth very little in a dispute, and asking for specifics is a reasonable request rather than a confrontational one.

• The replacement commit, with a time limit. "Replacements within 24 hours" is checkable. "We look after our customers" is not.
• A device list or model count for your cohort. OnTesters averages 8-9 unique device models per campaign, which is the kind of number a real pool can quote from its own records.
• What counts as a refund trigger, in the vendor's own words, so you can compare it against their published policy.
• Whether retesting is included, and for how long.

A provider that supplies all four without hesitation is describing an operation. One that answers each question with a reference to its guarantee is describing a marketing position, and the difference tends to surface on day 8 rather than during the sales conversation.

The five-minute version of this checklist

If you only have time for a few questions, ask these four, in this order.

1. What happens if a tester opts out on day 8, and how fast is the replacement? A provider selling continuity will have a rehearsed answer.
2. Which device models will my testers use? "Real devices" is a claim; a model list is a specification.
3. How many testers do you recommend for a new personal account, and why? Twelve is the floor, 15-16 is the working number, and 20 means their playbook predates December 2024.
4. What feedback do I get, and in what format? You must summarise testing feedback when you apply for production access, and How to Apply for Google Play Production Access explains why that summary matters.

Frequently Asked Questions

Should I buy 12 testers or more?

Buy 15-16. The requirement is a minimum of 12 opted in continuously for 14 days, and the extra seats are what protect you from a single dropout breaking the run.

Is retesting usually included?

It varies. OnTesters includes free retesting and replacements at the base tier. Ask any provider directly, because it determines whether a re-run costs you money as well as time.

What refund terms are normal?

Terms vary by provider. Read the published policy for specific triggers rather than accepting a sales description. A policy tied to a defined failure — such as a rejection caused by tester engagement — is more meaningful than a general satisfaction guarantee.

Can a service guarantee my app will be approved?

No. Meeting the 12-tester, 14-day criteria makes you eligible to apply, and Google reviews each application. Google states review usually takes seven days or less.

Do I still need to check Play Console myself?

Yes. The opted-in count in Play Console is the number Google measures, and daily checking is your only reliable way to catch a drop in time to recover from it.

Sources

Google Play Console Help — App testing requirements for new personal developer accounts (requirement, feedback summary obligation, review timing, continued testing reasons, diverse-tester best practice). Google Play Developer Community — closed testing metrics thread (community content, not policy text). OnTesters pricing, replacement policy, and campaign figures are our own.