App QA testing where the deliverable is a defect report
QA is not a phase where a tester confirms everything looks fine. It is the process of finding the ways a build fails, reproducing each one reliably, and describing it precisely enough that someone else can fix it without asking a question. The output is defects, and its quality is measured by how many of them were actionable.
We run manual QA on physical Android devices across five layers. Automated regression has its place in your pipeline; a human being deciding that something feels wrong is not something a test suite replaces.
What are the five layers of an Android QA pass?
Each layer answers a different question and produces a different kind of defect. A QA pass that skips compatibility and only re-runs functional checks will pass a build that breaks on half the market.
Layer 1
Functional
Does each feature do what the spec says?
Every user-facing flow gets walked end to end with valid input, invalid input, and no input. Onboarding, sign-in, the paid path, settings, export, and deletion. The failures here are unambiguous: a button that does nothing, a form that saves nothing, a total that arrives wrong.
Defect looks like: "Checkout applies a coupon, then removes it when the address changes on the next screen. Totals reconcile incorrectly at the order summary."
Layer 2
Regression
Did this release break something that used to work?
The flows that passed last cycle get re-run against the new build. This is the layer most teams skip when they are shipping fast, and it is the layer that catches the bug you introduced while fixing another one.
Defect looks like: "The export fix in build 41 broke file naming — exports now overwrite the previous file instead of creating a new one."
Layer 3
Compatibility
Does it survive hardware and software you did not develop on?
The same build on different manufacturers, Android versions, screen sizes and densities. Layout clipping, missing permissions, camera and media differences, and vendor background restrictions show up here and nowhere else.
Defect looks like: "The three-column dashboard grid collapses to overlapping cards below 360dp width. Reproduces on Moto G models, not on Pixel."
Layer 4
Stability and performance
What happens under sustained use?
Long sessions, backgrounding mid-flow, process death, rotation during a request, and use while the device is warm and throttled. Crash reproduction matters more than crash counting: a stack trace without a reliable reproduction is only half a report.
Defect looks like: "Force-stopping the app during an upload leaves the job in a stuck state. Reopening shows an indefinite spinner with no cancel option."
Layer 5
Accessibility basics
Can someone using TalkBack or a larger font complete the key flows?
Unlabelled controls, touch targets below the recommended size, text contrast failures, and layouts that break at increased system font scale. This is a baseline pass, not a conformance audit — the distinction matters and we do not blur it.
Defect looks like: "The primary submit control exposes no accessibility label; TalkBack announces it as an unlabelled button. Also clips at the largest system font size."
How do you classify defects?
Severity is assigned by impact, not by how hard the bug was to find. A rare crash that loses user data outranks a layout problem that happens on every screen, and the classification decides when you hear about it.
| Level | Meaning | Definition | When it reaches you |
|---|---|---|---|
| P0 | Blocker | The app crashes on launch, data loss occurs, or a core flow cannot be completed at all. | Reported immediately, not held for the daily summary. |
| P1 | High | A core flow works but produces a wrong or damaging outcome — failed payment state, corrupted export, lost user input. | Reported within the first day of observation. |
| P2 | Medium | A feature is degraded, confusing, or fails only under a specific configuration or device. | Reported in the next scheduled batch. |
| P3 | Low | Cosmetic, copy, or minor polish issues with no functional impact. | Batched into a single summary so the signal is not buried. |
How does regression work inside a 14-day window?
If your QA pass runs alongside a Google Play closed testing window, the shape of the work changes. You have 14 consecutive days, a build that can be updated at any point, and a decision to make at the end about whether the app is ready.
The pattern that works: test the current build for the first three or four days and bank an initial defect list, then fix in batches and re-run the affected flows after each release. By day twelve you are verifying fixes rather than hunting for new problems, which is the only state in which a final release decision is worth making.
Shipping at least two updates during the window also correlates with a materially higher production approval rate in our data — 89% against 53% for a window with no updates. QA work that produces fixes is doing double duty. The 14-day cycle covers how the two pieces run together.
What every defect includes
- Summary in one sentence
- Steps to reproduce
- Device model and Android version
- Expected versus observed behaviour
- Reproduction rate (for example, 4 of 4 attempts)
- Severity with a stated reason
What QA does not cover
- Security audits, penetration testing, and secrets or dependency scanning. These are specialist services and a manual tester with a phone is not a substitute.
- Performance benchmarking to a numeric standard. We describe observed behaviour on named hardware; we do not produce certified frame-rate or memory profiles.
- Legal or regulatory conformance, including accessibility certification under the ADA, EAA or any equivalent standard.
- Automated test authoring. If you need Espresso or Appium suites written and maintained, that is a different engagement and we will say so.
Manual QA and automated tests are not competing
An automated suite is a guard rail. It runs in seconds, it never gets bored, and it will tell you within minutes whether the login flow still works after a refactor. Nothing manual competes with that, and any team shipping regularly should have some of it.
What automation cannot do is notice something it was not told to look for. It will not report that a screen is confusing, that a spinner sits long enough for a user to give up, that the export produces technically correct output with an unusable filename, or that the app looks broken on a phone with a wider font setting. Those findings come from a person using the app with no prior knowledge of how it is supposed to behave.
A useful split
- Automate: sign-in, the paid path, data round-trips, and anything that has broken twice before.
- Manual, every release: new features, anything touching money or stored data, and the flows your analytics say are most used.
- Manual, each cycle: onboarding as a first-time user, error and empty states, and the device spread.
Usability findings are a separate discipline from defect reports. If that is what you are after, the usability testing page is the better fit.
QA scope and delivery
What a cycle covers, how defects arrive, and where the boundaries are.
What is included in an app QA testing cycle?
Five layers: functional testing of every user-facing flow, regression testing against the previous build, compatibility testing across devices and Android versions, stability testing under sustained use and process death, and a baseline accessibility check covering TalkBack labelling, touch targets, contrast and font scaling.
How do you decide bug severity?
By impact, not by how hard the bug was to find. P0 blockers crash on launch, lose data, or make a core flow impossible and are reported immediately. P1 issues produce wrong or damaging outcomes and are reported within a day. P2 issues are configuration-specific degradations. P3 covers cosmetic and copy issues, batched into one summary.
Do you write automated tests?
No. We do manual QA on physical devices. Automated regression suites in Espresso, Appium or similar are a different engagement. Our view is that automation is a guard rail and manual testing is the discovery layer — they solve different problems rather than competing.
Does QA testing include security or penetration testing?
No. Security audits, penetration testing, and dependency or secrets scanning are specialist services and a manual tester with a phone is not a substitute. We also do not certify accessibility conformance to any legal standard, and we do not produce certified performance benchmarks.
Can QA run alongside a Google Play closed testing window?
Yes, and that is usually how it is booked. Testing the current build for the first few days banks an initial defect list, fixes ship in batches, and the affected flows are re-run after each release. Shipping two or more updates during the window correlates with an 89% approval rate in our data against 53% for no updates.
What does each defect report contain?
A one-sentence summary, numbered steps to reproduce, the device model and Android version, expected versus observed behaviour, a reproduction rate such as 4 of 4 attempts, and a severity level with the reason for the classification.
Written and maintained by
Afrin Asha
Founder, OnTesters
Android developer and QA specialist. Built OnTesters after working through Google Play’s 12-tester closed testing requirement on real devices.
Platform figures on this page come from campaigns run through OnTesters. Read how the platform works or see the guides library.
Sources
- Set up an open, closed, or internal test— the tracks that host a QA cycle on Google Play
- App testing requirements for new personal developer accounts— the closed testing obligation, where relevant
- r/AndroidClosedTesting— evidence of the tester-recruitment problem this testing solves
Defect examples on this page are drawn from real findings in OnTesters campaigns, with app-specific details removed. Approval correlations come from our own campaign data.
Policy and pricing reviewed September 2026
Find the defects before your users do
Manual QA across five layers on physical Android devices, with every defect reported as steps, device, expected, observed and severity.