QA & Testing

Not with a plan to cover the codebase, because that never finishes and the first three months produce nothing anybody notices. Start with the paths where a defect costs money: checkout, login, whatever your invoices depend on. Write those at the API level, where they run in seconds and rarely break for cosmetic reasons. Then add a test with every bug fix, so the suite grows along the lines your product actually fails on rather than along the shape of the file tree. Two months in you have a small suite that catches real things, which is worth considerably more than a large one nobody believes. It also means the first thing your team sees is a bug that did not escape, rather than a chart.

No, and teams that try it tend to end up with an expensive suite and worse quality. Automation is for checks that are known, repeatable and boring, which is exactly what a person does badly on the fortieth run. It cannot tell you that a screen is confusing, that an error message blames the user for something the system did, or that a flow which reads fine on paper is impossible in a shop with one hand full. That is exploratory testing, it is a skilled job, and it works best timeboxed with written notes rather than as an unstructured click around. A one off migration check is usually cheaper done by hand than automated, and we will say so rather than bill you for the script.

Emulators are fine for most of the day and wrong at the moments that matter. What they do not reproduce is the manufacturer layer sitting on top of Android, aggressive battery management killing your background work, a camera or fingerprint sensor behaving differently, slow storage, or how the app feels on a two year old mid range handset with fifty other apps installed. So development runs on emulators, and anything touching hardware, permissions, notifications or performance gets confirmed on real devices chosen from your analytics rather than from a wish list. A device cloud covers the long tail cheaply. Two or three physical handsets on a desk cover the models you ship to most, and those are the ones worth owning.

Do not start by buying more parallel runners. That hides the problem and doubles the bill, and you still cannot tell anyone what the suite is for. Look at what the four hours is made of first. It is usually a large number of full stack browser tests, many checking a rule a unit test could confirm in milliseconds, plus a group that share an environment and therefore cannot safely run at the same time. Push checks down a layer, split the suite so a fast subset runs on every push and the full one runs nightly, and delete anything that has not failed for a real reason in a year. Parallelism afterwards, once the suite deserves the hardware.

It is the quickest answer and it creates two problems you meet later. The first is legal: a full copy on a laptop, or in a staging environment with weaker access control, is a breach waiting for an audit, and under most privacy regimes testing is not a defensible reason to be holding it. The second is practical: real data drifts, so a test that depends on customer 4471 being mid subscription fails the morning that customer cancels. We prefer builders in code that create the state a test needs, plus a masked extract where a case is genuinely too expensive to construct. Masked properly, meaning unrecoverable, not shifted by a few characters and called anonymised.

You need it when there is a date, a number and a consequence: a campaign, a ticket release, a payroll run, a partner integration with a contractual response time. Then the work is real, because you are testing against a figure somebody has agreed to rather than against curiosity. It is a waste when nobody can tell you what good looks like, because the report will say the system slowed down under load and everybody will nod. It is close to useless against an environment a quarter the size of production holding an empty database, which is the most common way it actually gets done. Sizing and data volume decide whether the numbers mean anything, and both cost money to get right.

Roughly a third, and that third is worth having because it is cheap and it runs on every merge: missing labels, contrast failures, images with no alternative text, form fields with no association to their label. What automation cannot judge is whether the alternative text says anything useful, whether focus order matches the visual order, whether a custom component announces its state when it changes, or whether a task can be completed with a keyboard alone. Those need a person, a keyboard and a screen reader they know how to drive. We run the automated layer in the pipeline and the manual layer per release on the flows that matter, and we tell you which findings are legal exposure and which are simply poor.

Not from the number of tests, which only ever goes up, and not from coverage. The measures worth reporting are how many defects reached customers this quarter against last, how long each one survived before something caught it, how often the same bug came back, and whether your team has stopped re-running failed builds out of reflex. Release frequency is a good indirect signal as well, because teams release rarely when releasing frightens them, and a suite people believe is most of what makes it dull again. We agree those numbers at the start and put them in front of you monthly, including the months when they did not move.