What a Penetration Test Tells You, and What It Does Not

A penetration test answers one bounded question. Could this tester, in this window of time, inside this agreed scope, reach this agreed goal. It is a useful question and a far narrower one than most buyers realise they are asking.

It does not tell you whether you are secure. Nobody can tell you that, and a supplier who offers to is selling you a certificate of comfort rather than a test.

Which means the value is set almost entirely before the work begins, in the scope, the goal and the rules. A carefully aimed test against a small scope will tell you more than a broad sweep with no objective, and it usually costs less. The other half of the value is set afterwards, by what happens to the report. A findings list that goes into a drawer is money spent on knowing rather than on fixing, and knowing has a short shelf life.

An automated scan connects to whatever it can reach, compares versions and settings against a database of known weaknesses, and prints the result. It is fast, it is cheap, and it is worth running continuously. What it cannot do is chain two small problems into a large one or understand what your application is for.

A test is a person. They read your application, notice that an identifier in a web address can be edited, discover that editing it returns another customer's invoice, and then work out what else that access opens. No scanner finds that, because finding it requires knowing what the numbers mean.

You can usually tell what you bought by reading the report. Findings that all carry a public vulnerability reference and a generic remediation paragraph, with no narrative, no attack path and nothing about how the application behaves, is a scan with a cover page. Ask during procurement how much of the work is manual and who performs it.

Both have their place. Scan continuously, because it catches the weakness that appeared last Tuesday. Buy a test when you need to know whether the pieces can be joined together by somebody who is trying.

Write the scope as a list of systems, addresses and applications rather than as a sentence. A phrase like the customer portal means three different things to three people, and the disagreement always surfaces on the day a finding turns out to sit just outside it.

Then settle the parameters. Does the tester get credentials, and at which permission levels. Do they get source code and documentation, or nothing. Is social engineering in or out. Is testing that could take a service down allowed at all. Is this production or a copy, and if it is a copy, how faithful a copy. Where a third party hosts the system, get their written permission before anybody starts.

Say what a successful attack would look like in your business. Read another customer's records. Place an order without paying for it. Turn a standard user into an administrator. Reach the finance system from the guest wifi. Goals like those give the tester something to aim at, and they turn the report into something you can act on.

Agree the rules of engagement in writing: testing hours, contact numbers on both sides, what happens if something breaks, and what happens if the tester finds evidence that somebody else is already inside. That last one comes up more often than people expect, and it is not a conversation to have for the first time while it is happening.

Most of the findings that matter in a business application are about logic rather than about missing patches. Whether one customer can see another's records by changing a value. Whether a discount code can be used again after it expires. Whether a cancelled subscription still returns data. Whether an ordinary user reaches an administrative page simply by typing its address.

Say so in the brief. Give the tester more than one account at different permission levels, because a good share of these findings require using one account against another. Give them a walkthrough of what the application is for, since a tester who understands the business is the only one who will find business problems.

If you have an API, scope it explicitly and separately. Applications get tested through the browser all the time while the interface behind them quietly accepts requests the browser would never send. That gap is a common home for real findings.

Mobile applications need naming too. What is stored on the device, what the application trusts from the server without checking, and whether the traffic can be read or altered by somebody who controls the phone.

Read the attack narrative before the findings table. The narrative is where the tester describes what they did and in what order, and it is the part that tells you what somebody could actually achieve. A report with no narrative is a list of observations wearing a suit.

Then check whether severity has been set against your business or against a generic scale. A high rating on an isolated test system sitting above a medium rating on the server holding customer records is the wrong order for you, whatever the scoring standard says.

Look for chains. Several findings that each look minor and together produce full control of the environment are the most important paragraph in the document, and they are easy to miss when you read the table sorted by severity.

Then find what you can act on this week. Every good report contains some findings that are a configuration change and some that need a funded project. Split them on the day it arrives, put owners and dates against them, and put them into the same tracker as the rest of your work rather than into a separate security list nobody opens.

Here is the caveat a sales driven article leaves out. A test says that within this scope, in this window, with this tester's particular skills and methods, these were the findings. Change any one of those inputs and the answer can change with it.

Your environment also moves the next day. A release ships. A firewall rule gets added for a supplier. Somebody stands up a server for a demonstration and forgets it. The report describes a system that no longer exists in quite that form by the time you have finished reading it.

Be suspicious of a very short report. An assessment that finds almost nothing usually had a scope that made finding something difficult, or ran out of days, or was a scan all along. Ask the tester what they tried and failed to do. The failed attempts tell you more about coverage than the findings do.

Testers differ as well. Two competent people will find overlapping but different sets of problems, which is a reason to change supplier every few cycles rather than a reason to distrust the one you have.

Agree the retest in the same contract as the test, with a window and a price attached. A finding is closed when somebody has verified it is closed, not when a ticket says so. Fixes that miss the point are common, and they are most common for anything involving access control.

Treat the report as one of the most sensitive documents you hold. It is a written route into your systems complete with reproduction steps. Control who receives it, do not forward it around by email, and think about how long it should live in a shared drive.

Customers and prospects will ask for it. Do not send the full report. A summary letter from the tester confirming that a test took place, its scope, its date and the remediation status is the normal way to answer, and any decent firm will produce one on request.

And do not let the report become the security programme. It is a sample taken on one day by one person. The patching, the access reviews and the logging are what carry you through the months between tests.

Ask for a redacted sample report before you buy anything. What you are purchasing is a document and a conversation, and the sample tells you quickly which of those you are getting.

Ask who will do the work, by name, and what they have tested that resembles your system. Testing a web application, testing a mobile app and testing an internal network are related skills rather than the same one, and firms are not equally good at all three.

Ask about method and about how they will keep you informed during the engagement. You want to hear about something serious on the day it is found rather than a fortnight later in a document, and a good firm offers that without being asked.

Then the commercial questions nobody enjoys asking. Insurance. Liability if something breaks. Where the evidence and any extracted data is stored, how long they keep it, and how it gets destroyed. A firm that has answered these before will answer them quickly and in writing.