In this post I am going to cover some pitfalls of Penetration Testing. It is kind of three rants stitched together. Loosely around the theme of how we generally interact with customers, as well as the reporting processes that I have seen over the last 15 years.

A person whose job it is to respond to penetration testing findings was asked this question:

• What are the pain points you have experienced when responding to Penetration test findings?

This is what they said:

It is also worth noting that this was not a customer of ours.

I yelled “preach!”. Whoever this was I really love that they hit the nail on the head. I opened my most recent report where I had tackled that concern , I hope, adequately:

I hope that if the anonymous responder were to have seen my report. That they would at least see that I considered their plight, and that I have given them an easy out when responding to their manager. “Look, this guy even said it is possibly a false-positive”.

The target had a server banner which, if true, was vulnerable to several things. Unfortunately the OS was not listed in the banner (and was not otherwise 100% confirmed) so I could not prove or disprove the versions without either exploiting the issue, or being given more access. Had the banner said “RedHat” then I would most definitely have changed what I said. It would say there is a high potential that backporting was being used.

This set me off thinking again about how our industry often fails the customers we are paid to help.

If our industry has heroes they may or may not wear capes. But they almost definitely work on the blue side in my opinion. The brave souls tasked with the gargantuan task of interpreting penetration testing reports. From multiple consultants, from different vendors. The variability of output is enormous. These warriors have to find someway to make it work regardless of what thing has arrived as the deliverable.

I have seen Pentest companies who try to solve it in two ways:

• Dictatorship – Based on one person’s vision you set a reporting standard.
  • You develop a rigid knowledge base of vulnerability write ups which tells everyone exactly how to report something. This includes fixed recommendations which must be provided.
  • You retrain every consultant in your team to meet that standard.
  • You yell at people during QA to remove any sense of individuality in reporting.
  • You fall out over CVSS risk ratings because “we need to risk this exactly the same way as the customer got an XSS which was 6.5 last week”.
  • Some Customers LOVE This. They don’t want any variability because the master spreadsheet they have with all vulns exists. They want the exact risk score for every instance of a vulnerability ever. They just like it neat.
  • The goal is to make every report as identical as possible across any customer and from any member of the team. Robotic Reporting.
• Cheerful Anarchy – You set a baseline standard for reporting by providing a structure for the reporting and a style guide. Then you let folks have at it!
  • You accept that Pentesting is consultancy profession. Which is influenced by the experience of the consultant doing the work along with their understanding of the risk appetite for the customer.
  • You provide a basic knowledge base of vulnerability write ups which covers a consistent title, background, and baseline risk score. Then encourage the consultant to produce the remaining content just for that project.
  • You train your consultants to understand risk calculation and expect them to alter the baseline risk considering every instance they see.
  • The goal of this is to make every report tailored. Therefore inconsistencies will exist such as two consultants finding the same vulnerability with the same impact but providing different risk ratings.

Of the two I have always preferred cheerful anarchy. I know that some customers absolutely want a penetration test to deliver consistent results over time. It helps them sleep at night. I argue that a little anarchy might be good since the consultant should be free to express their opinions SO LONG AS THEY EXPLAIN THEM WELL ENOUGH.

In truth you need to essentially support both in 2020. Big accounts who want the consistency need to get it. Other customers who are perhaps in earlier stages of their security maturity processes should be given tailored findings in my opinion. They haven’t necessarily encountered an SQLi before, so you need to contextualise it a lot more. So I recommend being so flexible that you can be rigid… I suppose?

Places where a penetration tester needs to be super clear is when dealing with potential false-positives. If the only evidence you have is from a vulnerability scanner then you have not done a good job. I implore you to always find some other means of confirmation.

In situations where the vulnerability is raised only based on banners.. Then your flow is to:

1. Find a working exploit. If you can, then try to exploit a docker container or VM with the same software first to verify the payload works well. Ask the customer if you can use the exploit. If you have done it in your lab first you can explain that it works well without a risk to stability. Otherwise you can warn them that it may trigger an outage. They can then make the decision themselves as it is their risk.
2. If no exploit is available. If you can, then execute OS commands to verify the installed patch. In most cases you do not have this access. You can either document the finding with caveats (as my report did), or.. and I appreciate this is a revolutionary idea. You can ASK the customer to confirm the installed version themselves and provide a screenshot. In my case the time was not available to do so and I was forced into the caveat approach.

I know, I know. I suggested you speak to the customer! Worse still I say you should ask them to support you improving the quality of how you serve them. You should not forget that a Penetration Test is a consultation, and that you are on the customer’s team for the duration of the engagement.

They say you should never meet your heroes. But it has been going really well for me when I speak to them so far.

Hope that helps.