There are three overarching areas to consider:
Often, teams perform security projects without questioning why they're doing it. None of these are acceptable reasons:
You need to have a clear goal in mind before choosing a pen testing consultancy.
There are several questions to ask yourselves before looking for an external solution. While there's no conclusive list of questions, these are a good start:
You may have internal knowledge in your team that can answer some of these questions. You may be able to answer them already.
The next step is to write them down, share them, and get feedback. Document your exact requirements and seek internal agreement before reaching out to vendors.
If you don't know exactly what you're looking for or there is internal disagreement, you need to figure it out:
When you reach out to vendors, ask them to define what they believe your requirements to be as part of their pitch. Look for synergy between what you believe you need and what they believe you need. It helps you confirm that they have your best interests in mind.
Should you go for a general IT contractor or a security specialist? There is no clear-cut answer and it depends on your needs more than anything else.
I’ve worked with big integrators where the security team was virtually non-existent. Of course, that didn’t prevent the business from selling security services. ‘Security consultants’ do IT deployments or code Java until a security project arrives. They then gather in a team and deliver it.
This may work for you or not, but you need to know before finding an outsourced pentesting solution. If you need support in several areas, it might make sense to choose a generalist consultancy.
Are you looking for a more traditional pentesting consultancy? One where they complete testing, create a report, and hand it back to your team for remediation. Or are you looking for a consultancy that will support you through the journey from findings to fixed?
Once you have an idea of your needs, start some initial research. Most of this can be done online, but it’s a good idea to reach out to the infosec community too. People are usually happy to share their experiences.
Assuming that you have decided to go for a security testing company. There are some important factors to consider before, and after reaching out to potential firms.
Look for trust signals:
Decide whether it’s important that they’re active in the community
There’s a school of thought that good penetration testing companies should be attending conferences and publishing research papers.
That’s not necessarily true anymore. There are three issues to consider:
If you conclude that it is important to you, don’t approach this as a ‘tick box’ exercise, where all consultancies that have appeared at a conference, or published research, qualify for your shortlist.
When you have a shortlist of 3-4 firms you think may be a good fit, it’s time to reach out and start having some conversations with them. When you do, there are some things you’re going to want from them.
Ask them to put you in contact with organizations of a similar profile to yours. It’s important that they are of a similar profile or otherwise their feedback might not be as valuable. If you are an SME owner in the tourism industry, a reference by the CSO of a huge high-street bank is of little value. Chances are they are pouring money over the vendor and the firm is bending over backwards to ensure the bank is fully satisfied.
Don’t satisfy yourself with a conference call or conversation on the subject; consultants (security or otherwise) are paid to sound good even when they aren’t experts in what they are talking about. Try to push for a sanitized report (not the marketing sample) to see what a real-world deliverable looks like (more on this later). This is a very technical service you’re shopping for. You have to be able to trust both the management team and the technical team in the firm. There are several things you can look into when trying to establish that trust.
This falls a bit out of the scope of this article in the sense that has nothing to do with the firm’s technical competence. However, it is essential you consider these points as part of your due diligence process.
Ask vendors under what circumstances would they advise a customer to bear the risk of a vulnerability. If they can’t give a good example of this you might be dealing with someone who views security in a vacuum and doesn’t consider other business factors when framing recommendations.
Your vendor’s approach needs to be aligned with your business goals. This type of question should be asked to the people that will be directly involved in the technical delivery of your projects and not to your salesperson or account manager. At the very least, you should have a conversation around this with the head of the pentest practice (or technical director).
When working with a technical consultancy, the bigger it gets, the bigger the risk of being affected by the “team lottery”: The variation on service you will notice depending on who gets assigned to deliver your projects. There are two factors that can minimise the risk of the team lottery: The company’s workflow/methodology and the overall composition of the team.
Remember that companies don’t perform penetration tests, people do. So no matter which company you go to, it always boils down to the person you have working on your account.
Cut through the sales layer and try to reach the technical director or pentest practice leader. If you are going to spend any significant amount of money request a conversation with the testers assigned to your project. Or at the very least request their CVs/bios.
Does their general work experience makes you comfortable (e.g. someone that just started their pentesting career may not be the best fit to test your critical AS/400 mainframe)?
If in doubt request a conversation or find out if someone else can be assigned to your project. Scheduling is very fluid in pentest firms and they should be able to accommodate such requests. The goal of this exercise is to minimise the team lottery by being vigilant and pushing back.
The firm’s size is also a factor, the bigger the consulting organization gets, the more likely the consultants will be generalists as opposed to specialists. This may or may not be an issue for you. Depending on your requirements, your needs may be better served by a generalist.
Finally, figure out if the company subcontracts any work. Don’t get me wrong, some of the finest testers I’ve worked with wouldn’t change freelancing for any job in the world. However, when third parties are involved you have to double-check the situation with the firm’s legal coverage (e.g. liability insurance) and the due diligence you performed on the main team’s technical leadership and members of the team should be extended to any third parties and contractors.
Even if they have a bunch of great people in the team, there are still some important things to consider about the firm’s methodology and processes.
The first one is the testing methodologies the company has for the different types of engagements that will be relevant to your company. It is of no use to you if the company excels at wireless assessments if you just need a code review. Creating and maintaining a high-quality testing methodology is not without its challenges and the bigger the penetration testing firm, the more important their methodology becomes.
There are a number of industry bodies that provide baseline testing methodologies including:
The fact that your point of contact is aware of some of these organisations does not mean that the team assigned to your engagement will follow their methodology.
Have a conversation with the technical director about the methodologies used by the team. And then have the same conversation with the team members assigned to your project. If you get different responses from the technical director and the team members the chances are the firm is not seriously following any defined testing methodologies.
Consider whether engagements are typically run by a single person or routinely involve several testers. Having multiple testers in your project ensures that a wide range of skills and expertise are brought to bear against your systems which maximises your chances of uncovering most of the problems.
If multiple testers are going to be involved in your assessment, how does the team coordinate their efforts to ensure there is no time wasted and that all points in the methodology are covered?
If your test team is on the same page and has the right collaboration tools, you will ensure there won’t be any time wasted, tasks will be split efficiently among the available team members and all points in the methodology will be covered.
In most cases, when you engage a penetration testing firm, the final deliverable you receive is a security report. Before making your decision and choosing a vendor, it is important that you are provided with a sample report by each prospect. You should also ask how they create their reports and how long it takes them. You don't want to be paying them to spend 10 hours per project when a pentest report automation tool could help reduce that to 3 hours.
The report needs to be able to stand on its own, providing comprehensive information about the project: from a description of the scope to a high-level, executive summary of the results and a detailed list of findings.
It should also provide remediation advice and any supporting information required to validate the work performed by the team (does it look like they attained sufficient coverage?) and verify that issues had been successfully mitigated after the remediating work is performed.
Whilst some of the report sections have to be very technical and full of proof-of-concept code, requests or tool output, the report also needs to present the results of the engagement in the context of your business.
Sure, you found three Highs, seventeen Mediums and twenty Lows, what does it mean for my business? Should I get the team to stop doing what they are doing and fix all the issues? Some of them? None of them?
All findings are not created equal, and some testers get carried away by the technical details or the technical mastery required to find and exploit the issues and forget about presenting them in a context that matters to your business.
In general the more experienced the tester, the more emphasis will be put on the business context around the findings uncovered (of course “experience” is not a synonym for “age”).
As a result, and to try to avoid the team lottery mentioned above, in an ideal world, you would like to be provided with a sanitised report written by the same person that will be writing your deliverable.
This may not be practical in every instance but if you are going to engage on a mid-size or larger assessment, I think it is reasonable to push for this sort of proof to ensure that the final document you will receive is legible, valuable to your business and of an overall high-quality standard.
This week in Cyber is a weekly email with the latest news, research, and discussions from the world of cyber security. Sign up:
Your email is kept private. We don't do the spam thing.