There’s been a recent post on the [pen-test] mailing list asking for advice on the things to consider when choosing a independent penetration testing company. The original request went as follows:
I’m currently in the process of sizing up/comparing various
Penetration Testing firms, and am having a bit of trouble finding
distinguishing characteristics between them. I’ve looked at a fair
few, but they all seem to offer very similar services with little to
recommend one over another.
The thread was full of good advice. However, having first-hand experience in a number of these penetration testing firms I thought it would be a good idea to dig a bit deeper into the subject: what makes a penetration testing company great?
What are your requirements?
But first things first, do you need a penetration test? Do you know what a penetration test consists of? What would be the goal of the test if you performed one? These are really the key questions that you need to be able to answer before even considering choosing an external security partner.
Unfortunately, in depth answers to those questions fall outside the scope of this article. Doing a bit of internet research as well as reaching out to industry colleagues, peers and business acquaintances that are in a role similar to your own would be a first step in the right direction.
It is not a bad idea to ask each of the vendors you evaluate to help you with the answers to those questions. It will enable you to understand their approach. Do they really have your best interest in mind? Will they make sure that you are able to define the problem and sketch your goals before jumping to their keyboards and sending you an invoice? Are they knowledgable enough on the relationship between security and your business and the tradeoffs involved?
Contrary to what one might think, in the majority of the cases security projects are performed “just because” without a clear goal in mind:
- We made a change in the app and policy says we need to have it pentested.
- We deployed a new server and IT said we need to pentest it.
- A year has passed since our last test and we have to do it again.
- …
If you are completely lost on this and don’t know what services you need or what services might be available, it may be worth getting some external help just to help you clarify your requirements. You can bring in an independent consultant for a couple of days to help you gain a sufficient understanding of your own requirements to ensure that when you go out and shop around for security partners you know what you are up against.
IT generalists vs penetration testing specialists
After correctly understanding what your problem is and what type of testing you need, the next thing to get out of the way is to decide whether you should go for a general IT contractor or a security specialist. As usual, there is no clear cut answer and it depends on your needs more than anything else.
I’ve worked with big integrators where the security team was virtually non-existent. Of course that didn’t prevent the business from selling security services. ‘Security consultants’ would spend their time doing IT deployments (e.g. firewall and router configurations, etc.) or coding Java until the off security project arrived when they would gather in a team and deliver it. This may work for you or not, but it’s worth thinking about. Do you need a lot of support in several IT areas? It would definitely be easier to establish a relationship with a single IT provider than shopping around for specialist vendors for each area.
When dealing with a generalist, make sure you understand their approach to security testing. Most of the advice given below for firms can be directly translated to the “security function” inside a bigger consultancy or integrator.
Company background
Lets assume that you have decided to go for a security testing company. What are the important factors to consider before making a decision?
Trust
As with all business decisions, trust is a very important factor when evaluating security services vendors. Can you get any verifiable references of any of these guys? If you approach a firm and made them aware that X from Y company recommended their services, you are likely to get a better deal than if you didn’t. The level of service you will get will also be different as failing/disappointing you can potentially have the risk of upsetting the existing relationship they have with the people that referred you on the first place.
You can always ask the different vendors to put you in contact with organisations of a similar profile to yours. It’s important that they are of a similar profile or otherwise their feedback might not be as valuable. If you are a SME owner in the tourism industry a reference by the CSO of a huge high-street bank is of little value. Chances are they are pouring money over the vendor and the firm is bending over backwards to ensure the bank is fully satisfied.
You can ask for examples of similar projects they have undertaken. Don’t satisfy yourself with a conference call or conversation on the subject, consultants (security or otherwise) are paid to sound good even when they aren’t experts in what they are talking about. Try to push for a sanitised report (not the marketing sample) to see what a real-world deliverable looks like (more on this later).
If you can’t get any solid references or pointers through your business contacts, you’ll need to establish trust by yourself. This will take a bit of work and time but it is definitely worth the investment. Also, beware that this is a very technical service you’re shopping for. You have to be able to trust both the management team and the technical team in the firm. There are several things you can look into when trying to establish that trust.
Research and conferences
Something that you hear often when shopping around for penetration testing providers is that the company “should present at conferences”. This in principle sounds like a reasonable idea: if the team is on top of their game, they will be performing cutting edge research that will be of interest to the security industry which will earn them a spot in the security conferences. However, the truth is that with an ever growing number of security conferences every year which in turn have an ever growing number of different tracks running in parallel, not all conference speaking slots are created equal.To give you an example, this month there are at least six of them (not counting BSides London that we are sponsoring ;)).
When evaluating security conference presence, it is important to analyze the contents that were presented. Was it really research or does the company employ a well known industry expert that is regularly invited to speak at the conferences to give their opinion on the state of the art? Does the research have sufficient breath and depth or was it put together in a rush to have it in time for the conference? Was it relevant to your business? For instance, imagine you need to get your SAP deployment tested. Even if a well-known company has someone finding amazing bugs in some cutting edge technology like NFC, you may be better served by a lesser known company presenting on a SAP testing methodology or on the SAP testing toolset they have created over the last few years doing this type of testing.
Something similar could be said about published advisories: the fact that a company has 100s of published advisories may or may not be relevant to your needs. Are the advisories in technologies your company uses? If all their advisories are on Microsoft-related technologies and you are a Linux/Solaris shop, that wouldn’t help. This is a tricky one to assess, especially for non-security people, but it is worth to be on the look for the “but we publish advisories!” line and ask a few follow up questions to see if the company’s background is aligned with your own needs.
Finally, is all this conference presence and research recent enough? The security industry changes quickly and even though security specialists are fairly loyal to their employers, they move on from time to time. Double-check your facts to ensure that the research the vendor is presenting you as proof of competence is recent and that the authors are still with the company. The same could be said of books, courses or tools that have been written “by company members”. Verify they are still around to help your company and if your find out they are not, at least call their bluff to see how your point of contact reacts. The savvier you look in their eyes the better 🙂
The legalese
This falls a bit out of the scope of this post in the sense that has nothing to do with the firm’s technical competence. However it is essential you consider these points as part of your due diligence process:
- Does the company carry sufficient insurance and reasonable legal agreements ?
- Are there any NDA terms that you need to discuss with them?
- Does the firm hold any relevant certifications that your company might care about (e.g. ISO 27001)?
Their approach to testing
After covering the basics of the company’s background, the next thing to focus on is their approach to testing.
There is a lot of solid advice on this subject on this 2007 post by Chris Eng at the Veracode blog, I’ll include a few references to it here, but please go and read it now, it’s well worth the time.
For instance, Chris recommends asking vendors under what circumstances would they advise a customer to bear the risk of a vulnerability. If they can’t give a good example of this, he continues, you might be dealing with someone who views security in a vacuum and doesn’t consider other business factors when framing recommendations. This hits the nail on the head. Your vendor’s approach needs to be aligned with your business goals. Otherwise the return on your investment will be very poor. This type of question should be asked to the people that will be directly involved in the technical delivery of your projects and not to your sales person or account manager. At the very least, you should have a conversation around this with the head of the pentest practice (or technical director).
Team lottery
When working with a technical consultancy, the bigger it gets, the bigger the risk of being affected by the “team lottery”: the variation on service you will notice depending on who gets assigned to deliver your projects. There are two factors that can minimise the risk of the team lottery: the company’s workflow/methodology and the overall composition of the team.
The team
I want to open this section with a quote from Avoid wasting money on penetration testing that makes a great point:
Finally, remember that companies don’t perform penetration tests, people do. So no matter which company you go to, it always boils down to the person you have working on your account.
It is key to cut through the sales layer and try to reach the technical director or pentest practice leader. If you are going to spend any significant amount of money, I’d push it even harder (at least for the first engagement, and every now and then too) and request a conversation with the testers assigned to your project. Or at the very least request their CVs/bios. Do they have experience working under your requirements? Does their general work experience makes you comfortable (e.g. someone that just started their pentesting career may not be the best fit to test your critical AS/400 mainframe)? If in doubt request a conversation or find out if someone else can be assigned to your project. Scheduling is very fluid in pentest firms and they should be able to accommodate such requests. The goal of this exercise is to minimise the team lottery by being vigilant and pushing back.
The firm’s size is also a factor in this equation, as Chris puts it, the bigger the consulting organization gets, the more likely the consultants will be generalists as opposed to specialists. This may or may not be an issue for you. Depending on your requirements, your needs will be better served by a generalist. On the one hand, you don’t want a reverse engineer that specialises in subverting DRM libraries for embedded systems running your external infrastructure pentest, on the other, you don’t want a generalist looking at your DRM library. Again, this goes back to square one: knowing your requirements.
Another way to try to avoid the lottery is to go for a fairly small team where you know each person is well worth their salt and you will get top-shelf service every time. However this isn’t easy to find (or evaluate) and depending on your own firm’s size you may need to use a bigger vendor (smaller firms can’t usually accommodate too many projects or multiple concurrent projects for a single client). Even though these are not very common, such specialist boutiques exist and depending on your situation, size and approach they could be a great fit.
Finally, another interesting subject is to figure out if the company subcontracts any work (to other firms or to freelancers). Don’t get me wrong, some of the finest testers I’ve worked with wouldn’t change freelancing for any job in the world. However, when third parties are involved you have to double check the situation with the firm’s legal coverage (e.g. liability insurance) and the due diligence you performed on the main team’s technical leadership and members of the team should be extended to any third parties and contractors. Moreover, subcontracting introduces additional challenges in the collaboration and methodology department, which as we will see in the next section, are not free of complications.
Workflow, tools and methodology
Even if they have a bunch of great people in the team, there are still some important things to consider about the firm’s methodology and processes.
The first one is the testing methodologies the company has for the different types of engagements that will be relevant to your company (e.g. it is of no use to you if the company excels are wireless assessments if you just need a code review). As discussed in Using testing methodologies to ensure consistent project delivery creating and maintaining a high quality testing methodology is not without its challenges and the bigger the penetration testing firm, the more important their methodology becomes.
There are a number of industry bodies that provide baseline testing methodologies including:
Be advised that the fact that your point of contact is aware of some of these organisations does not mean that the team assigned to your engagement will follow their methodology (or any other methodology for that matter). Have a conversation with the technical director about the methodologies used by the team. And later on have the same conversation with the team members assigned to your project. Protip: if you get different responses from the technical director and the team members or different responses from different team members chances are the firm is not seriously following any defined testing methodologies. For example, if the technical director mentions OSSTM and CREST, and the team leader mentions OWASP and another team member says he mainly relies on his years of experience, that should be a red herring.
Another key part of the firm’s workflow to consider is whether engagements are typically run by a single person or they routinely involve several testers. I’ve already discussed about the importance of collaboration in the past, having multiple testers in your project ensures that a wide range of skills and expertise are brought to bear against your systems which maximises your chances of uncovering most of the problems.
If multiple testers are going to be involved in your assessment, how does the team coordinate their efforts to ensure there is no time wasted and that all points in the methodology are covered? If your test team is on the same page and have the right collaboration tools, you will ensure there won’t be any time wasted, tasks will be split efficiently among the available team members and all points in the methodology will be covered. If on the other hand, the company does not have the tools or processes defined to ensure seamless collaboration and task splitting, some of the time allocated to your projects will be wasted and some areas of the methodology may remain unexplored while the team is spending time trying to manage the collaboration overhead.
The penetration testing report
In the majority of the cases, when you engage a penetration testing firm, the final deliverable you receive is security report. Before making your decision and choosing a vendor, it is important that you are provided with a sample report by each prospect.
The report needs to be able to stand on its own, providing comprehensive information about the project: from a description of the scope, to a high-level, my-CEO-would-understand-this-language executive summary of the results and a detailed list of findings. It should also provide remediation advice and any supporting information required to both validate the work performed by the team (does it look like they attained sufficient coverage?) and verify that issues had been successfully mitigated after the remediating work is performed.
Whilst some of the report sections have to be very technical and full of proof-of-concept code, requests or tool output, the report also needs to present the results of the engagement in the context of your business. Sure, you found three Highs, seventeen Mediums and twenty Lows, what does it mean for my business? Should I get the team to stop doing what they are doing and fix all the issues? Some of them? None of them? All findings are not created equal, and some testers get carried away by the technical details or the technical mastery required to find and exploit the issues and forget about presenting them in a context that matters to your business. In general the more experienced the tester, the more emphasis will be put in the business context around the findings uncovered (of course “experience” is not a synonym of “age”).
As a result, and to try to avoid the team lottery mentioned above, in an ideal world you would like to be provided with a sanitised report written by the same person that will be writing your own deliverable. This may not be practical in every instance but if you are going to engage on a mid-size or larger assessment, I think it is reasonable to push for this sort of proof to ensure that the final document you will receive is legible, valuable to your business and of an overall high-quality standard.
tl; dr;
- Your requirements, to get the best value for your investment you need to know what you need help with, is it a pentest? or just a VA? or help with some basic security awareness training for your development team?
- Trust, can someone recommend you a trustworthy security vendor? If no, then for each prospect partner try to figure out what’s the firm’s background? Have they worked with clients in your industry? Are they interested in your business? Do they perform any research in areas that are relevant to you?
- Their approach, who will be delivering your assessment? Do they understand your business and motivations? What is their workflow like? Do they have a process in place to ensure consistent, high quality results every time?
There are a lot of moving parts in this process, and not all of them will apply to every vendor and every company looking for a penetration testing provider.
Here are Security Roots we can’t help you with your security testing needs (we will stick to doing what we know best), but hopefully now you should be better equipped to consider all the pros and cons and some of the gotchas involved in deciding what security firm you should trust with your business.
