There’s been a theme of dishonesty and thievery in the Penetration Testing industry for as long as we can remember. Much in the same way that merchants sold “snake-oil” as a cure-all for what ails you, Penetration Testing vendors sell one type of service and brand it as another thus providing little more than a false sense of security. They do this by exploiting their customers lack of expertise about penetration testing and make off like bandits. We’re going to change the game; we’re going to tell you the truth.
Last week we had a new financial services customer approach us. They’d already received three proposals from three other well-known and trusted Penetration Testing vendors. When we began to scope their engagement we quickly realized that the IP addresses that they’d been providing were wrong. Instead of belonging to them they belonged an e-commerce business that sold beer-making products! How did we catch this when the other vendors didn’t? Simple, we actually take the time to scope our engagements carefully because we deliver genuine Penetration Testing services.
Most other penetration testing vendors do what is called count based pricing which we think should be a major red-flag to anyone. Count based pricing simply says that you will pay X dollars per IP address for Y IP addresses. If you tell most vendors that you have 10 IP addresses they’ll come back and quote you at around $5,000.00 for a Penetration Test ($500.00 per IP). That type of pricing is not only arbitrary but is fraught with serious problems. Moreover, it’s a solid indicator that services are going to be very poor quality.
Scenario 1: The Overcharge (Too much for too little)
If you have 10 IP addresses but none of those 10 IP addresses are running any connectable services then there’s zero seconds worth of work to be done. Do you really want to pay $5,000.00 for zero seconds worth of work? Moreover, is it ethical for the vendor to charge you $5,000.00 for testing targets that are not really testable? While we don’t think its ethical we’ve seen many, many vendors do this very thing.
Scenario 2: The Undercharge (Too little money for too much work)
What if those 10 IP addresses were serving up medium complexity web applications? Lets assume that each web application would take 100 hours to test totaling 1,000 hours of testing time (not including the reporting, presentation, etc.). If you do the math then that equates to an absolutely absurd hourly rate of $5.00 per hour for the Penetration Tester! Of course, no penetration tester is going to work for that much money so what are you really paying $5,000.00 for?
Well, lets assume that the very-low-end cost of a penetration tester is around $60.00 per hour (its actually higher than that). In order to deliver 1,000 hours of work at $5,000.00 the test would need to be 92.7% automated resulting in an exact hourly rate of $60.24 per hour. Do you really want to pay $5,000.00 for a project that is 92.7% automated? Moreover, is that even a Penetration Test?
The terms Penetration Test and Vulnerability Scan only have one correct definition. The definition of Penetration Test is a test that is designed to identify the presence of points where something can make its way into or through something else. Penetration Tests are not assessments (best guesses) of any kind. A Penetration Test either successfully penetrates or it does not not, there is no grey zone; there are no false positives. (This is why we can guarantee the quality of our penetration testing services.)
A Vulnerability Assessment on the other hand is a best guess or an educated guess as to how susceptible something is to risk or harm. Because it is an assessment and not a test there is room for error (guessing wrong) and so false positives should be expected. A Vulnerability Scan is similar to a Vulnerability Assessment only instead of a human doing the guess work a computer program (with a much higher margin of error) does the guess work.
So, if 92.7% of a service is based on Vulnerability Scanning then how is it that Penetration Testing vendors can label such a service a Penetration Test? They should call it what it really is, which is a Vetted Automated Vulnerability Scan; and a Vetted Automated Vulnerability Scan is about as effective as Penetration Testing a bulletproof vest with a squirt gun. We’re not sure about you but we wouldn’t want to wear that vest into battle. These types of services provide little more than a false sense of security.
Back on track…
The question becomes how should a vendor price their services and build their proposals? While we won’t disclose our methodology here because we don’t want to enable copycats, we will provide you with some insight through an analogy.
Your car breaks down and you call a random mechanic. You tell the mechanic what sort of car you drive and how many miles are on it but never provide any details as to what happened. The mechanic then quotes you $300.00 to fix your car (without ever really diagnosing it) and $50.00 for a tow. Would you bring your car to that mechanic? How can he afford to fix your car for $300.00 and make a profit? To accomplish that must have an arsenal of junk parts gnome slaves working for peanuts, right? Would you really trust the quality of his work? That is count based pricing.
Fortunately automobile mechanics (most of them anyway) are more ethical than that (and gnome slaves don’t really exist anyway). Most of them won’t deliver a quote until after they’ve evaluated your car and successfully diagnosed the problem. Once diagnosed they’ll provide you with an itemized quote that includes parts, labor, taxes, and a timeframe for service delivery. They won’t negotiate much on price because in most cases you are getting what you pay for.
Genuine Penetration testing vendors are no different than genuine mechanics. All Penetration Testing vendors should be held to the same standard (including us). What you pay for in services should be a direct reflection the amount of work that needs to be done. The workload requirement should be determined through a careful, hands-on assessment. This means that when pricing is done right there is no room to adjust pricing other than to change workload. Any vendor that offers a lower price if you close before the end of the month is either offering arbitrary pricing or they are padding their costs.
What if a vendor truly needs to discount services? When we deliver Penetration Testing services we don’t charge for automation. If we automate 10% then our services are discounted 10%. If we automate 100% then our services are delivered to our customers free of charge. (Yes, that’s right, we’ll scan you for free while other vendors might charge thousands of dollars for that). Why charge for automation when it takes less than 3 minutes of our time to kick off a scan? Just because we can charge for something doesn’t mean that it’s ethically right.
Anyway, this article is one of many to come in our whistle blower series. Please feel free to share or contact us with any questions.
If you feel that what we’ve posted here is inaccurate and can provide facts to prove the inaccuracy then please let us know. We don’t want to mislead anyone and will happily modify these entries to better reflect the truth.