Leveraging web application scanners for PCI compliance
October 15th, 2007 by admin Posted in Compensating Controls, Web Applications
Several people have written in to ask me about the different web application scanners and their applicability to PCI. One should remember that there are several requirements that a web application scanner could use used for, mainly 6.5, 6.6, and 11.3. This is not to say that a web application scanner will meet these requirements, but that it could be leveraged to assist compliance in these areas.
Liquid Information has a blog post referencing a paper on ha.ckers.org comparing the top tools.
He chose three scanners for the test, less known NTOSpider and two major players that are WebInspect and AppScan. He then ran these in default configurations against different kind of web applications and collected statistics/metrics on coverage and accuracy (e.g. amount of false positives, found vulnerabilities etc).
It looks like NTOSpider came our ahead, but read the results and let me know what you think.
![[del.icio.us]](http://images.del.icio.us/static/img/delicious.small.gif){kind=small}
4 Responses to “Leveraging web application scanners for PCI compliance”
By dre on Oct 17, 2007
NTOSpider did not come out ahead, as many people are alluding to. It’s not just because the scanners were run in default mode. The vulnerabilities should be categorized by criticality and well as weakness category. Take a look at the work Justin Schuh did as an example, is his review of automated code scanners. His data doesn’t even correlate to Larry Suto’s data.
I also spoke with Brian Chess (at Fortify Software, the vendor who’s tool Larry Suto used). Their tool, Tracer, is wildly mis-understood by the industry. These are expert tools written for experts, and this wild misinterpretation will not stand for long. Brian Chess said that he has done his own independent analysis of the web application security scanners using Tracer… and I assume only with different and better (possibly even more accurate) results.
There also has been discussion about the paper (and how Tracer works by measuring code coverage) on the fuzzing-l, securitymetrics-l, and dailydave mailing-lists. The consensus is that code coverage is great only up to around 10% per tool, in which case it hardly matters other than that you hit the important cluster of code with “good tests”.
It’s not the crawler or the default-mode, or how much it covers - but it always goes back to the data-driven security tests (built into the tool) and the tester (driving the tool).
For PCI DSS, since compliance does not equal security - it is best that those seeking compliance only run the automated “default” part of the tools that go after things only important for compliance in the most accurate way. What compliance seeks is specificity, while web application security scanners in general seek completeness in sensitivity. Sensitivity will give you all sorts of false positives, which is not something an auditor wants (or can deal with). There are plenty of smart attack modules for various web application security scanners that will do just the compliance piece. Stick with that stuff, and you’ll be ok.
It’s really an art form to turn a false positive into an exploitable, so many of the WebInspect FP’s in Larry Suto’s paper might actually be true positives. There is no way to know because his analysis is incomplete.
As a final point, he didn’t even include the most mature product, Cenzic Hailstorm.
By Adam Muintner on Oct 18, 2007
dre is 100% right
also check some of the comments at
http://ha.ckers.org/blog/20071014/web-application-scanning-depth-statistics/#comments
and on the dailydave list about this article.
I do not believe it was a meaningful test, and neither do any other well known and respected folks who’ve commented on it. Appeal to Authority isn’t always a logical fallacy, either.
By dre on Oct 18, 2007
Adam, I think the take from this is that you know the buzzwords “coverage” and “Tracer” are going to be thrown around by the masses as if they’re the new Christ. They’re all looking for the NTOSpider solution… a point-and-click interface so that they can claim they are the next web application security problem-solvers.
The tools will not turn you into an expert, especially a tool that does all the work for you. “A fool with a tool is still a fool”.
By Andrew van der Stock on Oct 22, 2007
Recently, I ran a automated test for a client who had invested a significant chunk of cash in one of the better tools. I had have some success with this tool on simple (non-Ajax PHP) software, but in this case, a Ajaxified JSF application, the scanner found literally nothing. Nada with the default settings, and even after manually guiding it to known bad areas and tweaking configuration, it still found nothing. It is thoroughly confused by JavaScript conventions and cannot detect even trivial to find XSS, DOM injection or anything like that. I was pretty disappointed, really.
I fear that if a lay person is aiming for PCI DSS compliance, buys one of these tools, they are going to get a wildly false negative more often than not, and will have their petard hoisted in the process.
Tools have their place in assisting with expert reviews, as they find low hanging fruit quickly, but they are not capable enough today to be on an equal par with a proper assessment using a qualified reviewer.
PCI DSS must be changed before someone loses a lot of money through believing these things.
Andrew van der Stock
OWASP Executive Director