Society of Payment Security Professionals - Payment Security Blog

1. 7 Responses to “PCI Certification doesn’t make a website harder to hack”

2. By DAG on Jun 19, 2007

   Where to start?

   I think if you pick any one PCI requirement you can blow holes in it. Consider more together, it becomes harder. Consider them all harder still. Lots of healthy room for debate, clarifications, improvements, etc. Perfect? Hardly. But that’s the nature of life.

   Back to scanning and web application scanning.

   1. Scanning is not an exact science. There is certainly room for improvement. There are also inherent limitations in the ASV standard. In short, you’re damned if you do and damned if you don’t. By way of example, the non-intrusive requirements force ASV’s to avoid hard tests for things like buffer overlows or DoS. Or even things that will be intrusive on a significant set of technologies. Enter false alarm/potential vulnerability appeals. Some of the OWASP requirements fall in this category.

   2. There is an implied objective in the ASV scanning program that scanning must scale to large numbers of merchants and that costs will come down. This requires high degrees of automation and accurate interprettation of results. Web application scanning is simply not as mature as more traditional vulnerability scanning. It’s coming along but there is a long way to go.

   3. Web application scanning tends to grow near exponentially. Every public and private page, times every field and attribute, times every variation of every test. It gets very large very very fast. I’ve seen relatively small sites generate millions of tests. It’s also got a much higher false alarm rate.

   4. Web application scanning is also inherently more complex than tradiitional vulnerability scanning. Acurate navigation through custom code can much more challenging than looking for known responses from known probes.

   5. Many other aspects of web application scanning are maintenance intense.

   6. What is important now is a realistic and level playing field with shortcomings and limitations addressed through other means. Many of these will be addressed over time and through other means. The qualification process and annual requalification helps address this.

   Scanning is only a piece of the puzzle. There are lots of other PCI requirements that hit upon this.
   - internal and external vulnerability scanning
   - annual web application security testing
   - annual network penetration testing
   - development processes that include OWASP
   - (soon) third party code review of web apps
   - sever hardening

   Consider, some of these are nowhere near as tightly defined. Consider the pen test. It doesn’t need to be exploitive (and shouldn’t have to be). But it needs to do more than just internal and external vulnerability scans. But how much more or what is not specified in detail. How’s the merchant to know? There are a lot of vendors out there that will try to sell the merchant a Caddy when they need a bicycle. How’s the merchant to know?

   Not all of these are required to be third party. Demonstration of compliance is based on levels which is based on risk. Large players will need to have ROC’s which will touch on all these requirements. Small players sign off on the questionnaire process and may miss some of these. Not perfect. But much better than if there were nothing.

   Even a compliant ROC is no promise. If you’re breached an approved forensic company will reassess. And you are on the hook if they find you non-compliant.

   A scan, any scan no matter how many bells and whistles or feature and technologies, cannot prove you are secure. Never. It can only prove you’re insecure.

   We also have the changing nature of OWASP. When it changes, what does PCI do? This is similar to someone enshrining PCI in LAW by name. What happens when PCI changes? I’m not a lawyer, but I suspect that anyone charged under such a law would have a good defense once the underlying standard changed.

   The answer is defense in depth. Given the other pieces of the puzzle noted above, PCI does that.

   Lastly, think about how many organizations did web application security testing now as compared to before PCI. I don’t have exact numbers or trends, but I would bet that PCI has dramatically increased the penetration (pun intended) of this technology.

   Mr. Grossman, I hear what you’re saying and understand the sentiment. But I really think you need to surface a bit and take a look around.

3. By apm on Jun 19, 2007

   For me, this comes down to approach. If you are using PCI DSS 1.1 as something to work up to then you are going to produce an environment which may be more secure than it was before compliance was achieved, but may not be as secure as it could be.

   Approach PCI DSS compliance as a starting point AND take it in the manner that it was intended, as a measurement of “minimum” standard achievement and you’re going to produce a much better environment.

   This is the old “compliance != security” issue which has been posted about before on this site.

   I agree that taking an individual element of PCI DSS requirements in isolation means that on face value, it may be lacking in some aspect. Even if this is only because we live in an ever changing world with ever changing threats and vulnerabilities.

   However, look at PCI DSS (and other compliance standards) from the viewpoint of “how can this _help_ us to improve” and not “what does this tell us to do” and you’re on a winner.

   Of course, most compliance standards and regulations worth the paper they are written on will require a qualified approach to Risk and Risk Management. PCI DSS does this so at the very least, providing it is used in the correct manner, this should allow identification of any omissions and appropriate treatment of same.

   PCI DSS isn’t perfect, but then neither is the approach taken by many companies to become compliant.

   “What’s the quickest, cheapest, easiest way to tick the box” is a bad approach, in my opinion. Unfortunately, so many organisations are doing exactly that.

4. By DAG on Jun 20, 2007

   There was an article with the last year, about companies not only trying to find the cheapest ASV but trying to find the one that results in the easiest pass.

   Anyone have a link?

5. By DAG on Jun 20, 2007

   @chitchcock

   1 it sounds like we’re in agreement

   2 I did not mean to suggest web app scanning is impotent. I’m just advocating that web app testing is not yet ready to fully get integrated with ASV like scanning solutions. The annual web app test recognizes this. If anything the two will grow closer together over time.

   3 Perhaps my point was not clear. The case I mentioned does not scale to all merchants getting scanned automatically. There are better approaches. In the case I mentioned, a short circuit test would have dramatically shortened the test because the sight should have failed on web app hardening due to a huge amount (> 100,000 pages) of sample code. A lot more work needs to be done with these tools. BTW, one of the network scanners finds this root cause. When one tool follows another and some analysis, human analysis, is conducted it works. Throwing it into a monster tool and throwing the switch. Well we just aren’t there yet.

   4/5 I’m not speaking of the complexity of the spider, or the test generator, or even the tool. I am talking of the complexities in setting it up, training it, dealing with all the variables that a custom web application presents, dealing with change. Again, the tools just require too much handholding to be effectively applied automatically en masse. Not yet ready.

   6. Remote scanning is not dealing with things directly. It’s very indirect and error prone. It has improved dramatically since it began. However, there are more effective and direct ways to test security. Wietse Venema, of SATAN fame, once said that “the best way to test a dark room for light leaks was not from the outside in bright daylight.”

   My point, the two technologies are not yet ready to be glued together in their entirety. There are slight differences in their application. And they are growing towards each other. In the meantime there are two requirements in PCI that TOGETHER address the point.

6. By Rob Newby on Jul 4, 2007

   It’s a classic mistake that even people with huge brains and great credentials often make. The point is, PCI is a business issue, OWASP is there for the techies. And whereas I’m not saying “never the twain shall meet”, there really is no comparison to be made.
   I think this merely suited Jeremiah’s purposes for the blog. It’s a shame to knock PCI though, what would be better, PCI or no PCI?

7. By DAG on Jul 5, 2007

   As a general rule, comparing hacking and certifications is a muggs game.

   A certification is someones minimum and fixed (or slowly evolving) standard. Posting a certification on a web site can be like hanging a kick me sign on your corporate assets.

   Hackers are like water looking for a leak. Water with a purpose and intent. They change their tactics. They go out of bounds. They adapt.

   In the long run NO CERTIFICATION or standard makes a web site unhackable. However, they do make them safer for a time. Continuous processes that include updating those certifications and standards make things much safer for much longer.

   Just like water finds leaks, the hackers will try softer targets, go arounds, end-points, corporate workstations, customers.

8. By Jeremiah Grossman on Jul 14, 2007

   Hi everyone, I appreciate all the feedback, which I happened upon a little late. Rather than responding point by point, I figured to stick the high-level concepts, sorry if it’s a little long winded.

   As we know, the goal of PCI-DSS is to protect cardholder information and in doing so make websites more secure. Probably like everyone here I’ve spent a lot of time studying the PCI-DSS, from version to version, and the truth is I like it. A lot in fact and have said so many times. What I don’t like is the validation of compliance implemtation, specifically the ASV certification process, because its WAY behind what the standard requires. This is what I’m saying needs to be improved in order to overcome the compliance != security checkbox game. My point is if Qualys/ScanAlert/Nessus/etc scans (as good as they may be at network VA) can vouch for the web application security requirements of PCI-DSS, that’s sad. This defeats the purpose of ensuring a website or cardholder information is “secure”.

   Sure, scanning custom web application vulnerabilities is tough, no doubt about it. I know this well because I’ve been building the technology to do so for almost a decade and have been in the webappsec field even longer. And yes, many scanners on the market suck. Scanners have been known to suck at crawling, suck at vulnerability identification, suck at false-positives, suck at scale, suck at process, and consume more assessment time than they save. I’ve blogged about these challenges often, but also explained why scanning is essential to the assessment process. Point #2 is just because many scanners suck, or the ones you’ve tried, it doesn’t mean all the vendors do and their technology/process hasn’t matured.

   Speaking for WhiteHat, our Sentinel Service (SaaS) is fully equipped to handle the size and complexities of the world’s largest websites (lots of them), verify all vulnerabilities, complete a comprehensive assessment process with qualified personal, on an ongoing basis, for less annually than the average one-time pen-test, which is above and beyond what the ASV standard SHOULD be. Again, I know this because we’ve been doing so for years for many of the biggest merchants on the Web. A number of which are also “PCI Compliant” through checkbox ASVs, but still get web application VA through us because they value REAL website security. Point #3: If merchants already raised their own webappsec bar voluntarily, then perhaps the PCI Council should be able to as well.

   I’m also under no illusions that the PCI Council has any plans to strengthen the ASV test to resemble anything like the best practices requirements they dictate in the PCI-DSS. Nor would I expect them to do so without ensuring there is an adequate number of vendors capable of providing the service. But consider this, a large amount (majority?) of ASV passed their test using either Qualys or ScanAlert, so in essence the PCI Council is already reliant upon a small number of scanning vendors to handle the load. In any event, my crystal ball says the PCI Council will continue setting their webappsec bar to whatever Qualys and ScanAlert is capable of. Which means waiting for the end of 2007 when their technology reaches where web application vulnerability scanners were back in 2002. Remember when they didn’t even log-in?

Sorry, comments for this entry are closed at this time.