What should a Penetration Test include?
November 13th, 2006 by datasecurity Posted in PCI DSS, Third-PartiesSome people have asked (and others added to the confusion) about what is required by PCI DSS regarding requirement 11.3 requiring an annual penetration test. Here are some answers to those questions?
Who?
The requirement does not specify who must perform the penetration test, but much like other requirements it can be done by virtually any party. The penetration test is typically performed by a third-party with experience in offering these services. There is NO requirement that it must be performed by the QSA or ASV. In fact, it can be performed by the entity themselves as long as they have the necessary skills and cover the appropriate areas (but this is rare.)
The responsibility of the QSA or auditor is to make sure the scope (both depth and breath) is appropriate, the methodology is in line with industry best practices, and that all high and medium risk vulnerabilities have been remediated and re-checked.
What type?
People ask what needs to be included in the penetration test and some even say that it should include things such as: war-dialing, physical security testing, social engineering (pretexting), and many other things.
PCI DSS v1.1 clearly states that it should address the following areas:
- Network-layer penetration tests
- Application-layer penetration tests
This means the testing should address all electronic network attacks and electronic application attacks of public facing applications. It does NOT include physical security testing or social engineering (pretexting). One thing it should include is war-dailing (checking modems) because these are methods of electronic remote access.
When?
The requirements call for an annual penetration test. This means that a report on compliance (ROC) cannot be submitted until the testing has been performed, vulnerabilities remediated, and re-checked to make sure they are no longer an issue.
It is not acceptable to include in the report that the penetration test “will be completed within one year.” It needs to be performed and the results addressed before submission of the report.
![[del.icio.us]](http://images.del.icio.us/static/img/delicious.small.gif){kind=small}
11 Responses to “What should a Penetration Test include?”
By Vinod on Feb 7, 2007
Could you please provide details on ‘Application-layer penetration tests’ with examples. I need to assess whether we do the same at our end or not.
By Daniel on Feb 7, 2007
We have written a guide to application layer penetration testing and this can be found at http://www.penetration-testing.com/
The problem with ver 1.1 is that it doesn’t go into depth about what should be checked for and the automated scanning tools DO NOT do the same type of testing as experience application level testers do.
This is a grey area of the PCI imho
By datasecurity on Feb 8, 2007
I would also investigate the OSSTM (Open Source Security Testing Methodology) Manual.
By datasecurity on Feb 9, 2007
Also, there is plenty of information on application penetration testing available on Google
By Roshen Chandran on Feb 27, 2007
You might also want to check out the Plynt Application Security Certification criteria.
http://www.plynt.com/criteria/
All of them can (should) be tested via an application penetration test.
Disclaimer: I was part of the team that designed the Plynt Certification criteria.
By Adam Muntner on Mar 17, 2007
Vinod,
Though the PCIco has ruled that automated application scanning solutions, if used by a properly trained personnel at the merchant, are sufficient, I can tell you that they are really not if you’re interested in security as opposed to filling in the compliance checkbox. It all comes down to, how valuable is your business? Your reputation? How valuable is the data you store?
Since Michael’s excellent post has turned into a services spamtrap, I figure I might as well join the fray and plug my company, but try to dispense some wisdom along with the blatant self-promotion.
http://www.quietmove.com
Running automated tools does not a penetration test make, whether in the application or network/system sphere. Don’t take that the wrong way - I’m not against automated tools. They are repeatable, they provide a good starting point, and they catch a lot of low hanging fruit. However, they also don’t catch a lot of other low hanging fruit.
On engagements, often we are able to take 2-3 “low” findings from an automated scanner and turn them into a compromised system or serious application information leakage issue with a little creative hackery. This is because automated scanners don’t measure risks or threats - they arbitrarily rank vulnerabilities without understanding the value of the resource or the likelihood of success or the difficulty of perpetrating the attack. Us humans like to use things like DREAD for that. http://msdn2.microsoft.com/en-gb/library/aa302419.aspx
Scarier than the false positive is the false negative. Automated tools have their limitations, and there are many classes of vulnerabilities that they can’t identify, both in the application and network/server space. It takes experienced hands-on testers to identify them.
That is our area of expertise - using automated tools to get the greatest amount of value in the shortest amount of time, but understanding where their blind spots are, and testing for them. Though we are a PCI ASV, many of our customers use a commodity scanning service but then engage us for ongoing penetration testing. We work with them to design the test appropriate to their budget and value of their information assets, support them through their remediation process, and provide ongoing metrics of their improvement.
If you have security or network staff who say they can do the penetration test on their own, keep in mind there’s a lot of value in hiring a 3rd party assessor. Mature methodology, experience from testing environment after environment, having a fresh set of eyes, a better tool set, and the fact that it’s probably not the most highly valued use of your employees time. It will get done faster, more thoroughly, and with greater quality by using a 3rd party penetration testing service.
I have a passion for this stuff and love to chat about it even if you’re not a prospective customer! You can communicate with me at adam.muntner (at) quietmove.com, on the PCI Answers forum http://forum.pcianswers.com, or on the Security Catalyst forum. http://community.securitycatalyst.com/, I am also a guest contributing blogger to the http://www.securitycatalyst.com/ blog run by Michael Santarcangelo.
By Atul Changela on Apr 14, 2008
If you don’t have an payment application do you still need to perform an application penetration test? What would that be testing?
By Standard Penetration Test on Jun 18, 2008
Plynt are standard penetration test specialists and network security analysts who service companies worldwide.
By Michael Dahn on Jun 30, 2008
There are many such companies that can perform a penetration test for you. You get what you pay for sometimes. Interview your prospective consultant before hiring anyone.