Society of Payment Security Professionals - Payment Security Blog

Lying with statistics

This release from EMC/RSA makes compelling reading, but needs some careful analysis. (Please bear in mind I am not knocking RSA here, some of my best friends are algorithms. I think that Messrs Rivest, Shamir and Adleman would want this to be analysed in a logical way however.)

There is a roughly 50/50 split between compliant and non-compliant respondents, not a random selection of retailers from across the spectrum. This implies a bias towards compliance before we’ve even read the results. There is reporting of these statistics as “RSA says 52% of retailers are still non-compliant”. That’s 52% of 80 people (42 people), not organisations, who visited the RSA website over 4 days earlier this year. RSA are quite clear to point this out, it is subsequent reporting of these figures that has been skewed. I have read at least 3 reports today stating this as gospel truth without quoting the following disclaimer from RSA:

“The RSA PCI DSS survey was conducted between February 26 and March 1, 2007. This Web-based survey polled organizations in the United States and Canada affected by the PCI standard, and individuals from eighty companies participated.” - from page linked above.

1. People looking at the RSA website are generally going to be more concerned about security in the first place.

2. People who do not know or care about PCI are generally not going to fill in a PCI questionnaire.

3. 80 people is not a representative slice of merchants, 4 days is not a very long time to collect data. 

4. People lie on web-based anonymous surveys.

I normally cover PCI in Europe, so I’m not up to date with current US and Canadian compliance figures, I am sure one of my co-writers can fill this in or post below to give me the exact figures. I would guess at this being out of line by quite some margin however. 

“RSA’s research shows that the majority of merchants approach the PCI DSS as an opportunity to protect their brand and their consumers, rather than as an opportunity to mitigate legal exposure.”

The majority of compliant merchants, that is, (an already skewed figure) presumably the non-compliant merchants didn’t fill this part of the survey in as they aren’t bothered about PCI, so we’re down to a sample of 38 people now. 36% (as the survey goes on to state) of 38 is roughly 14 people, who like to look after their customers. And who wouldn’t say that in a questionnaire? Especially if you’re the kind of people-pleaser who likes to fill in another company’s market research. However, that isn’t what bothers me quite so much as this:

“One-fourth of respondents, however, were motivated because PCI DSS is a requirement.”

A quarter of compliant merchants are still complying because they have to! This is not good news, especially when they were given the option of saying “it’s because we love our customers”. This suggests to me a real ignorance of PCI which isn’t just inadvisable in terms of fines and losing customer data, but an absolutely insane business practice.

PCI is there to help you and your business. Don’t just toe the line, or I can guarantee you will still be at the very least exposed, and it will end up costing you more. Compliance is a business issue. Treat it as one and it will help you as much as it seems to hinder you at first glance.

However, it could be that these 9 people were just told to fill in the questionnaire?