This is part 2 of a recent post: “Smoke and Mirrors; What are they really saying?” In reading an article on the ETA website today I found a statistic that did not seem quite accurate.  It stated that: “PCI Compliant merchants fare better, but 36% still report breaches over 2-year span.” In reviewing the article the statistic was taken from the The 2011 PCI DSS Compliance Trends Study was released in April of this year.  After reading the ’study’ I had to admit I was somewhat taken aback.  Two major items stuck out immediately.

1) Respondents ’self reported’ and were “…deemed to be PCI compliant if they chose “all” or “most applications and databases are compliant.” According to the definition 66% of companies were ‘deemed’ compliant.   This definition is inconsistent with the PCI SSC’s, and the card brands’ definition of compliance which is meeting compliance with all application PCI DSS requirements.  Even using the more liberal interpretation, only 33% of the respondents should be considered ‘compliant’.  Additionally, it does not differentiate between when the companies were reported (self, I might add) “compliant” and when the “breach” occurred.  It simply asks if the company was ‘all’ or ‘mostly’ compliant and whether they had a breach.

2) While the study references data breaches numerous times and survey questions Q8a and Q8b ask about data breaches, there is no definition of “data breach” provided in the study.  Is an anti-virus infection considered a data breach?  Is a data breach defined as an intrusion which exposes confidential or protected personal data?  Is encrypted data that is exposed, considered a data breach?  Without a consistent definition, there is no way to understand the intent of the respondent.  It is possible that many respondents feel that a virus infection is a data breach.

Another interesting comment is found on page 1, paragraph 3 of the study.  It states: “In fact, virtually all (99 percent) compliant organizations in this study report that they have had only one or no data breaches involving credit card data compared to 85% of non-complaint organizations that had one or no such breach incidents.” What?!?  Having one breach involving credit card data is a bad thing and requires reporting to the card brands.  It then goes on to state that: “…the percentage of respondents reporting that their organization had a data breach in the past 24 months increased from 79% to 85% in 2011.” These numbers are difficult to understand and more difficult to believe.   So, reading these two statistics together, it appears that 85 of organizations (doesn’t say compliant or non compliant) had a breach in the previous 24 months.  Given that there are 6.5 million merchants in the US, if this represents a statistically valid representative sample, it would suggest that roughly 5.5 million companies experienced a breach in the previous 24 months.  This certainly seems unlikely.  It is more likely that the ’self reported’ breaches include virus infections, web server vulnerabilities and other issues that are not considered ‘data breaches’.

While it is important for companies to understand that protecting data is critical to their company’ success, it does not help to provide information that spins data in an attempt to support a pre-defined position.  When I first began working in data security I learned a new word.  FUD.  FUD Is an acronym for Fear, Uncertainty, and Doubt and is how many companies choose to sell security products.  In essence, you scare the heck out of your clients and they buy the products.  The lesson to be taken from this post is to continue to critically analysis statistics that are provided.  Sometimes a close look will reveal information that is not quite as it seems.