Search Results for “smoke”.


This is part 2 of a recent post: “Smoke and Mirrors; What are they really saying?” In reading an article on the ETA website today I found a statistic that did not seem quite accurate.  It stated that: “PCI Compliant merchants fare better, but 36% still report breaches over 2-year span.” In reviewing the article the statistic was taken from the The 2011 PCI DSS Compliance Trends Study was released in April of this year.  After reading the ’study’ I had to admit I was somewhat taken aback.  Two major items stuck out immediately.

1) Respondents ’self reported’ and were “…deemed to be PCI compliant if they chose “all” or “most applications and databases are compliant.” According to the definition 66% of companies were ‘deemed’ compliant.   This definition is inconsistent with the PCI SSC’s, and the card brands’ definition of compliance which is meeting compliance with all application PCI DSS requirements.  Even using the more liberal interpretation, only 33% of the respondents should be considered ‘compliant’.  Additionally, it does not differentiate between when the companies were reported (self, I might add) “compliant” and when the “breach” occurred.  It simply asks if the company was ‘all’ or ‘mostly’ compliant and whether they had a breach.

2) While the study references data breaches numerous times and survey questions Q8a and Q8b ask about data breaches, there is no definition of “data breach” provided in the study.  Is an anti-virus infection considered a data breach?  Is a data breach defined as an intrusion which exposes confidential or protected personal data?  Is encrypted data that is exposed, considered a data breach?  Without a consistent definition, there is no way to understand the intent of the respondent.  It is possible that many respondents feel that a virus infection is a data breach.

Another interesting comment is found on page 1, paragraph 3 of the study.  It states: “In fact, virtually all (99 percent) compliant organizations in this study report that they have had only one or no data breaches involving credit card data compared to 85% of non-complaint organizations that had one or no such breach incidents.” What?!?  Having one breach involving credit card data is a bad thing and requires reporting to the card brands.  It then goes on to state that: “…the percentage of respondents reporting that their organization had a data breach in the past 24 months increased from 79% to 85% in 2011.” These numbers are difficult to understand and more difficult to believe.   So, reading these two statistics together, it appears that 85 of organizations (doesn’t say compliant or non compliant) had a breach in the previous 24 months.  Given that there are 6.5 million merchants in the US, if this represents a statistically valid representative sample, it would suggest that roughly 5.5 million companies experienced a breach in the previous 24 months.  This certainly seems unlikely.  It is more likely that the ’self reported’ breaches include virus infections, web server vulnerabilities and other issues that are not considered ‘data breaches’.

While it is important for companies to understand that protecting data is critical to their company’ success, it does not help to provide information that spins data in an attempt to support a pre-defined position.  When I first began working in data security I learned a new word.  FUD.  FUD Is an acronym for Fear, Uncertainty, and Doubt and is how many companies choose to sell security products.  In essence, you scare the heck out of your clients and they buy the products.  The lesson to be taken from this post is to continue to critically analysis statistics that are provided.  Sometimes a close look will reveal information that is not quite as it seems.

Yesterday while at the supermarket I played one of my favorite marketing games and looked on the back of a can of supplements to see if any interesting statements or disclaimsers were used.  The number of qualifiers and disclaimers were incredible yet the language was clearly crafted to convince prospective buyers of the benefits of using the product. The qualifiers are highlighted in bold while the disclaimers are underlined.

“Vitamin D is being investigated for breast health benefits. While no medical consensus yet exists, emerging science suggests there may be a correlation between adequate levels of Vitamin D and breast health. More research is needed.”

In reading a recently published report pertaining to PCI DSS compliance and data breaches I found some interesting language that gave pause.

From these results, it cannot be said that the PCI DSS fails to address the most prevalent threats to cardholder data.”

While it certainly appears that an attorney crafted the language it seems to contradict the purpose of the report. Additionally, inaccurate or incorrect statistics and methodologies are worse than no statistics or methodologies. The methodology section of the report states;

“…QSAs perform hundreds of PCI assessments each year. For several reasons, this report does not include them all. Rather, a simple selection process was used to create a sample of these reports for inclusion in the study…”

To conduct a statistically valid study a statistically valid representative sample should be used. Without a valid sample, the results are spurious, at best. To give credit to the report, the language does provide clarification and a disclaimer stating that: “from these results…”. In short, the report is little more than information presented in a format to look like a formal, statistically valid study. While it provides interesting reading “…it is suggested by the author “…that readers should evaluate whether they want to consider using the information to formulate a recommended strategy for considering security.” The last sentence was an effort to use levity and was intended to highlight the point I was making.

The purpose of this blog post is not to disparage or criticize rather it is intended to point out common tactics used to convince readers of a position when little or no real valid evidence exists. Data security thrives on using numbers, statistics, and implications, and innuendo to sell products and services.

When reading reports, labels, claims, summaries, and statistics read with a critical eye looking for validity, accuracy and relevance. While a report, label, or claim want a reader to infer one message, it may in truth have little basis for such inference. A great reference for learning about statistics and one what many readers may be familiar with is the ever popular “How to Lie with Statistics.” First published in 1954, it is still relevant today.

Chris Mark

In reading throug merchants and service provider’s website, I often take a look at their security statements to see what, if anything, is being said about security.  More often than not these statements are little more than smoke and mirrors written to give consumers and others peace of mind yet don’t really provide any information on the security posture of the company.  In the vast majority of cases the statements are ‘marketing fluff’ and provide little value.  Here are some of the more common and interesting statements I have come across:

-”We use industry leading encryption, including SSL, to protect your data as it is transmitted to us.”  Encrypting transmission of credit card data is not only required by the card brands and the PCI DSS, it is also required by some laws and is simply good practice.  The fact that a company feels compelled to state that they are using SSL to protect transmitted data leads to more questions.  It doesn’t say anything about how your data is used (privacy discussion) or whether the stored data is adequately protected.

-”We use multi-tiered firewall controls to protect sensitive data.” Again, multi-tiered firewall architectures are required by the PCI DSS and being that we are now in the year 2010, operating without a multi-tiered network would be irresponsible at best.  This statement only states that the company has implemented firewalls between various segments of their network.  It does not state anythinga bout whether the devices are configured correctly nor does it differentiate between application layer and network layer firewalls. 

-”All customer data is housed in our secure data centers.” For those unfamiliar with the term, a data center is nothing more than a building that is used to house computer servers.  Data centers are designed with safety, physical security, and redundancy in mind.  The fact that data is housed in a 4th generation data ceneter does not provide any information on the technical security controls implemented to protect customer data.

-”we use robust encryption and change the encryption key at least annually.”  The use of ecryption technology is a good step but encryption is only as good as the algorythms used and the key management employed.  This statement simply says that once again, the company is following PCI controls.  While changing encryption keys periodically is good practice, it doesn’t say anything about how the keys are managed in the intervening periods.

When evaluating a vendor, it is suggested that you take the time to really ask the difficult questions about security.  Simply reading website marketing fluff will not provide you with the assurance that the company is protecting your data.  In some cases the information provided is not simply irrelevant but may provide a false sense of security the the buyer.   If you are not confident in your own technical skills to evaluate a vendor, it is worth taking the time to find a consultant or some other trusted party to support you in your evaluation.

Chris Mark