Authenticity and 'Independent' Test Reports

(c) Can Stock Photo

This post is dedicated to every wireless vendor who hires independent labs to 'test' their products against a competitor's product. I'll save you the trouble of reading to the end: The lack of authenticity in these reports is downright insulting.

To me the problem with these reports is really a problem with authenticity. Companies hire an 'independent' lab to perform the test because they want the results to appear unbiased and authentic, yes? On the surface this seems like a smart idea, however, it really only leads down the road to inauthentic reports. Everyone knows you don't bite the hand that feeds you. So why would we ever expect an independent lab to report negative results against that company paying the bills? It's a foolish expectation.

Here is one example of an independent lab report by Miercom. Go ahead and read it, I'll wait...

After reading that report, can you honestly tell me that was unbias and independent? It reads like propaganda more than an unbias lab test. Now, let me pause right here and drop in a little disclaimer: This is just one example lab report and I've linked to it simply because it is the most recent one I've read. My goal is not to single out Cisco or Miercom, nor is it to imply that other companies and labs are not doing the exact same thing. The results in the report might actually be accurate. The problem is, anyone capable of critical thought has no choice but to question the validity of the result based on the way it is being presented.

What's the solution?

I'm not one to complain without at least trying to offer a solution. So, here goes: If you want your lab results to be seen as authentic and genuine, then be authentic and genuine. You need to put your own reputation on the line instead of hiding behind an independent test lab. 

I would have no problem if company A were to publish a lab report stating that they out performed companies B and C, as long as company A stated that they actually did the testing, and that they are willing to stand behind the results. Will there still be bias in the report? Yes, absolutely. The difference is that ownership of the bias is being acknowledged instead of hidden behind the protective cloak of an independent test lab. The scientific community has to behave in this manner. Why shouldn't vendors be held to the same standard and be expected to follow the same methodology.

Closing Thoughts

While I find these types of reports entertaining, I think they are only seen as independent by those people who have already made up their minds and just need an official document to justify their decisions to their bosses. I cannot envision a situation where I would find these reports to be valuable in a decision making process and I implore vendors to rethink this approach.

Daniel

What are your thoughts on the validity and authenticity of the results in independent lab reports? Do you find them useful? Do you think I am correct or way off base? Share your thoughts in the comments section.