Authenticity and 'Independent' Test Reports

(c) Can Stock Photo

This post is dedicated to every wireless vendor who hires independent labs to 'test' their products against a competitor's product. I'll save you the trouble of reading to the end: The lack of authenticity in these reports is downright insulting.

To me the problem with these reports is really a problem with authenticity. Companies hire an 'independent' lab to perform the test because they want the results to appear unbiased and authentic, yes? On the surface this seems like a smart idea, however, it really only leads down the road to inauthentic reports. Everyone knows you don't bite the hand that feeds you. So why would we ever expect an independent lab to report negative results against that company paying the bills? It's a foolish expectation.

Here is one example of an independent lab report by Miercom. Go ahead and read it, I'll wait...

After reading that report, can you honestly tell me that was unbias and independent? It reads like propaganda more than an unbias lab test. Now, let me pause right here and drop in a little disclaimer: This is just one example lab report and I've linked to it simply because it is the most recent one I've read. My goal is not to single out Cisco or Miercom, nor is it to imply that other companies and labs are not doing the exact same thing. The results in the report might actually be accurate. The problem is, anyone capable of critical thought has no choice but to question the validity of the result based on the way it is being presented.

What's the solution?

I'm not one to complain without at least trying to offer a solution. So, here goes: If you want your lab results to be seen as authentic and genuine, then be authentic and genuine. You need to put your own reputation on the line instead of hiding behind an independent test lab. 

I would have no problem if company A were to publish a lab report stating that they out performed companies B and C, as long as company A stated that they actually did the testing, and that they are willing to stand behind the results. Will there still be bias in the report? Yes, absolutely. The difference is that ownership of the bias is being acknowledged instead of hidden behind the protective cloak of an independent test lab. The scientific community has to behave in this manner. Why shouldn't vendors be held to the same standard and be expected to follow the same methodology.

Closing Thoughts

While I find these types of reports entertaining, I think they are only seen as independent by those people who have already made up their minds and just need an official document to justify their decisions to their bosses. I cannot envision a situation where I would find these reports to be valuable in a decision making process and I implore vendors to rethink this approach.

Daniel

What are your thoughts on the validity and authenticity of the results in independent lab reports? Do you find them useful? Do you think I am correct or way off base? Share your thoughts in the comments section.

Comments

[Andrew vonNagy]

I whole-heartedly agree with your assessment. The problem lies in the business model of payment for the testing and report. This ensures that the "independent lab" conforms to the test as laid out and designed by the vendor providing payment. It also ensures that any negative results that are found are either redacted or the report never sees the light of day.

There is no easy answer to provide truly independent testing without payment or compensation of some form involved. And testing the increasingly complex nature of Wi-Fi, or any network solution for that matter, makes customer testing all but impossible except for the largest of customers with deep pockets to hire expertise staff and give them proper time to execute.

I believe we need a better way, and am open to suggestions. I have a few of my own. And, full disclosure, since I now work for a Aerohive Networks (a vendor), I am discussing these ideas internally and seeing what makes sense to pursue.

Cheers,
Andrew vonNagy

[Ryan Nyboer]

Great article. I also agree, it is very frustrating to find that each vendor's "independent tests" conveniently always outperform their competition. I believe this is largely thanks to the sponsor company not optimally configuring the competition's system for the test (ie not using that vendor's recommended settings, and deliberately setting up the configuration for failure).

@nyfy

[Daniel]

Thanks for the comments Andrew and @nyfy.

@nyfy Do you have any suggestions or thoughts on how the independent test system can be tweaked to be less bias? I admit that it's a difficult issue so I'm always interested in hearing how other people would fix things.

@Andrew - Good luck with your internal ideas! I hope something good comes out of them and we can all find a solution that works for everyone. Definitely, if I hear of any neat ideas I'll pass them on.