My First Look at Meraki's Air Marshal

Note: I performed a second review with new hardware and updated firmware. Many of this issues identified in this post have been resolved. 

After playing around with Meraki's new wireless intrusion prevention system (WIPS) called Air Marshal for a few days, I have to say that I am left feeling surprisingly underwhelmed. 

Usually, when Meraki releases a product it is a straight-forward, and elegant solution which is controlled via their amazingly simple administrative interface. Unfortunately, based on my limited testing of Air Marshal, Meraki has missed the mark right out of the gates with this one. Here is how my experience with Air Marshal went:

The Setup

As with most new features, Meraki is rolling out Air Marshal gradually to all existing cloud controllers. I, being impatient as I am, couldn't wait for gradual to happen so I opted for the expedited approach:

Twitter conversation with MerakiI called Meraki technical support and spoke with a very nice gentleman who promptly pushed the feature to my cloud controller. He also informed me that my MR-16 access point would need to receive a firmware update as well, and that he could push the update immediately. We chatted about Air Marshal while my AP was updated and performed its reboot. After that, I thanked him for his time and converted my AP to an Air Marshal sensor by following the instructions posted on the Meraki website.

With my MR-16 happily running as an Air Marshal sensor, I turned thoughts toward giving it some friends to play with. In addition to the 2 SSIDs already being broadcast, I plugged in a Linksys SOHO access point broadcasting 1 SSID, and an Aruba IAP-135 broadcasting 2 SSIDs. This gave me a total of 5 SSIDs being broadcast by 3 access points which were all sitting on the same flat switch as my MR-16. Let the fun begin!

Detecting and Classifying Rogues

This is where my experience started to turn south. To my surprise, my MR-16 didn't pick up all of the SSIDs that were being broadcast by access points that were located 3 meters away.

Detection and classification.As seen in the picture above, Air Marshal detected and classified 7 wireless networks (some are neighbouring APs outside of my control). Unfortunately, of the SSIDs shown in the picture, 4 were incorrectly classified as 'Other' when they should have been classified as 'Rogue'. Those SSIDs are: SimplyWiFi, SOHO-Linksys, NewSchool, and 1 hidden SSID. Causing further concern was the fact that 2 of my SSIDs were not even detected: OldSchool and SimplyWiFi-G. 

I gave Air Marshal the benefit of the doubt and pressed on thinking a little traffic might improve its accuracy. Sadly, connecting to each of my wireless networks and surfing the web didn't improve the classifications. It was while looking at the classifications that I noticed another error, all of the detected networks were tagged as 'unencrypted'. This was inaccurate for all networks shown in the above image. All of my networks where running WPA2-PSK with the exception of the old Linksys AP which was running WPA-PSK. Here is shot showing my SimplyWiFi networks:

Aerohive AP connections showing WPA2-PSK for both SSIDsAgain, still willing to give my beloved MR-16 the benefit of the doubt I logged into my IAP-135 and checked out the IDS logs. It was able to see both of my SimplyWiFi SSIDs just fine even though it was not acting as a dedicated sensor since it was serving up two SSIDs. 

IAP-135 detecting my Aerohive broadcast SimplyWiFi SSIDs.I decided to let Air Marshal do it's thing and move on to containment.

Rogue Containment

You might have noted, in one of the pictures above, that one of the SSIDs was whitelisted and another was marked as 'partially contained'. This was due to my testing of the containment functionality. Once I marked my SOHO-Linksys SSID for containment, I connected to it with my iPod and waited to be unceremoniously disconnected; the disconnection never came. I was able to happily surf and use NetFlix without any issues. I even tried moving to within a meter of the MR-16 in hopes that it could over power the Linksys from such a position. Nothing. 

Intrigued, I fired up my Backtrack 5 R1 laptop and used airodump-ng to capture everything on channel 6 expecting to catch a glimpse of some deauth or disassociation frames. Not a single death or disassociation frame was seen. 

Patience is Rewarded

I'm happy-ish to report that Air Marshal did eventually reclassify my SimplyWiFi SSID as rogue with no intervention by me. Total time to correct classification was about 24 hours.

Air Marshal rogue classification by keyword and automatic.You'll also notice that I tested the keyword classification feature and it worked just fine. I never did managed to get the containment feature to do its magic though. Also, I had the cloud controller configured to send me an email if it detected a rogue but no email was ever received.

Rogue email alert settingClosing Thoughts

 I really did a lot of soul searching before writing this post. Part of me didn't want to post it because it doesn't paint the best picture of Air Marshal and comes off rather negative. However, I decided to post this because I feel it is my obligation to give honest feedback regardless of positive or negative outcome. In this case, the product fell short of expectations but my feedback is not based on negativity or malice. 

I genuinely want to see Air Marshal, and Meraki, succeed. It just seems that perhaps, in this case, a new feature has been rolled out prematurely. I did, after all, call and ask them if the could give me Air Marshal ahead of the typical schedule. I also acknowledge that there could very well be something wrong my with MR-16 (I don't have a second one to test with) or even with my configuration. Stranger things have happened.

Until this is proven, and in the meantime, I believe there is much work that still needs to be done before Air Marshal is ready for primetime. WIPS is one of those solutions that requires a lot of dials and knobs because it needs to be tweaked to fit each different environment. While a simple administrative interface is normally a big plus for Meraki, I feel it hurts them when it comes to Air Marshal. I would love to be able to tweak, adjust, and debug Air Marshal a lot more than I am able to today. Without being able to look under-the-hood I can't really say why Air Marshal performed as poorly as it did during my testing.

Daniel

I openly invite correction or collaboration with Meraki or anyone who has managed to get Air Marshal working as advertised. In fact, I really do hope that we can chalk this one up to user error. I look forward to retesting Air Marshal after some time and posting a more positive review. 

Note: I performed a second review with new hardware and updated firmware. Many of this issues identified in this post have been resolved.

Comments

[Pablo Estrada]

Hi Dan,

Thanks for taking the time to review the new Air Marshal features. We're looking into the issues you've seen, and will be getting back to you shortly.

Thanks,
Pablo

[Pablo Estrada]

Hi Dan (and readers of this blog),

I've discussed your findings with our engineering and product teams, and wanted to write a follow-up to my earlier comment. First, we're sorry that Air Marshal didn't work as expected on your network. We're currently rolling Air Marshal out exclusively to opt-in early adopters like you who explicitly request an upgrade. Unfortunately, in rare cases early adopters see issues based on your environment that we did not experience in our labs, and that our beta testers also did not experience on their networks.

The next scheduled rollout of Air Marshal (also only to early adopters) will be going out in one week, and will include fixes to this issue. We'll be sure to upgrade your network at that time.

I'm sorry your first impression of Air Marshal didn't meet our standards. Our teams appreciate the candid feedback from early adopters like yourself who help us to ensure that complex features like this have broad field testing before being deployed to our mainstream users.

We appreciate your patience, and we hope you'll give Air Marshal another go after next week's update. Again, thanks for your support!

Thanks,
Pablo

[Daniel]

Hi Pablo,

Always a pleasure to hear from you. I've been in touch with your Product Manager and working with your TAC team to find some fixes. Things are starting to look up and I will be posting a follow-up blog post once I have to time to do some more testing and retesting.

Stay tuned!

Dan

[Daniel]

Hi readers,

Please read the follow-up review I conducted at http://www.simplywifi.co/blog/2012/8/6/a-second-look-at-merakis-air-marshal.html

Many of the issues identified have been resolved and were caused by faulty hardware and incorrect firmware versions being pushed to my APs.