Network Router and Adapter Testing Methodology

Before we dive right into the router and adapter performance, let’s discuss our testing methodology again.  As mentioned in earlier reviews we’ve worked to take as many variables out of the equation and maintain consistency across multiple tests.  We’re using two identical Dell Optiplex machines as the “WAN” and “LAN” networks with the following specifications:

Dell Optiplex 745 Desktop PC

Both Dell machines are hooked up to a keyboard, mouse and 20” LCD using a Syba 2-Port USB KVM Switch.  When needed, we use 2 foot CAT 6 network cables from monoprice.com.

As a baseline to compare both wired and wireless connectivity, we’re using one of the better routers out there, the Fifth Generation Apple Airport Extreme Base Station (A1408).  This little box from Apple has Gigabit WAN (1) and LAN (3) ports as well as 3×3:3 simultaneous dual band 802.11 a/b/g/n wireless that should theoretically be able to achieve speeds of up to 450 Mbps.  Though Apple, unlike most other vendors, is smart enough not to tout this number, as it’s unlikely end users will be able to achieve it.

To test PCI Express network adapters, we do have an open PCI Express slot in each of the Optiplex machines that are available.  For USB based adapters, we’ll do testing with a laptop.  Using a laptop for USB adapter testing will give us the option to not only test from our “Lab” where the Optiplex machines are located, but will allow us to test from other locations to simulate some real use test cases as well.  For this testing I’m using an MSI laptop with the following specifications: 

MSI EX625-227US Laptop

The MSI EX625 came with a built in 802.11 b/g/n adapter, but we’ve upgraded the internal Wi-Fi adapter to the Centrino Ultimate N as it’s one of the better like one of the better mini PCI Express adapters on the market that might be found in a laptop.  With 3×3:3 triple streams and advertised speeds up to 450 Mbps it’ll be perfect to use as a baseline to test router Wi-Fi speeds as well comparisons to other USB based Wi-Fi adapters.

Hardware is only half the equation though and we still need a way to actually measure network throughput and other stats of the routers and adapters so we can compare and contrast them against each other with some sort of consistency.  There are simple online speed tests, but there’s just too much variance if we use that.  For consistency sake, the machines will not be connected to the Internet at all and only networked with the test routers and each other.  We’ll then use three pieces of software that I’ve wrapped up in a script to give us a good overall picture of network performance.

The first tool we use is a simple DOS based application contributed by Matrix21 over on Smallnetbuilder.com, aptly named, “Max_session_tool.”  The Max Session Tool will open up UDP connections between two hosts until the receiving end starts to drop the packets.  This will tell us the maximum number of network sessions that can be maintained between the LAN and WAN machine ang give us an idea of how well the router can handle multiple sessions.  Since this test mainly tests a router’s capability to open and maintain sessions, we will only run this test for the wired tests through a router. 

Next up is the ever useful ping test.  Anyone that’s done even basic network troubleshooting will be familiar with the ping command that is used to check connections between two machines.  This simple tool will test the reachability of a host on the network and measure rout trip time for messages sent from the host to the destination machine.  With this simple tool we can gather tthe minimum, maximum and average time needed to send a packet of data between machines, as well as let us know if there are any dropped packets. 

Our final tool is actually a a pair of great tools from Pete over at Totusoft.   Using Totusoft’s LAN Speed Test (v3) paired with the LST Server software we can get data throughput speeds between two machines.  With LAN Speed Test, you can run file upload/downloads against a file share on another machine to give you Average, Minimum and Maximum Upload/Download speeds.  However, adding LST Server into the mix lets you upload/download the data into the target system’s memory as opposed to hard disk.  This way you remove any impact the hard disk speeds/cache would have on the actual speed values.  While there is a free ‘Lite’ version of the LAN Speed Test available, at only $5 each, both LAN Speed Test and LST server are worth every penny and then some. 

With these three tools we can get a pretty good snapshot of network connectivity and throughput.  Initially I was manually running each test after each other, but as there’s such a large amount of tests that needs to be run, I strung together a batch file to automate each set of tests. 

The batch file does the following:

  • For routers, it will run the Max Sessions Tool 7 times and document the maximum sessions between the LAN and WAN machine for each run.
  • It will then run 7 tests of 100 pings each from the LAN to the WAN computer and document the Minimum, Maximum and Average ping time as well as any dropped packets.
  • Finally it will run 7 upload and download tests between the LAN and WAN using LAN Speed Test and LST Server.  The LAN speed test will upload and download a 10 MB file 10 times for each run and then document Average, Minimum and Maximum Speeds in Mbps for both uploading and downloading.

Once all the tests have run, we transfer the data from the batch file logs over to a master spreadsheet.  To minimize the impact of any extreme test results we drop the highest and lowest values for each test and then average the remaining ones for our ‘final’ test value for each metric.

We do have a few extra test procedures involved for wireless testing.  We will always test both 5 GHz and 2.4 GHz networks separately if they are available.  To ensure wireless testing is as consistent as possible across all our tests, we will always attempt to use the following Wireless networking settings on routers/adapters.

  • Security: WPA2 Personal / AES
  • 2.4 GHz Channel: 11
  • 5 GHz Channel: 157
  • Use Wide Channels/Channel Bandwidth: 40 Mhz
  • Multicast Rate: Low

If a wireless adapter has more than one mode, say a “Standard” and a “High Power” mode, we’ll also run all the tests for each mode to see if there is any benefit to using one mode over the other.

Finally, since most people are not going to be sitting right next to their router when using USB based wireless adapters and a laptop, we will run all of our wireless tests in three different locations.  For simplicity’s sake we are calling them the ‘Lab Test’, the ‘Normal Use Test’, and the ‘Torture Test’.   In the ‘Lab Test’ I’ve set up both the router and Wi-Fi adapter in my office approximately 3 feet from each other.  For the second ‘Normal Use Test’ the signal will have to travel half way across the house through two walls and one floor into the Living room about 45 feet away.  Finally, we’ll wrap things up with a ‘Torture Test’ where the wireless signal has to travel 70 feet into my Laundry Room and punch through four walls, a floor, and everything in between.  I don’t know what the builders lined the walls of the Laundry room with, but not only does it lay the smack down on most wireless signals, but I think I can use it as a nuclear bunker. 

With that said, let’s dive into the actual testing and see how well everything performed!

« PreviousNext »