Router and Adapter Testing Methodology

With our testing methodology we’ve strived to remove as many variables as possible from the equation and maintain consistency throughout multiple tests.  We’re using two identical Dell Optiplex machines as the “WAN” and “LAN” networks with the following specifications:

Dell Optiplex 745 Desktop PC

Both Dell machines are hooked up to a keyboard, mouse and 20” LCD using a Syba 2-Port USB KVM Switch.  When needed, we use 2 foot CAT 6 network cables from monoprice.com.

19-test-bed-setup-0.jpg

As a baseline to compare both wired and wireless connectivity, we’re using the Fifth Generation Apple Airport Extreme Base Station (A1408).  This little box from Apple has Gigabit WAN (1) and LAN (3) ports as well as 3×3:3 simultaneous dual band 802.11 a/b/g/n wireless that should theoretically be able to achieve speeds of up to 450 Mbps.  Though Apple, unlike most other vendors, is smart enough not to flaunt this number in their marketing, as it’s unlikely end users will be able to achieve it.

20-apple-airport-stock.jpg

To test PCI Express network adapters, there is a pair of open PCI Express slot in each of the Optiplex machines to network cards into.  On the other hand, for USB based adapters, we’ll do all our testing with a laptop.  By using a laptop for USB adapter testing we gain the ability to test not only from our “Lab” where the Optiplex machines are located, but also from other locations to simulate some real use test cases.  For the laptop testing I’m using a MSI laptop with the following specifications: 

MSI EX625-227US Laptop

The MSI EX625 came with a built in 802.11 b/g/n adapter, but we’ve upgraded the internal Wi-Fi adapter to the Centrino Ultimate N as it’s one of the better mini PCI Express adapters on the market that might be found in a laptop.  With 3×3:3 triple streams and advertised speeds up to 450 Mbps it’ll be perfect to use as a baseline to test router Wi-Fi speeds as well comparisons to other USB based Wi-Fi adapters.

Both the Dell Optiplex machines and the MSI EX625 laptops are running Windows 7 Professional that is kept at the same patch level, with all critical updates installed.   Anti-Virus is not running or installed on any of the machines to remove any lag introduced into speed tests from the AV scanning file read/writes.

21-msi-ex625.jpg

Hardware is only half the equation and we still need a way to actually measure network throughput and other important statistics for the various routers and adapters.  Using this data we can consistently compare and contrast them against each other.  While there are simple online speed tests, there are just too many variables that could impact those tests.  For consistency sake, the machines will not be connected to the Internet at all and only networked with the test routers and each other.  There are three pieces of software that I’ve wrapped up in a batch file to give us a good overall picture of network performance.

The first tool we use is a simple DOS based application contributed by Matrix21 over on Smallnetbuilder.com, aptly named, “Max_session_tool.”  The Max Session Tool will open up UDP connections between two hosts until the receiving end starts to drop the packets.  This will tell us the maximum number of network sessions that can be maintained between the LAN and WAN machine and give us an idea of how well the router can handle multiple sessions.  Since this test mainly tests a router’s capability to open and maintain sessions, we will only run this test for the wired tests through a router. 

22-max-sessions-tool.jpg

Next up is the ever useful ping utility test.  Anyone that’s done even basic network troubleshooting will be familiar with the ping command that is used to check connections between two machines.  This simple tool will test the reachability of a host on the network and measure round trip time for messages sent from the host to the destination machine.  With this simple tool we can gather the minimum, maximum and average time needed to send a packet of data between the two machine machines (and through the network router/adapter we are testing), as well as let us know if there are any packets dropped in the process.

23-ping-tool.jpg

Our final “tool” is actually a pair of great applications from Pete over at Totusoft.   We are using Totusoft’s LAN Speed Test (v3) paired with their LST Server software to obtain data throughput speeds between our LAN and WAN machines.  We could have just used LAN Speed Test to run file upload/downloads against a file share between the machines to gather Average, Minimum and Maximum Upload/Download speeds.  However, adding the LST Server into the mix pushes the upload/download test data into the target system’s memory as opposed to hard disk.  By doing this, we remove any impact the hard disk speeds/cache would have on the actual speed values.  While there is a free ‘Lite’ version of the LAN Speed Test available, at only $5 each, both LAN Speed Test and LST server are worth every penny. 

24-lan-speed-tests.jpg

With these three tests we can get a pretty good snapshot of network connectivity and throughput.  Initially I was manually running each test one at a time, but since there is such a large amount of tests that needs to be run, I strung together a batch file to automate each set of tests. 

25-network-testing-batch-1.jpg

The batch file does the following:

  • For routers, it will run the Max Sessions Tool 7 times and document the maximum sessions that can be opened between the LAN and WAN machine before they start being dropped.
  • It will then run 7 tests of 100 pings each from the LAN to the WAN computer and document the Minimum, Maximum and Average ping time in addition to any dropped packets.
  • Finally it will run 7 upload and download tests between the LAN and WAN machines using LAN Speed Test and the LST Server.  Each LAN Speed test uploads and then downloads a 10 MB file 10 times in a row and then calculates Average, Minimum and Maximum Speeds in Mbps that were achieved while uploading and downloading the files.

26-network-testing-batch-2.jpg

Once all the tests have run, we transfer the data from the batch file logs over to a master spreadsheet.  To minimize the impact of any extreme test results we drop the highest and lowest values for each test and then average the remaining ones for our ‘final’ test value for each metric.

27-test-spreadsheet.jpg

Wireless testing requires a few extra test procedures over and above what we’ve already done.  We will always test both 5 GHz and 2.4 GHz networks separately when the router/adapter has the capability to do so.  Additionally, to ensure wireless testing is as consistent as possible across all our tests, we will always attempt to use the following Wireless networking settings on routers/adapters.

  • Security: WPA2 Personal / AES
  • 2.4 GHz Channel: 11
  • 5 GHz Channel: 157
  • Use Wide Channels/Channel Bandwidth: 40 Mhz
  • Multicast Rate: Low

If a wireless adapter has more than one mode, say a “Standard” and a “High Power” mode, we’ll also run all the tests for each mode to see if there is any benefit to using one mode over the other.

Finally, since most people will likely not be sitting right next to their router when using USB based wireless adapters and a laptop, we will run all of our wireless tests in three different locations.  For simplicity’s sake we are calling them the ‘Lab Test’, the ‘Normal Use Test’, and the ‘Torture Test’.   In the ‘Lab Test’ I’ve set up both the router and Wi-Fi adapter in my office approximately 3 feet from each other.  For the second ‘Normal Use Test’ the signal will have to travel half way across the house through two walls and a floor into the Living room about 45 feet away.  Finally, we’ll wrap things up with a ‘Torture Test’ where the wireless signal has to travel 70 feet into my Laundry Room and punch through four walls, a floor, and everything in between.  I don’t know what the builders lined the walls of the Laundry room with, but not only does it lay the smack down on most wireless signals, I think it would be usable as a nuclear bunker. 

28-house-test-setups.jpg

With all the testing methodology spelled out, let’s dive into the actual testing and see how well our test subjects perform.

« PreviousNext »