The test network consists of four such routers using their factory omnidirectional antennas (click on the image to enlarge and see a photo of each node).
They are located at aproximately 50m distance from each other in a zigzag line. In this semi-rural setting all wifi 2.4Ghz noise comes from our community network and the 5Ghz band is completely clean. A typical scan in this particular area only shows two of our nodes with a signal strength of -78/-80.
Vegetation purposely serves as a partial barrier between the nodes. The originators table from each router shows that they are seeing each other the way we expected; that is: every router chooses it’s nearest neighbors as next hop in each direction and they are linked on 2.4Ghz and 5Ghz simultaneously. The multicast rate is set to 54000 and tx power is set to 9 on every node. In this particular setup, those values gave the cleanest results while maintaining good overall performance.
So now to the console dump and then some conclusions. Iperf output has been truncated to shorten the resulting text of so many tests. Each test was actually run a number of times and the outputs published are those we deemed more representative.
ping across the test network
The initial ping test is run from a computer with a wired connection to node_a.
$ sudo ping node_d -i 0.1 -c 100
--- node_d ping statistics ---
100 packets transmitted, 100 received, 0% packet loss, time 9966ms
rtt min/avg/max/mdev = 2.762/6.951/62.385/7.231 ms
There’s no packet loss in the path to node_d and delay is low.
originators table from each node
As explained in the introduction, the tables show that the test nodes are routing traffic as expected.
These tables are dynamic by nature. It’s interesting to note that when the network has no traffic nearly all nodes choose the 5Ghz links, but when traffic starts to move they’ll start shifting between 2.4 and 5.
root@node_a:~# batctl o
[B.A.T.M.A.N. adv 2012.4.0, MainIF/MAC: wlan0-2/66:70:02:ed:f9:81 (bat0)]
Originator last-seen (#/255) Nexthop [outgoingIF]: Potential nexthops ...
node_d_2.4Ghz 0.340s (198) node_b_5Ghz [ wlan1]: node_b_5Ghz (198) node_b_2.4Ghz (182)
node_b_5Ghz 0.420s (255) node_b_5Ghz [ wlan1]: node_b_5Ghz (255)
node_b_2.4Ghz 0.530s (255) node_b_5Ghz [ wlan1]: node_b_5Ghz (255) node_b_2.4Ghz (234)
node_c_5Ghz 0.440s (225) node_b_5Ghz [ wlan1]: node_b_5Ghz (225)
node_c_2.4Ghz 1.820s (222) node_b_5Ghz [ wlan1]: node_b_5Ghz (222) node_b_2.4Ghz (209)
root@node_b:~# batctl o
[B.A.T.M.A.N. adv 2012.4.0, MainIF/MAC: wlan0-2/66:70:02:ee:02:e9 (bat0)]
Originator last-seen (#/255) Nexthop [outgoingIF]: Potential nexthops ...
node_d_2.4Ghz 0.520s (221) node_c_5Ghz [ wlan1]: node_c_5Ghz (221) node_c_2.4Ghz (206)
node_a_2.4Ghz 0.800s (251) node_a_5Ghz [ wlan1]: node_a_5Ghz (251) node_a_2.4Ghz (235)
node_d_5Ghz 0.740s (225) node_c_5Ghz [ wlan1]: node_c_5Ghz (225)
node_c_5Ghz 0.630s (255) node_c_5Ghz [ wlan1]: node_c_5Ghz (255)
node_a_5Ghz 0.920s (251) node_a_5Ghz [ wlan1]: node_a_5Ghz (251)
node_c_2.4Ghz 0.950s (255) node_c_5Ghz [ wlan1]: node_c_5Ghz (255) node_c_2.4Ghz (242)
root@node_c:~# batctl o
[B.A.T.M.A.N. adv 2012.4.0, MainIF/MAC: wlan0-2/66:70:02:ee:01:b7 (bat0)]
Originator last-seen (#/255) Nexthop [outgoingIF]: Potential nexthops ...
node_d_2.4Ghz 0.790s (255) node_d_2.4Ghz [ wlan0-2]: node_d_5Ghz (251) node_d_2.4Ghz (255)
node_a_2.4Ghz 0.750s (217) node_b_5Ghz [ wlan1]: node_b_5Ghz (217) node_b_2.4Ghz (213)
node_b_5Ghz 0.600s (251) node_b_5Ghz [ wlan1]: node_b_5Ghz (251)
node_d_5Ghz 0.900s (255) node_d_5Ghz [ wlan1]: node_d_5Ghz (255)
node_b_2.4Ghz 0.630s (251) node_b_5Ghz [ wlan1]: node_b_5Ghz (251) node_b_2.4Ghz (242)
node_a_5Ghz 0.820s (221) node_b_5Ghz [ wlan1]: node_b_5Ghz (221)
root@node_d:~# batctl o
[B.A.T.M.A.N. adv 2012.4.0, MainIF/MAC: wlan0-2/66:70:02:ed:f8:dd (bat0)]
Originator last-seen (#/255) Nexthop [outgoingIF]: Potential nexthops ...
node_a_2.4Ghz 0.170s (192) node_c_5Ghz [ wlan1]: node_c_5Ghz (192) node_c_2.4Ghz (169)
node_b_5Ghz 0.060s (221) node_c_5Ghz [ wlan1]: node_c_5Ghz (221)
node_b_2.4Ghz 0.160s (221) node_c_5Ghz [ wlan1]: node_c_5Ghz (221) node_c_2.4Ghz (192)
node_c_5Ghz 0.330s (255) node_c_5Ghz [ wlan1]: node_c_5Ghz (255)
node_c_2.4Ghz 0.670s (255) node_c_5Ghz [ wlan1]: node_c_5Ghz (255) node_c_2.4Ghz (221)
iperf along the path from node_a to node_d
root@node_a:~# iperf -c node_b
[ 3] 0.0-10.0 sec 75.5 MBytes 63.1 Mbits/sec
root@node_b:~# iperf -c node_c
[ 3] 0.0-10.0 sec 69.5 MBytes 58.3 Mbits/sec
root@node_c:~# iperf -c node_d
[ 3] 0.0-10.0 sec 77.0 MBytes 64.5 Mbits/sec
iperf from node_a to node_c and node_d
root@node_a:~# iperf -c node_c
[ 3] 0.0-10.1 sec 46.5 MBytes 38.7 Mbits/sec
root@node_a:~# iperf -c node_d
[ 3] 0.0-10.1 sec 39.0 MBytes 32.4 Mbits/sec
independent iperf on 2.4 and 5Ghz links between nodes
Every node in this network is linked to it’s neighbors through 2.4Ghz and 5Ghz, so in this test we intend to find out what’s the actual performance of each link along our tested path (from node_a to node_d).
wlan0-2 is 2.4Ghz and wlan1 is 5Ghz in every node. We force the traffic through each interface using local IPv6 addresses.
root@node_a:~# export node_b_24_v6=fe80::6470:2ff:feee:2e9
root@node_a:~# iperf -V -c $node_b_24_v6%wlan0-2
[ 3] 0.0-10.0 sec 61.9 MBytes 51.8 Mbits/sec
root@node_a:~# export node_b_5_v6=fe80::6670:2ff:feee:2e8
root@node_a:~# iperf -V -c $node_b_5_v6%wlan1
[ 3] 0.0-10.0 sec 70.8 MBytes 59.2 Mbits/sec
root@node_b:~# export node_c_24_v6=fe80::6470:2ff:feee:1b7
root@node_b:~# iperf -V -c $node_c_24_v6%wlan0-2
[ 3] 0.0-10.0 sec 49.4 MBytes 41.3 Mbits/sec
root@node_b:~# export node_c_5_v6=fe80::6670:2ff:feee:1b6
root@node_b:~# iperf -V -c $node_c_5_v6%wlan1
[ 3] 0.0-10.0 sec 61.8 MBytes 51.7 Mbits/sec
root@node_c:~# export node_d_24_v6=fe80::6470:2ff:feed:f8dd
root@node_c:~# iperf -V -c $node_d_24_v6%wlan0-2
[ 3] 0.0-10.0 sec 56.1 MBytes 47.0 Mbits/sec
root@node_c:~# export node_d_5_v6=fe80::6670:2ff:feed:f8dc
root@node_c:~# iperf -V -c $node_d_5_v6%wlan1
[ 3] 0.0-10.0 sec 72.3 MBytes 60.5 Mbits/sec
Conclusions
The slowest link in the path is the 2.4Ghz link between node_b and node_c at 41.3Mbit/s and the next slowest is the contiguous 2.4Ghz link from node_c to node_d at 47Mbit/s. So even if our routing protocol is alternating correctly between available paths, an iperf from end to end will always have a 47 or 41.3Mbit/s bottleneck. As a result, the path from node_a to node_c at a rate of 32.4Mbit/s is suffering roughly 25% throughput degradation. It’s also worth mentioning that comparing the results of running iperf from node_a to node_c with running it from node_a to node_d, thus adding only one hop in the path, there’s a 6Mbit/s drop in throughput, which is consistent with the degradation observed for the complete path.
In a single radio mesh, the theoretical degradation is of 50% per hop, so in our setup, the initial 59.2Mbit/s (node_a_5Ghz – node_b_5Ghz) would have been degraded to aproximately 15Mbit/s. At 2.4Ghz the initial 51.8Mbit/s would have been degraded to 13Mbit/s. Comparing this optimistic theoretical single-radio values to the real world multi-radio experiment conducted here, we can assure that multi-radio/multi-band mesh networking has an enormous potential, even on cheap off-the-shelf routers.
Future tests could try to establish optimal links in both 2.4Ghz and 5Ghz between neighboring nodes to test the real end to end performance limit of such a network. It would also be interesting to establish if the observed ~6Mbit/s per hop degradation observed in this experiment is an actual limit of such networks or not.
For now, this series of tests is enough to prove that multi-radio mesh networks do not suffer from the halved bandwidth per hop degradation of single-radio setups.
Coda: udp iperf from node_a to node_d
We know the weakest link is 41Mbit/s, so we send a 40Mbit/s iperf from node_a to node_d, and the result reported by the server side at node_d is 38.5Mbit/s.
root@node_a:/tmp# iperf -c node_d -b 40M -u
WARNING: option -b implies udp testing
------------------------------------------------------------
Client connecting to node_d, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 160 KByte (default)
------------------------------------------------------------
[ 3] local 10.5.0.24 port 57273 connected with 10.5.0.27 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 47.5 MBytes 39.9 Mbits/sec
[ 3] Sent 33912 datagrams
[ 3] Server Report:
[ 3] 0.0-10.0 sec 46.0 MBytes 38.5 Mbits/sec 0.437 ms 1104/33913 (3.3%)
