Multiple hop mesh performance with multi-band dual-radio routers

The test network consists of four such routers using their factory omnidirectional antennas (click on the image to enlarge and see a photo of each node).

They are located at aproximately 50m distance from each other in a zigzag line. In this semi-rural setting all wifi 2.4Ghz noise comes from our community network and the 5Ghz band is completely clean. A typical scan in this particular area only shows two of our nodes with a signal strength of -78/-80.

Vegetation purposely serves as a partial barrier between the nodes. The originators table from each router shows that they are seeing each other the way we expected; that is: every router chooses it’s nearest neighbors as next hop in each direction and they are linked on 2.4Ghz and 5Ghz simultaneously. The multicast rate is set to 54000 and tx power is set to 9 on every node. In this particular setup, those values gave the cleanest results while maintaining good overall performance.

So now to the console dump and then some conclusions. Iperf output has been truncated to shorten the resulting text of so many tests. Each test was actually run a number of times and the outputs published are those we deemed more representative.

ping across the test network

The initial ping test is run from a computer with a wired connection to node_a.

$ sudo ping node_d -i 0.1 -c 100
--- node_d ping statistics ---
100 packets transmitted, 100 received, 0% packet loss, time 9966ms
rtt min/avg/max/mdev = 2.762/6.951/62.385/7.231 ms

There’s no packet loss in the path to node_d and delay is low.

originators table from each node

As explained in the introduction, the tables show that the test nodes are routing traffic as expected.
These tables are dynamic by nature. It’s interesting to note that when the network has no traffic nearly all nodes choose the 5Ghz links, but when traffic starts to move they’ll start shifting between 2.4 and 5.

root@node_a:~# batctl o
[B.A.T.M.A.N. adv 2012.4.0, MainIF/MAC: wlan0-2/66:70:02:ed:f9:81 (bat0)]
Originator      last-seen (#/255)           Nexthop [outgoingIF]:   Potential nexthops ...
node_d_2.4Ghz    0.340s   (198)       node_b_5Ghz [     wlan1]:       node_b_5Ghz (198)     node_b_2.4Ghz (182)
node_b_5Ghz    0.420s   (255)       node_b_5Ghz [     wlan1]:       node_b_5Ghz (255)
node_b_2.4Ghz    0.530s   (255)       node_b_5Ghz [     wlan1]:       node_b_5Ghz (255)     node_b_2.4Ghz (234)
node_c_5Ghz    0.440s   (225)       node_b_5Ghz [     wlan1]:       node_b_5Ghz (225)
node_c_2.4Ghz    1.820s   (222)       node_b_5Ghz [     wlan1]:       node_b_5Ghz (222)     node_b_2.4Ghz (209)
root@node_b:~# batctl o
[B.A.T.M.A.N. adv 2012.4.0, MainIF/MAC: wlan0-2/66:70:02:ee:02:e9 (bat0)]
Originator      last-seen (#/255)           Nexthop [outgoingIF]:   Potential nexthops ...
node_d_2.4Ghz    0.520s   (221)       node_c_5Ghz [     wlan1]:       node_c_5Ghz (221)     node_c_2.4Ghz (206)
node_a_2.4Ghz    0.800s   (251)       node_a_5Ghz [     wlan1]:       node_a_5Ghz (251)     node_a_2.4Ghz (235)
node_d_5Ghz    0.740s   (225)       node_c_5Ghz [     wlan1]:       node_c_5Ghz (225)
node_c_5Ghz    0.630s   (255)       node_c_5Ghz [     wlan1]:       node_c_5Ghz (255)
node_a_5Ghz    0.920s   (251)       node_a_5Ghz [     wlan1]:       node_a_5Ghz (251)
node_c_2.4Ghz    0.950s   (255)       node_c_5Ghz [     wlan1]:       node_c_5Ghz (255)     node_c_2.4Ghz (242)
root@node_c:~# batctl o
[B.A.T.M.A.N. adv 2012.4.0, MainIF/MAC: wlan0-2/66:70:02:ee:01:b7 (bat0)]
Originator      last-seen (#/255)           Nexthop [outgoingIF]:   Potential nexthops ...
node_d_2.4Ghz    0.790s   (255)     node_d_2.4Ghz [   wlan0-2]:       node_d_5Ghz (251)     node_d_2.4Ghz (255)
node_a_2.4Ghz    0.750s   (217)       node_b_5Ghz [     wlan1]:       node_b_5Ghz (217)     node_b_2.4Ghz (213)
node_b_5Ghz    0.600s   (251)       node_b_5Ghz [     wlan1]:       node_b_5Ghz (251)
node_d_5Ghz    0.900s   (255)       node_d_5Ghz [     wlan1]:       node_d_5Ghz (255)
node_b_2.4Ghz    0.630s   (251)       node_b_5Ghz [     wlan1]:       node_b_5Ghz (251)     node_b_2.4Ghz (242)
node_a_5Ghz    0.820s   (221)       node_b_5Ghz [     wlan1]:       node_b_5Ghz (221)
root@node_d:~# batctl o 
[B.A.T.M.A.N. adv 2012.4.0, MainIF/MAC: wlan0-2/66:70:02:ed:f8:dd (bat0)]
  Originator      last-seen (#/255)           Nexthop [outgoingIF]:   Potential nexthops ...
node_a_2.4Ghz    0.170s   (192)       node_c_5Ghz [     wlan1]:       node_c_5Ghz (192)     node_c_2.4Ghz (169)
node_b_5Ghz    0.060s   (221)       node_c_5Ghz [     wlan1]:       node_c_5Ghz (221)
node_b_2.4Ghz    0.160s   (221)       node_c_5Ghz [     wlan1]:      node_c_5Ghz (221)     node_c_2.4Ghz (192)
node_c_5Ghz    0.330s   (255)       node_c_5Ghz [     wlan1]:       node_c_5Ghz (255)
node_c_2.4Ghz    0.670s   (255)       node_c_5Ghz [     wlan1]:       node_c_5Ghz (255)     node_c_2.4Ghz (221)

iperf along the path from node_a to node_d

root@node_a:~# iperf -c node_b
[  3]  0.0-10.0 sec  75.5 MBytes  63.1 Mbits/sec
root@node_b:~# iperf -c node_c
[  3]  0.0-10.0 sec  69.5 MBytes  58.3 Mbits/sec
root@node_c:~# iperf -c node_d
[  3]  0.0-10.0 sec  77.0 MBytes  64.5 Mbits/sec

iperf from node_a to node_c and node_d

root@node_a:~# iperf -c node_c
[  3]  0.0-10.1 sec  46.5 MBytes  38.7 Mbits/sec
root@node_a:~# iperf -c node_d
[  3]  0.0-10.1 sec  39.0 MBytes  32.4 Mbits/sec

independent iperf on 2.4 and 5Ghz links between nodes

Every node in this network is linked to it’s neighbors through 2.4Ghz and 5Ghz, so in this test we intend to find out what’s the actual performance of each link along our tested path (from node_a to node_d).

wlan0-2 is 2.4Ghz and wlan1 is 5Ghz in every node. We force the traffic through each interface using local IPv6 addresses.

root@node_a:~# export node_b_24_v6=fe80::6470:2ff:feee:2e9
root@node_a:~# iperf -V -c $node_b_24_v6%wlan0-2
[  3]  0.0-10.0 sec  61.9 MBytes  51.8 Mbits/sec
root@node_a:~# export node_b_5_v6=fe80::6670:2ff:feee:2e8
root@node_a:~# iperf -V -c $node_b_5_v6%wlan1
[  3]  0.0-10.0 sec  70.8 MBytes  59.2 Mbits/sec
root@node_b:~# export node_c_24_v6=fe80::6470:2ff:feee:1b7
root@node_b:~# iperf -V -c $node_c_24_v6%wlan0-2
[  3]  0.0-10.0 sec  49.4 MBytes  41.3 Mbits/sec
root@node_b:~# export node_c_5_v6=fe80::6670:2ff:feee:1b6
root@node_b:~# iperf -V -c $node_c_5_v6%wlan1
[  3]  0.0-10.0 sec  61.8 MBytes  51.7 Mbits/sec
root@node_c:~# export node_d_24_v6=fe80::6470:2ff:feed:f8dd
root@node_c:~# iperf -V -c $node_d_24_v6%wlan0-2
[  3]  0.0-10.0 sec  56.1 MBytes  47.0 Mbits/sec
root@node_c:~# export node_d_5_v6=fe80::6670:2ff:feed:f8dc
root@node_c:~# iperf -V -c $node_d_5_v6%wlan1
[  3]  0.0-10.0 sec  72.3 MBytes  60.5 Mbits/sec

Conclusions

The slowest link in the path is the 2.4Ghz link between node_b and node_c at 41.3Mbit/s and the next slowest is the contiguous 2.4Ghz link from node_c to node_d at 47Mbit/s. So even if our routing protocol is alternating correctly between available paths, an iperf from end to end will always have a 47 or 41.3Mbit/s bottleneck. As a result, the path from node_a to node_c at a rate of 32.4Mbit/s is suffering roughly 25% throughput degradation. It’s also worth mentioning that comparing the results of running iperf from node_a to node_c with running it from node_a to node_d, thus adding only one hop in the path, there’s a 6Mbit/s drop in throughput, which is consistent with the degradation observed for the complete path.

In a single radio mesh, the theoretical degradation is of 50% per hop, so in our setup, the initial 59.2Mbit/s (node_a_5Ghz – node_b_5Ghz) would have been degraded to aproximately 15Mbit/s. At 2.4Ghz the initial 51.8Mbit/s would have been degraded to 13Mbit/s. Comparing this optimistic theoretical single-radio values to the real world multi-radio experiment conducted here, we can assure that multi-radio/multi-band mesh networking has an enormous potential, even on cheap off-the-shelf routers.

Future tests could try to establish optimal links in both 2.4Ghz and 5Ghz between neighboring nodes to test the real end to end performance limit of such a network. It would also be interesting to establish if the observed ~6Mbit/s per hop degradation observed in this experiment is an actual limit of such networks or not.

For now, this series of tests is enough to prove that multi-radio mesh networks do not suffer from the halved bandwidth per hop degradation of single-radio setups.

Coda: udp iperf from node_a to node_d

We know the weakest link is 41Mbit/s, so we send a 40Mbit/s iperf from node_a to node_d, and the result reported by the server side at node_d is 38.5Mbit/s.

root@node_a:/tmp# iperf -c node_d -b 40M -u
WARNING: option -b implies udp testing
------------------------------------------------------------
Client connecting to node_d, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size:  160 KByte (default)
------------------------------------------------------------
[  3] local 10.5.0.24 port 57273 connected with 10.5.0.27 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  47.5 MBytes  39.9 Mbits/sec
[  3] Sent 33912 datagrams
[  3] Server Report:
[  3]  0.0-10.0 sec  46.0 MBytes  38.5 Mbits/sec   0.437 ms 1104/33913 (3.3%)