Redundant Communication

There is currently a pull request to integrate a redundant communication system into the existing ground segment agents: [1]. The goal of this project is to allow multiple radio modems or other links be used between the aircraft and the ground station in a redundant fashion so that even if one link is lost, communication can be maintained. So far, the plane to ground component has been created and tested. The ground to plane component has not been created yet.

Reasons to use this

Communication from the aircraft to the ground is called telemetry. This communication allows the status of the aircraft such as location, altitude, airspeed, battery level, current flight block, and much more to be monitored in real time. Communication from the ground to the aircraft is called datalink. This communication allows the commands to be sent to the aircraft from the ground. This includes telling the aircraft what navigation to perform: waypoint navigation, search area, landing. If a single link is used, as is typical, then if the plane goes out of range, or that link is otherwise lost, no telemetry information can be displayed in the GCS and commands sent from the GCS won't be sent to the aircraft. Using the following redundant link system allows multiple links to be used in an effort to avoid this. Typically, you would introduce diversity when adding links. For example:

• If you have an on-board computer, you could tunnel messages through Wi-Fi over UDP in addition to a traditional 900 MHz XBee radio modem.
• You could use a satellite modem for long-range monitoring in addition to a radio modem.
• You could use a cellular modem in addition to a radio modem.
• With additional ground infrastructure, you could have radio modems located at different locations.
• You could use radio modems at different frequencies or with different antenna gains or orientations.
• You could use this system to evaluate different types of radio modems or antenna placements - you'll get real time info on each such as the ping time, received messages rate, and more information can easily be added.

There are undoubtedly many more possibilities.

Compatibility

This project was made to be fully backwards compatible and support the awesome functionality that Paparazzi already has. This includes:

• Completely backwards compatible - includes aircraft configuration files, previous flight logs, existing sessions, etc...
• Logging and replay - if multiple links are used, data received through either link is recorded. Also, replays will include the status of each link as it was during the flight.
• Link features such as USB-serial, audio, UDP, different transport layers, etc...
• Multi-aircraft sessions - you can use redundant links for one aircraft, but not for another if desired
• Multiple aircraft connected to the same link - such as might come up in a mesh configuration where one aircraft relays data from another aircraft

Aircraft to Ground

Features

Enabling redundant communication from the aircraft to the ground enables the following features:

• Redundancy - as long as one link is receiving data, it will be logged and displayed on the GCS
• Monitoring - the status, ping time, and rate of receiving messages for each link is shown (this makes it useful even if you're only using one link)
• Alerts - new alerts are added to the console in the GCS to alert you when a link has been connected for the first time, lost, or re-gained.

How to use

For an example using two USB-serial adaptors, see the Flight USB-serial Redundant session saved in the example control_panel.xml configuration (on master branch, once the pull request has been accepted).

For the details, keep reading: In order to implement redundant links using this code, there are three simple steps to follow:

• Connect the hardware - on the airborne side, connect the Rx pin of the autopilot to multiple radio modems or whatever other device which will get the information to the ground. On the ground side, connect multiple modems as normal (using USB to serial adapters, a UDP connection, or whatever).
• Launch a Link agent for each modem. Configure the agents as you would have before (serial port, baud rate, UDP, etc...). None of these features have been modified or removed. Then, use two new command line arguments: -redlink and -id. The -redlink flag tells Link that it's one of multiple links. The -id flag lets you specify an integer number as the id of that particular link.
• Launch the Link Combiner agent. This agent listens to all of the Link agents that have their -redlink set and combines the data.

The following screenshot shows a session with two links, connected to serial ports ttyUSB0 and ttyUSB1, and with id's 1 and 2:

Caption: The Agents Running in a Redundant Link Setup

Interface

Once the redundant link system is started (even if only one Link is used), several changes appear in the GCS. This is achieved by the new message LINK_STATUS which is sent by the Link Combiner. These new features are designed to allow closer monitoring and better management of the link(s) between the aircraft and the ground station.

The following screenshot clearly shows these changes:

Firstly, we can see that there is a new page in the notebook named Link. This page contains critical information for each link. More information can easily be added, but for now, it shows the status of the link (just like the main Link status indicator in the strip), the current ping time, and the rate of received messages. So if one link is lost, it's status indicator will turn red and start counting up just like link 2 in the above screenshot.

Next, we can see that the main Link status indicator that's located in the strip has been modified slightly. It now shows the ratio of connected links to the total number of links. I.e. 3/4 means that there are four links and one of them has been lost. Furthermore, the indicator will be green if all links are connected, yellow if one or more are lost, and red if all links are lost. Once all links are lost, the indicator will count up just like before.

Lastly, alerts have been added to the console. Whenever a link is lost or reconnected, a message will appear in the console stating so. Messages also appear when a new link is added.

Design

Here's a block diagram showing how the plane talks to the various ground segment programs (also known as agents) when not using the redundant link system:

Caption: Paparazzi Ground Segment

The ivy bus lets the programs talk to each other through the use of short messages. Pay close attention to the coloured arrows, which show the main flow of these messages. Next we have a block diagram of the redundant link system:

Caption: Paparazzi Ground Segment with Redundant Links

Here we can see a new agent, the Link Combiner. We also see multiple occurrences of the Link agent. When operating in redundant mode, each occurrence of Link transmits the data it receives from the plane to the Link Combiner instead of the Server. It does this by transmitting the received telemetry messages within the TELEMETRY_MESSAGE message which the Server doesn't listen to. The Link Combiner on the other hand does. TELEMETRY_MESSAGE messages are sent by each link and include the id of that link. The Link Combiner listens to all of them and then transmits a single set of messages to the Server with the combined data. This results in the Server and all other ground segment agents (such as Messages and the Real Time Plotter) getting their telemetry data from Link Combiner as if it was coming from a single occurrence of Link.

Link Combiner

The Link Combiner agent is written in Python. It receives messages from all of the Links (when they're in redundant mode), and outputs messages to the Server and other ground segment agents. There is a bit involved however since the Server and GCS shouldn't be sent identical messages. I.e. if the plane sends a message and it's received successfully by more than one link, then that particular message should only sent by Link Combiner once. This is a somewhat tricky requirement since there is no guarantee that the message will be received by all of the links, and it's possible that the plane actually sends a second message that's exactly the same as the first. No extra info (such as a time-stamp or counter) is added to the messages sent from the plane to the ground, so the Link Combiner employs an algorithm to detect duplicate messages. Testing has shown that for typical links, this algorithm performs extremely well.

The algorithm gives each link a circular buffer to store the last N messages. When a new message comes in on a particular link, the algorithm first checks if it's already in that link's buffer. If it is, then it's obviously a new message, so its removed from all of the other buffers if it's in any of them. Then, the message is sent. If the message isn't already in the link's buffer, then it's added to the buffer. After that, all of the other link's buffers are checked for the message. If it's in any of them, then the message was already received by another link, so isn't sent. If it's not in any of them, then the message is sent. The following flowchart depicts this algorithm:

Caption: Algorithm to prevent duplicate messages

As the plane sends various, possibly identical, messages and the links gain and lose connection, there is no guarantee that this algorithm will do a perfect job of filtering out identical and only identical messages. However, in typical operation, it seems to work very well. And for the application of displaying aircraft data, the occasional missing or duplicate message is acceptable. There are only a few scenarios where the Link Combiner won't produce perfect results, and they should come up infrequently:

• A message will be sent twice if the buffer size is too small to hold all of the messages received from one link between that link receiving the message and another link receiving the identical message. When the second link receives the message that has already been received, if the original isn't in the buffer anymore, than the algorithm won't know it's a duplicate message. This scenario should be considered whenever implementing a redundant link system. Partly because it is the most likely to come up, but also because if it does occur, then all messages will be duplicated. In order to avoid this scenario, you can set the buffer size as a command line argument of Link Combiner. To know what a good value would be, you can observer the Ping time and Rx messages/s of the links in the new Link page of the notebook. Calculate the minimum buffer size by taking the difference between the largest ping time and the smallest ping time and multiplying it by the largest Rx messages/s. Then apply a safety factor when setting the buffer size - 10 times the calculated value should be good.
• A message won't be sent when it should be if the autopilot sends a message that's the same as one it already sent (same name and exact same values) AND at least one link was lost between the first message and the second AND the link that's receiving the second message doesn't have the first message in it's buffer still. That's three conditions. If this happens, then the Link Combiner will see the second message as a duplicate of the first and will skip it. Only this one message will be missed though - it won't start happening to all messages. Also, the data will have been data that was already received. So if it's for example the battery voltage message, then it's not a big deal to miss it. As long as there aren't links rapidly being lost and gained, then this shouldn't happen often. Note that as the buffer size is increased, I believe that this becomes more likely to happen.