Redundant Communication

Tools

Antenna Tracker

GPSd position

Input2Ivy

Ipodrom

Kinomap

Plotter

PPRZonDroid

Qwip

Redundant Communication

RTPlotter

Server

SoftwareTools

TCP Aircraft Server

WeatherStationInterface

There is currently a pull request to integrate a redundant communication system into the existing ground segment agents: github.com/paparazzi/paparazzi/pull/429. The goal of this project is to allow multiple radio modems or other links be used between the aircraft and the ground station in a redundant fashion so that even if one link is lost, communication can be maintained. So far, the aircraft to ground component has been created and tested in the lab. More comprehensive testing should be performed before the system is considered to be stable. The ground to aircraft component has not been created yet.

Reasons to redundant links

Communication from the aircraft to the ground is called telemetry. This communication allows the status of the aircraft such as location, altitude, airspeed, battery level, current flight block, and much more to be monitored in real time. Communication from the ground to the aircraft is called datalink. This communication allows commands to be sent to the aircraft from the ground. This includes telling the aircraft what navigation to perform: fly to a waypoint, start the search area, come in for a landing. If a single link is used, as was always the case before this project, then if the aircraft goes out of range, or that link is otherwise lost, no telemetry information can be displayed in the GCS and commands sent from the GCS won't be received by the aircraft. Using the following redundant link system allows multiple links to be used in an effort to avoid this. Typically, you would introduce diversity when adding links. For example:

• If you have an on-board computer, you could tunnel messages through Wi-Fi over UDP in addition to a traditional 900 MHz XBee radio modem.
• You could use a satellite modem for long-range monitoring in addition to a radio modem.
• You could use a cellular modem in addition to a radio modem.
• With additional ground infrastructure, you could have radio modems located at different locations.
• You could use radio modems at different frequencies or with different antenna gains or orientations.
• You could use this system to evaluate different types of radio modems or antenna placements - you'll get real time info on each such as the ping time, received messages rate, and more information could easily be added.

There are undoubtedly many more possibilities.

Aircraft to Ground

Compatibility

This project was made to be fully backwards compatible and support the awesome functionality that Paparazzi already has. This includes:

• Completely backwards compatible - includes aircraft configuration files, previous flight logs, existing sessions, etc...
• Logging and replay - if multiple links are used, data received through either link is recorded. Also, replays will include the status of each link as it was during the flight.
• Link features such as USB-serial, audio, UDP, different transport layers, etc...
• Multi-aircraft sessions - also, you can use redundant links for one aircraft, but not for another if desired.
• Multiple aircraft connected to the same link - such as might come up in a mesh configuration where one aircraft relays data from another aircraft.

Features

Enabling redundant communication from the aircraft to the ground enables the following features:

• Redundancy - as long as one link is receiving data, it will be logged and displayed on the GCS.
• Monitoring - the status, ping time, and rate of receiving messages for each link is shown (this makes it useful even if you're only using one link).
• Alerts - new alerts are added to the console in the GCS to alert you when a link has been connected for the first time, lost, or re-gained.

Future Improvements

Several different improvements can be made to this system to improve it's performance and help satisfy future needs that may arise:

• Additional information can easily be added to the Link page of the GCS for each link. For example: errors, % uptime, or a scrolling graph of Rx messages/s.
• The ability to mark different links as either critical or backup. The idea here is that not all links should necessarily be treated equal. Maybe only one link is able to send datalink data to the aircraft (actually, this is the case right now) and so that link should be considered critical. If any other link is lost, then the overall telemetry can be counted as OK. But if the critical link is lost, then the GCS should change the link indicator light to red and start counting up even if the other link(s) are OK.
• The Link Combiner algorithm currently employs a simple circular buffer. This could be improved on by adding a time-stamp to each element of the buffer and only counting two buffer elements to be the same if their timestamps are close enough (say within 2 seconds - this could be configured)
• The Link Combiner algorithm could be improved by making the buffer size adjust automatically with the ping time and Rx messages/s rate. Or if the timestamp improvement is implemented, then the maximum time difference could be adjusted.
• A performance and resource use analysis could be done to optimize the Link Combiner. The circular buffer is currently implemented using Python lists which could probably be improved upon. Also, significantly more ivy messages are sent and received. This might make optimizations to the regex or other parts of the ivybus worthwhile.

How to use

For an example using two USB-serial adaptors, see the Flight USB-serial Redundant session saved in the example control_panel.xml configuration (on master branch, once the pull request has been accepted).

For the details, keep reading: In order to implement redundant links using this code, there are three simple steps to follow:

• Connect the hardware - on the airborne side, connect the Rx pin of the autopilot's UART port to multiple radio modems or whatever other device which will get the information to the ground. On the ground side, connect multiple modems as normal (using USB to serial adapters, a UDP connection, or whatever). (note: Once the ground to aircraft redundancy has been implemented, you won't have to split the Rx signal and can just plug each modem into a UART port)
• Launch a Link agent for each modem (in Paparazzi Center, select it from the tools menu). Configure the agents as you would have before (serial port, baud rate, UDP, etc...). None of these features have been modified or removed. Then, use two new command line arguments: -redlink and -id. The -redlink flag tells Link that it's one of multiple links. The -id flag lets you specify an integer number as the id of that particular link.
• Launch the Link Combiner agent (again, from the tools menu). This agent listens to all of the Link agents that have their -redlink set and combines the data.

The following screenshot shows a session with two links, connected to serial ports ttyUSB0 and ttyUSB1, and with id's 1 and 2:

The Agents Running in a Redundant Link Setup

• Test the system and make sure that it works correctly under all circumstances. Do this before flying. In particular, make sure that the buffer size if appropriate. Take a look at the Algorithm Considerations section for details.

Interface

Once the redundant link system is started (even if only one Link is used), several changes appear in the GCS. This is achieved by the new message LINK_STATUS which is sent by the Link Combiner. These new features are designed to allow closer monitoring and better management of the link(s) between the aircraft and the ground station.

The following screenshot clearly shows these changes:

Firstly, we can see that there is a new page in the notebook named Link. This page contains critical information for each link. More information can easily be added, but for now, it shows the status of the link (just like the main Link status indicator in the strip), the current ping time, and the rate of received messages. So if one link is lost, it's status indicator will turn red and start counting up just like link 2 in the above screenshot.

Next, we can see that the main Link status indicator that's located in the strip has been modified slightly. It now shows the ratio of connected links to the total number of links. I.e. 3/4 means that there are four links and one of them has been lost. Furthermore, the indicator will be green if all links are connected, yellow if one or more are lost, and red if all links are lost. Once all links are lost, the indicator will count up just like before.

Lastly, alerts have been added to the console. Whenever a link is lost or reconnected, a message will appear in the console stating so. Messages also appear when a new link is added.

Design

Implementation

Here's a block diagram showing how the aircraft talks to the various ground segment programs (also known as agents) when not using the redundant link system:

Paparazzi Ground Segment

The ivy bus lets the programs talk to each other through the use of short messages. Pay close attention to the coloured arrows, which show the main flow of these messages. Next we have a block diagram of the redundant link system:

Paparazzi Ground Segment with Redundant Links

Here we can see the new agent, the Link Combiner. We also see multiple occurrences of the Link agent. When operating in redundant mode, each occurrence of Link transmits the data it receives from the aircraft to the Link Combiner instead of the Server. It does this by transmitting the received telemetry messages within the TELEMETRY_MESSAGE message which the Server doesn't listen to. The Link Combiner on the other hand does. TELEMETRY_MESSAGE messages are sent by each link and include the id of that link. The Link Combiner listens to all of them and then transmits a single set of messages to the Server with the combined data. This results in the Server and all other ground segment agents (such as Messages and the Real Time Plotter) getting their telemetry data from Link Combiner as if it was coming from a single occurrence of Link.

Link Combiner

The Link Combiner agent is written in Python. It receives messages from all of the Links (when they're in redundant mode: -redlink), and outputs messages to the Server and other ground segment agents. There is a bit involved however since the Server and GCS shouldn't be sent identical messages. I.e. if the aircraft sends a message and it's received successfully by more than one link, then that particular message should only sent by Link Combiner once. This is a somewhat tricky requirement since there is no guarantee that the message will be received by all of the links, and it's possible that the aircraft actually sends a second message that's exactly the same as the first. No extra info (such as a time-stamp or counter) is added to the messages sent from the aircraft to the ground, so the Link Combiner employs an algorithm to detect duplicate messages. Testing has shown that for typical links, this algorithm performs extremely well.

The algorithm gives each link a circular buffer to store the last N messages. When a new message comes in on a particular link, the algorithm first checks if it's already in that link's buffer. If it is, then it's obviously a new message, so its removed from all of the other buffers if it's in any of them. Then, the message is sent to the other agents. If the message isn't already in the link's buffer, then it's added to the buffer. After that, all of the other link's buffers are checked for the message. If it's in any of them, then the message was already received by another link, so isn't sent. If it's not in any of them, then the message is sent. The following flowchart depicts this algorithm:

Algorithm to prevent duplicate messages

Algorithm Considerations

As the aircraft sends various messages, possibly with the exact same data, and the links gain and lose connection, there is no guarantee that this algorithm will do a perfect job of filtering out identical and only identical messages. However, in typical operation, it seems to work very well. And for the application of displaying aircraft data, the occasional missing or duplicate message is acceptable. There are only a few scenarios where the Link Combiner won't produce perfect results, and they should come up infrequently:

• A message will be sent twice if the buffer size is too small to hold all of the messages received from one link between that link receiving the message and another link receiving the identical message. When the second link receives the message that has already been received, if the original isn't in the buffer anymore, than the algorithm won't know it's a duplicate message. This scenario should be considered whenever implementing a redundant link system. Partly because it is the most likely to come up, but also because if it does occur, then all messages will be duplicated. In order to avoid this scenario, you can set the buffer size as a command line argument of Link Combiner. To know what a good value would be, you can observer the Ping time and Rx messages/s of the links in the new Link page of the notebook. Calculate the minimum buffer size by taking the difference between the largest ping time and the smallest ping time and multiplying it by the largest Rx messages/s. Then apply a safety factor when setting the buffer size - 10 times the calculated value should be good.
• A message won't be sent when it should be if the autopilot sends a message that's the same as one it already sent (same name and exact same values) AND one or more links were lost between the first message and the second AND the link that's receiving the second message doesn't have the first message in it's buffer still. That's three conditions. If this happens, then the Link Combiner will see the second message as a duplicate of the first and will skip it. Only this one message will be missed though - it won't start happening to all messages. Also, the data will have been data that was already received. So if it's for example the battery voltage message, then it's not a big deal to miss it. As long as there aren't links rapidly being lost and gained, then this shouldn't happen often. Note that as the buffer size is increased, it is believed that this becomes more likely to happen.

Ground to Aircraft

This part of the redundant communication system hasn't been implemented yet. All information here is what it is envisioned to look like.

Features

• Enable multiple datalink links to be used for communicating with the autopilot.
• Let different telemetry be sent out each link.
• Let the choice of what telemetry is sent be moved from the settings page of the GCS to the Link page, since it can be different for each link.
• Add a link status indicator (this is the light that turns red when the link is lost) for the datalink to the Link page. This will allow better monitoring of the links by not assuming that if one direction is good, the other direction is good as well.
• Let each link be configured as telemetry only, datalink only, or by-directional.

Compatibility

Communication in the autopilot is enabled with the ground by including the telemetry subsystem in the airframe file. For example:

File: conf/airframes/myplane.xml

<firmware name="fixedwing or rotorcraft"> ... <subsystem name="telemetry" type="transparent"/> </firmware>

In order to use multiple links, changes should be made to allow a telemetry subsystem for each link. Related to this, would it be advantageous to have multiple instances be supported for each different subsystem? Once the design for this is determined, would actual support for each subsystem have to be added one at a time? Because of the logic for how multiple subsystems should behave together is dependent on the subsystem, this might be the case. The telemetry subsystem for example should default to sending telemetry out each one and combining the telemetry received.

Hardware

Unlike aircraft to ground redundancy, ground to aircraft redundancy requires multiple UART ports in the hardware of the autopilot. The author is not particularly familiar with this, but believes that there are ways to get multiple UART ports for this purpose:

• Telemetry can be sent out over USB by specifying the telemetry type to be "transparent_usb".
• For the Lisa/M, the USB port can actually be used as a UART port with some hardware modifications.
• Using an I2C GPS instead of a UART one can free up a UART port on most autopilots.
• For the Lisa/M, can UART 1 and 5 be used if Spektrum Satellite receivers aren't used? If so, where are the Tx pins?

In addition, future hardware designs could includes more UART ports for redundant communications or any other purpose.

Design

Unlike the aircraft to ground communication, for ground to aircraft communication it is not acceptable to have the same message be acted on twice. This could cause the aircraft to behave unexpectedly. In order to implement a 100% guaranteed link combiner for the aircraft, the author proposes a system that adds a counter to each message sent to the aircraft. This would allow the autopilot to easily ignore identical messages - they'll have the same value for the counter. Also, since minimal data is sent from the ground to the aircraft, adding the extra bytes to the payload will have a minimal impact on performance - this is not the case for the telemetry.

The challenging part of this design will be integrating it properly into the existing systems.