Load Screenshots from FlightGear

WORK IN PROGRESS

This page describes the steps required to obtain screenshots from FlightGear into the memory of PaparazziUAV. First the required functions are presented and described. Than the exact details about running the applications are mentioned. This has been tested with PaparazziUAV v5.10 and FlightGear 2017.1.2.

Required functions and libraries

FlightGear can be run with an HTTP server which can be used to obtain screenshots of the current scenery in FlightGear. There is a limit to the rate at which these screenshots (I only tested this with approximately 1 fps) can be obtained thus it cannot be used to simulate high frame rate vision. One requires the libcurl library to download the image from the HTTP server into memory. Following this, the jpeg image data needs to be decompressed to obtain the pixel values using libjpeg. All of this code have been obtained from the reference material of the corresponding libraries and required modifications made for the specific implementation. Note that in order to indicate to the compiler that libcurl and libjpeg

Before doing anything a module needs to be added to PaparazziUAV (create a module). By using the 3 functions below (WriteMemoryCallback, curl2mem, get_bmp) one can download the screenshot from the FlightGear HTTP server and convert the jpeg image into bitmap data. The WriteMemoryCallback is a function that is used by the libcurl function curl_easy_setopt to write the downloaded data into the memory. The function curl2mem takes the pointer to a MemoryStruct structure as input. This structure contains two elements namely a pointer to memory and the size of the memory. curl2mem basically writes the information from http://localhost:1234/screenshot (this is where FlightGear will have its screenshot) to a predefined memory location.

struct MemoryStruct
{
  char *memory;
  size_t size;
};

static size_t WriteMemoryCallback(void *contents, size_t size, size_t nmemb, void *userp)
{
    /* Helper function for curl2mem. Handles the writing of the data from
     * the source HTTP into the memory. Needs to follow format provided by
     * libcurl.
     */
  size_t realsize = size * nmemb;
  struct MemoryStruct *mem = (struct MemoryStruct *)userp;

  mem->memory = realloc(mem->memory, mem->size + realsize + 1);
  if(mem->memory == NULL) {
    /* out of memory! */
    printf("not enough memory (realloc returned NULL)\n");
    return 0;
  }

  memcpy(&(mem->memory[mem->size]), contents, realsize);
  mem->size += realsize;
  mem->memory[mem->size] = 0;

  return realsize;
}

void curl2mem(struct MemoryStruct *chunk)
{
    /* Uses libcurl functions to load the screenshot from the FlightGear HTTP
     * server into the programs memory
     */
  CURL *curl_handle;
  CURLcode res;

  curl_global_init(CURL_GLOBAL_ALL);

  /* init the curl session */
  curl_handle = curl_easy_init();

  /* specify URL to get */
  curl_easy_setopt(curl_handle, CURLOPT_URL, "http://localhost:1234/screenshot");

  /* send all data to this function  */
  curl_easy_setopt(curl_handle, CURLOPT_WRITEFUNCTION, WriteMemoryCallback);

  /* we pass our 'chunk' struct to the callback function */
  // curl_easy_setopt(curl_handle, CURLOPT_WRITEDATA, (void *)&chunk);
  curl_easy_setopt(curl_handle, CURLOPT_WRITEDATA, (void *)chunk);

  /* some servers don't like requests that are made without a user-agent
     field, so we provide one */
  curl_easy_setopt(curl_handle, CURLOPT_USERAGENT, "libcurl-agent/1.0");

  /* get it! */
  res = curl_easy_perform(curl_handle);

  /* check for errors */
  if(res != CURLE_OK) {
    fprintf(stderr, "curl_easy_perform() failed: %s\n",
            curl_easy_strerror(res));
  }

  /* cleanup curl stuff */
  curl_easy_cleanup(curl_handle);

  /* we're done with libcurl, so clean it up */
  curl_global_cleanup();
}

After obtaining the jpeg screenshot, it is decompressed to obtain the pixel data. This is done by the get_bmp function which takes 3 input parameters. The first input is a pointer to the chunk of memory containing the jpeg data. The second one is the total length of the memory chunk (i.e. the jpeg data) in bytes. The last input is a pointer to a structure which contains the decompressed pixel data from the jpeg file.

struct BmpStruct
{
    unsigned char *buffer;
    unsigned long size;
    uint16_t width;
    uint16_t height;
    uint8_t pixel_size;
    uint16_t row_stride;
};

uint8_t get_bmp(unsigned char *jpg_buffer, unsigned long jpg_size, struct BmpStruct *bmp)
{
    // // INTIALIZE
    uint8_t rc;
	// Variables for the decompressor itself
	struct jpeg_decompress_struct cinfo;
	struct jpeg_error_mgr jerr;

	// Variables for the output buffer, and how long each row is
	unsigned long bmp_size;
	unsigned char *bmp_buffer;
	uint16_t row_stride, width, height, pixel_size;

    // holds the output bmp and its metadata

    // // SETUP AND CHECK
	cinfo.err = jpeg_std_error(&jerr);
	jpeg_create_decompress(&cinfo);

	jpeg_mem_src(&cinfo, jpg_buffer, jpg_size);

    // 	skipping error check for now. file should be JPEG
	rc = jpeg_read_header(&cinfo, TRUE);
	if (rc != 1) {
        printf("File to get_bmp() not JPEG");
        return 1;
	}

	jpeg_start_decompress(&cinfo);

	width = cinfo.output_width;
	height = cinfo.output_height;
	pixel_size = cinfo.output_components;

	bmp_size = width * height * pixel_size;
	bmp_buffer = (unsigned char*) malloc(bmp_size);

	// The row_stride is the total number of bytes it takes to store an
	// entire scanline (row).
	row_stride = width * pixel_size;

    // // READ THE LINES
	while (cinfo.output_scanline < cinfo.output_height) {
		unsigned char *buffer_array[1];
		buffer_array[0] = bmp_buffer + (cinfo.output_scanline) * row_stride;

		jpeg_read_scanlines(&cinfo, buffer_array, 1);

	}

    bmp->buffer = bmp_buffer;
    bmp->size = bmp_size;
    bmp->width = width;
    bmp->height = height;
    bmp->pixel_size = pixel_size;

	jpeg_finish_decompress(&cinfo);
	jpeg_destroy_decompress(&cinfo);

    return 0;
}

The sum_blues function below is an example of a function that uses the obtained pixel data from the FlightGear screenshot to sum up all the blue pixels. It iterates over all the bytes in the decompressed image which are for the blue channel of each pixel and sums them.

uint8_t sum_blues(void)
{
    struct MemoryStruct chunk; // structure holding download data
    chunk.memory = malloc(1);
    chunk.size = 0;

    struct BmpStruct bmp; // structure holding decompressed image

    // download the jpeg into internal memory
    curl2mem(&chunk);

    // decompress the image and store it in preallocated structure
    get_bmp((unsigned char*)chunk.memory, chunk.size, &bmp);

    // free up memory of downloaded jpeg
    free(chunk.memory);

    uint32_t bluesum = 0;
    unsigned long curinc = 2;
    unsigned char *curby;
    while (curinc <= bmp.size) {
        curby = bmp.buffer + curinc;
        bluesum += *curby;
        curinc = curinc + 3;
    }
    printf("BlueSum: %d\n",bluesum);

    // free up memory of the bmp
    free(bmp.buffer);

    return 0;
}

The user needs to make required implementation and plumbing to call these functions. They can be called by the users modules or through blocks in the flight plans.

Running the application

After creating the required modules and functions, one must start FlightGear and the PaparazziUAV with settings that allow them to communicate with each other. I obtained the information for how to do this from this page.

First, FlightGear should be started with the following command:

fgfs --fdm=null --native-gui=socket,in,30,,5501,udp

There are many other options for FlightGear; refer to this page for details. One may want to include their own 3D models into the environment when trying to simulate computer vision functionalities. The way to do this has been described in this page of the FlightGear wiki.

Second, an extra flag needs to be included when running the simulator of PaparazziUAV.

--fg_host 127.0.0.1

This should be typed in the text box running the simulator in the Paparazzi center. Note that in order to speed up or pause the simulation one has to run it from the terminal as described here.