Module/rl obstacle avoidance - Revision history

Gvdam at 09:58, 13 May 2019

2019-05-13T09:58:18Z

← Older revision		Revision as of 03:58, 13 May 2019
Line 172:		Line 172:
	</source>		</source>

	== Training ==		== Training & Evaluation ==
	As the module uses reinforcement learning, the module requires training of the used reinforcement learning agent. It is recommended that multiple agents are trained and evaluated in order to find the best performing agent.		As the module uses reinforcement learning, the module requires training of the used reinforcement learning agent. It is recommended that multiple agents are trained and evaluated in order to find the best performing agent.

Gvdam at 09:57, 13 May 2019

2019-05-13T09:57:40Z

← Older revision		Revision as of 03:57, 13 May 2019
Line 176:		Line 176:

	''Suggestion: Train 5 agents for 50 episodes each. Evaluate each for 20 episodes and select the best one.''		''Suggestion: Train 5 agents for 50 episodes each. Evaluate each for 20 episodes and select the best one.''

			=== Suggested training & evaluation procedure ===
			# Perform a small experiment to estimate the drag coefficients for the quadrotor (or use already known approximations)
			# Set the RL hyperparameters in PAPARAZZI_HOME/sw/tools/rl_obstacle_avoidance/rl_obstacle_avoidance_critic.py.
			# Train multiple agents in flight (''Repeat the procedure below for each of the multiple agents that need to be trained'')
			## Run the Python tool (Real flight, # of training episodes, Training, new critic)
			## Build & Upload Paparazzi to the quadrotor
			## Execute the block "Start Obstacle Avoiance Initialization"
			## Start engines & takeoff
			## Execute the block "Estimate prop gains"
			## Fly to the episode start waypoint
			## Ensure that '''Exploring starts''' is TRUE
			## Execute the block "Start episode"
			## Start the first few episodes manually if there are no issues turn "Autostart Episode" ON.
			## Replace batteries if need be, but ensure that the Python tool keeps running uninterrupted until training is finished
			# Evaluate each agent (''Repeat the procedure below for each of the multiple agents that need to be evaluated'')
			## Run the Python tool (Real flight, # of evaluation episodes, Evaluation, ID of the trained agent)
			## Build & Upload Paparazzi to the quadrotor
			## Execute the block "Start Obstacle Avoiance Initialization"
			## Start engines & takeoff
			## Execute the block "Estimate prop gains"
			## Fly to the episode start waypoint
			## Ensure that '''Exploring starts''' is FALSE
			## Execute the block "Start episode"
			## Start the first few episodes manually if there are no issues turn "Autostart Episode" ON.
			# Compare evaluation performance and select the best performing agent

			=== Implementation of a trained agent ===
			In the current version of this module, the RL agent is stored on the ground station. As such, the Python tool is required to upload the tool to the quadrotor at the start of every flight.

			'''''Suggested improvement:''' For operational usage of this module, a major improvement would be to have the option to upload a trained & selected agent to the quadrotor, after which it could run without requiring the Python tool. (This improvement for future research/development is also listed in the ToDo section at the bottom of this page)''

	== Onboard module ==		== Onboard module ==

Gvdam: /* Flight status */

2019-05-13T09:49:07Z

Flight status

← Older revision		Revision as of 03:49, 13 May 2019
Line 132:		Line 132:

	=== Flight status ===		=== Flight status ===
	The obstacle avoidance is active when the flight status >= 50 and <60. The flight status can be set from any block in the flight plan using the following code. With XX being the desired flight status.		'''The obstacle avoidance is active when the flight status >= 50 and <60.''' The flight status can be set from any block in the flight plan using the following code. With XX being the desired flight status.
	<source lang="html">		<source lang="html">
	<call_once fun="rl_obstacle_avoidance_flight_status(XX)" />		<call_once fun="rl_obstacle_avoidance_flight_status(XX)" />

Gvdam at 09:38, 13 May 2019

2019-05-13T09:38:16Z

← Older revision		Revision as of 03:38, 13 May 2019
Line 80:		Line 80:
	=== Flight plan ===		=== Flight plan ===
	For this module to work, the following additions to the flight plan need to be made.		For this module to work, the following additions to the flight plan need to be made.

			'''Add the following waypoints'''<br />
			With the x and y location set to the point in the local reference frame where training/evalution should be conducted
			<source lang="html">
			<waypoint name="EPISODE_START" x="2.2" y="-2.0" height="1.0" />
			<waypoint name="EPISODE_TARGET" x="2.2" y="-2.0" height="0.0" />
			</source>

	'''Insert in the exceptions section of the flightplan'''		'''Insert in the exceptions section of the flightplan'''

Gvdam at 09:25, 13 May 2019

2019-05-13T09:25:10Z

← Older revision		Revision as of 03:25, 13 May 2019
Line 176:		Line 176:
	* '''Exploration rate''', this is controlled by the Python tool and should not be manually changed		* '''Exploration rate''', this is controlled by the Python tool and should not be manually changed
	* '''Autostart episode''', start episodes automatically, or require a user action each episode.		* '''Autostart episode''', start episodes automatically, or require a user action each episode.
	* '''Exploring starts''', if this is activated the RL agent is 'frozen' until a randomly generated height between 0.4 and 1.0m above the obstacle is reached. This should be TRUE for training, but FALSE for evaluation.		* '''Exploring starts''', if this is activated the RL agent is 'frozen' until a randomly generated height between 0.4 and 1.0m above the obstacle is reached. '''This should be TRUE for training, but FALSE for evaluation.
			'''

	== Python application ==		== Python application ==

Gvdam at 09:21, 13 May 2019

2019-05-13T09:21:21Z

← Older revision		Revision as of 03:21, 13 May 2019
Line 169:		Line 169:

	''Suggestion: Train 5 agents for 50 episodes each. Evaluate each for 20 episodes and select the best one.''		''Suggestion: Train 5 agents for 50 episodes each. Evaluate each for 20 episodes and select the best one.''

			== Onboard module ==
			The following options are available for this module
			* '''Termination height''', minimum height above an obstacle. If the distance becomes smaller the safety mechanism of the module will intervene, and the RL episode will be considered a 'crash'.
			* '''Descend speed''', used in training and evaluation. [m/s]
			* '''Exploration rate''', this is controlled by the Python tool and should not be manually changed
			* '''Autostart episode''', start episodes automatically, or require a user action each episode.
			* '''Exploring starts''', if this is activated the RL agent is 'frozen' until a randomly generated height between 0.4 and 1.0m above the obstacle is reached. This should be TRUE for training, but FALSE for evaluation.

	== Python application ==		== Python application ==
Line 185:		Line 193:
	* Turn off logging when not in training		* Turn off logging when not in training
	* Add the Python tool to the Paparazzi dashboard		* Add the Python tool to the Paparazzi dashboard
			* Automatically delete created files in /tmp after usage, or after shutdown

	[[Category:User_Documentation]] [[Category:Modules]]		[[Category:User_Documentation]] [[Category:Modules]]

Gvdam at 09:13, 13 May 2019

2019-05-13T09:13:47Z

← Older revision		Revision as of 03:13, 13 May 2019
Line 178:		Line 178:
	</source>		</source>

	When running the tool the user can choose between ''Simulation'' or ''Real flight'' and ''Training'' or ''Evaluation''. Finally one can choose to use an already existing RL agent (agents are stored in PAPARAZZI_HOME/sw/tools/rl_obstacle_avoidance/OnlineLearning), or train a new RL agent from scratch.		When running the tool the user can choose between ''Simulation'' or ''Real flight'', the amount of episodes, and ''Training'' or ''Evaluation''. Finally one can choose to use an already existing RL agent (agents are stored in PAPARAZZI_HOME/sw/tools/rl_obstacle_avoidance/OnlineLearning), or train a new RL agent from scratch.

	== Todo ==		== Todo ==

Gvdam at 14:35, 10 May 2019

2019-05-10T14:35:21Z

@@ Line 33: / Line 33: @@
 * <math display="inline">k_{D,z}</math>, the drag coefficient iin body z-axis, estiamted during a system identification experiment
 * <math display="inline">w</math>, the speed in body z-axis
 == Setup ==

Gvdam: /* Python application */

2019-05-10T14:29:15Z

Python application

← Older revision		Revision as of 08:29, 10 May 2019
Line 164:		Line 164:
	python3 rl_obstacle_avoidance_critic.py		python3 rl_obstacle_avoidance_critic.py
	</source>		</source>

			When running the tool the user can choose between ''Simulation'' or ''Real flight'' and ''Training'' or ''Evaluation''. Finally one can choose to use an already existing RL agent (agents are stored in PAPARAZZI_HOME/sw/tools/rl_obstacle_avoidance/OnlineLearning), or train a new RL agent from scratch.

	== Todo ==		== Todo ==

Gvdam: /* Python application */

2019-05-10T14:26:35Z

Python application

← Older revision		Revision as of 08:26, 10 May 2019
Line 158:		Line 158:

	== Python application ==		== Python application ==
			The python application required for training and evaluation is located in PAPARAZZI_HOME/sw/tools/rl_obstacle_avoidance/.

			It can be run from the commandline using the following command:
			<source>
			python3 rl_obstacle_avoidance_critic.py
			</source>

	== Todo ==		== Todo ==