Offline RL tuning for AUV control
Use offline reinforcement learning to select controller settings for depth and heading maneuvers.
Visual description
Research in development. The detailed video explains the tuning variables, the reset–trial–evaluation cycle, response trade-offs and use of selected settings. It does not report a trained policy, measured improvement, global optimality or guaranteed zero overshoot.
Video chapters
Read the narration
The tuning problem
A working controller still needs suitable parameters. An aggressive response can overshoot, while a conservative response may settle slowly. This project explores reinforcement learning to tune the adaptive reference dynamics before deployment, balancing tracking behavior and control effort for underwater maneuvers.
Where reinforcement learning acts
The control architecture contains adaptive reference management followed by model predictive control. Reinforcement learning acts on selected tuning parameters of the adaptive layer. It does not replace the predictive controller, and it does not issue new actuator commands at every control step.
Two natural frequencies to select
The tuning variables are the natural frequencies of the desired depth and heading reference dynamics. Changing them shapes the requested transient response. Their effects must be evaluated in the complete closed loop, because faster desired dynamics do not automatically produce less overshoot or better performance.
One candidate, one complete trial
Before each simulation episode, the agent chooses a candidate pair of frequencies. The simulation is reset and a complete maneuver is evaluated. That parameter pair stays fixed throughout the episode. The trial response then informs the next offline choice.
Evaluate the trade-offs
Each response is assessed for tracking error, overshoot, settling behavior, and control effort. A useful candidate balances these objectives rather than improving one by making another unacceptable. The illustrative comparison shows why the fastest-looking initial response need not be the preferred tuning.
Choose and check a candidate
Offline learning searches for a useful parameter set within the considered domain. A selected candidate then needs validation on representative commands and conditions. The aim is better tuning for those cases, not proof of a globally ideal setting or a universal guarantee of zero overshoot.
Use the settings during the maneuver
The selected frequencies are loaded into the adaptive reference layer. During operation, those tuning values remain fixed while the controller responds to new depth and heading commands. Adaptive compensation can still evolve with measurements; fixed tuning does not mean that the adaptive layer is switched off.
What this project would establish
The final comparison should use the original and learned tuning under the same conditions, measuring overshoot, settling time, tracking error, and control effort. This preview explains that workflow. It does not present trained-policy results or measured improvement, because the research is still in development.
The problem
A working controller still requires suitable tuning: aggressive settings may overshoot, while conservative settings can settle slowly. The aim is to balance tracking, settling and control effort for the considered maneuvers.
Research direction
An offline reinforcement-learning agent selects the natural frequencies of the desired depth and heading reference dynamics in the L1 layer. A candidate pair is held fixed for a complete simulation episode, then the response informs the next choice. The selected tuning is validated and held fixed during deployment; adaptive compensation itself can continue to evolve.
About this preview
Research in development. The detailed video explains the tuning variables, the reset–trial–evaluation cycle, response trade-offs and use of selected settings. It does not report a trained policy, measured improvement, global optimality or guaranteed zero overshoot.