RL Dresden

Reinforcement Learning Research Group

Contact us »

Motivation

Optimal control for traffic dynamics

Reinforcement Learning (RL) is gaining traction in our increasingly connected world. It excels in uncertain and complex scenarios, aiming to maximize long-term rewards by guiding agents through actions in their environment. RL finds applications in self-driving vessels, traffic optimization, and mastering complex games like Starcraft or Go. Its potential lies in automating tasks prone to errors and accidents, enhancing both efficiency and safety in human labor.


RLCDD 2022

On September 15-16, 2022, the "Friedrich List" faculty of transport and traffic sciences hosted a conference on Reinforcement Learning. The conference was organized by the Reinforcement Learning Group Dresden and was held in a hybrid format, with both online and on-site participation and was attended by over 100 participants from more than 10 countries.

Check out our conference website here .

News

September 10-13, 2024

Ostap Okhrin delivered a keynote "Gumbel Lecture" at the Statistishe Woche 2024 on the topic "Reinforcement Learning in Transportation". Ankit Chaudhari, Saad Sagheer, Haoze Jiang, Shashank Rajput, and Shivachetan Venkateshappa presented at the same place.


September 2-6, 2024

Ankit Chaudhari presented work on " Drone-Based Trajectory Data for an All-Traffic-State Inclusive Freeway with Ramps." at the TRC30 and MFTS conferences in Heraklion, Crete, Greece.


May 20-24, 2024

Ostap Okhrin gave a keynote speech at the PROBASTAT in Slovakia on „Two-sample testing in reinforcement learning".


March 13-15, 2024

Ostap Okhrin gave a talk at SMSA 2024 in Delft on „A Platform-Agnostic Deep Reinforcement Learning Framework for Effective Sim2Real Transfer in Autonomous Driving".


December 5-6, 2023

Martin Waltz gave a presentation entitled "Local Path Planning in Transportation using Reinforcement Learning" on the, 1st Symposium On Lifelong Explainable Robot Learning in Nürnberg.


September 24-28, 2023

Dianzhao Li paricipated in the 26th IEEE International Conference on Intelligent Transportation Systems (ITSC) where he introduced his work on "Vision-based DRL Autonomous Driving Agent with Sim2Real Transfer".


September 11-14, 2023

Ostap Okhrin and Niklas Paulig paricipated in the Statistical Week 2023 at TU Dortmund, where Ostap Okhrin gave a talk on "Two-sample Testing in Reinforcement Learning" and Niklas Paulig presented his work on "Robust Path Following on Rivers Using Bootstrapped Reinforcement Learning".


August 29, 2023

Chair member Ankit Chaudhari gave an interview at the workshop on 'Integrated Engineering for Future Mobility' in Delhi. The workshop aimed at fostering collaboration and generating innovative research ideas related to urban mobility by using a "Design Thinking" approach.


June 8, 2023

Martin Waltz paper "Spatial-temporal recurrent reinforcement learning for autonomous ships" has been published in Neural Networks


June 7, 2023

Ostap Okhrin and Dianzhao Li participated in the 2. Sächsische KI-Kongress des Freistaates Sachsen. This prestigious event brought together more than 250 distinguished guests representing business, science, society, and politics, creating a dynamic platform for discussions on the latest developments and trends in the field of AI. Livestream


May 8, 2023

The chair recently conducted drone-based traffic data collection effort on the A50 highway in Milan, Italy. With a co-ordinated fleet of 7 Drones flying in a line, traffic was captured over a 1000m road section for 130 minutes. Our team member Ankit Chaudhari oversaw the data collection on site.


April 27, 2023

Ankit Chaudhari recently participated in a "design thinking based" workshop on 'Integrated Engineering for Future Mobility' organized by the German Centre for Research and Innovation (DWIH) in New Delhi, India.


April 25, 2023

Fabian Hart's paper "Vessel-following model for inland waterways based on deep reinforcement learning" is accepted for publication in the journal of Ocean Engineering.


Projects

Path following validation

Water

Navigating maritime traffic poses multifaceted challenges, including the intricate interplay of environmental factors like wind, waves, and currents, alongside the dynamic presence of other vessels with varying behaviors. These complexities demand precise control strategies that can adapt to real-world conditions, considering factors such as shallow water impacts and spatial restrictions imposed by waterway geometry. Within our research, we aim to address these challenges through modularized frameworks and advanced reinforcement learning techniques, facilitating realistic path planning and robust control of autonomous vessels in diverse maritime environments.

Ground

Addressing urban congestion and outdated infrastructure while improving traffic flow and reducing pollution is crucial. Implementing advanced traffic management systems equipped with AI algorithms can optimize flow and reduce congestion through dynamic route planning. We do our best to develop those AI algorithms. Among other tasks, to overcome the challenge of transferring trained autonomous vehicle agents from simulation to reality, we focus on separating agents into perception and control modules, enabling smoother integration into real-world environments. Similarly, by incorporating real-world driving datasets into training reinforcement learning agents, we aim to enhance their behavior and generalization capabilities, ultimately improving autonomous driving systems' performance on actual roads.

Map of the north sea area
shared policy for decentralized actions

Air

Urban air mobility presents unique challenges in managing eVTOL vehicles around vertiports, including congestion, safety, and unpredictable factors like weather and passenger demand. Among others, in our project, we address these complexities by proposing self-organized arrival systems using deep reinforcement learning. Treating each aircraft as an individual agent, we develop a shared policy for decentralized actions based on local information.

Methods

The continual development of new algorithms and models is imperative to tackle practical challenges with reinforcement learning. Even established methods like Q-Learning encounter limitations, particularly regarding the maximization operator in target computation. Inherent in the Bellman optimality equation, this operator often inflates Q-values for state-action pairs, impacting subsequent updates. As part of our broader theoretical endeavors in reinforcement learning, we are currently developing a novel suite of estimators tailored to address the max-mu problem, seamlessly integrating them within the framework of deep neural network-based function approximations.

Plots showing overestimatin bias in RL

Software

RL Dresden Algorithm Suite [GitHub]

This suite implements several model-free off-policy deep reinforcement learning algorithms for discrete and continuous action spaces in PyTorch.

Plots showing overestimatin bias in RL

Mixed Traffic Web Simulators [mtreiber.de]

Fully operational, 2D, Javascript-based web simulator implementing the "Mixed Traffic flow Model" (MTM). This simulation is intended to demonstrate fully two-dimensional but directed traffic flow and visulalize 2D flow models.

Mixed traffic web simulator

Sim2Real Transfer package with Duckiebot [GitHub]

This package includes the training and evaluation code under ROS platform for Sim2Real Transfer with Duckiebot for multiple autonomous driving behaviors.

Sim2Real Transfer package with Duckiebot

Publications

Card image cap
Two-step dynamic obstacle avoidance

Waltz, M., Okhrin, O., Hart, F. (2024). Knowledge-Based Systems (To Appear)

Card image cap
Addressing maximization bias in reinforcement learning with two-sample testing

Waltz, M., Okhrin, O. (2024). Artificial Intelligence 336, 104204

Card image cap
Self-organized free-flight arrival for urban air mobility

Waltz, M., Okhrin, O., Schultz, M. (2024). Transportation Research Part C: Emerging Technologies, Volume 167, 104806

Card image cap
Towards robust car-following based on deep reinforcement learning

Hart, F., Okhrin, O., Treiber, M. (2024). Transportation Research Part C: Emerging Technologies, Volume 159, 104486

Card image cap
Robust Path Following on Rivers Using Bootstrapped Reinforcement Learning

Paulig, N., & Okhrin, O. (2024). Ocean Engineering, Volume 298, 117207

Card image cap
Towards Autonomous Driving with Small-Scale Cars: A Survey of Recent Development

Li, D., Auerbach, P., & Okhrin, O. (2024). arXiv preprint arXiv:2404.06229.

Card image cap
An open-source framework for data-driven trajectory extraction from AIS data -- the α-method

Paulig, N., & Okhrin, O. (2024). arXiv preprint arXiv:2407.04402.

Card image cap

Vision-based DRL Autonomous Driving Agent with Sim2Real Transfer

Li, D., & Okhrin, O. (2023). In 2023 IEEE 26th International Conference on Intelligent Transportation Systems (ITSC) (pp. 866-873).

Card image cap
2-Level Reinforcement Learning for Ships on Inland Waterways

Waltz, M., Paulig, N., & Okhrin, O. (2023). arXiv preprint arXiv:2307.16769

Card image cap
Spatial-temporal recurrent reinforcement learning for autonomous ships

Waltz, M., & Okhrin, O. (2023). Neural Networks, 165, 634-653.

Card image cap
Vessel-following model for inland waterways based on deep reinforcement learning

Hart, F., Okhrin, O., & Treiber, M. (2023). Ocean Engineering, 281, 114679

Card image cap
DDPG car-following model with real-world human driving experience in CARLA

Li, D., & Okhrin, O. (2023). Transportation Research Part C: Emerging Technologies, Volume 147

Student Works

Linienlose Tabelle
Year Name Degree Thesis Title
2023 Peilin Wang Master (D.Li) Autonomous Navigation with Deep Reinforcement Learning in Carla
2023 Yuhua Zhu Master (D.Li) Autonomous Driving with Deep Reinforcement Learning
2021 Paul Ziebarth Diplom (F.Hart) Entwicklung eines Reinforcement Learning Agenten zur Realisierung eines Schifffolgemodells

Team

Prof. Dr. Ostap Okhrin

Ostap Okhrin is the professor for Statistics and Econometrics at the Department of Transportation at the TU Dresden. His expertise lies in mathematical statistics and data science with applications in transportation and economics.

Prof. Dr. Ostap Okhrin
Dr. Martin Treiber

Dr. Martin Treiber

Martin Treiber is a senior expert in traffic flow models including human and automated driving, bicycle, and pedestrian traffic. He also works in traffic data analysis and simulation (traffic-simulation.de, mtreiber.de/mixedTraffic).

Niklas Paulig

Niklas Paulig is a RL Group research associate, with his main field of research being the modeling of autonomous inland vessel traffic based on reinforcement learning methods, and HPC implementations of currently in use algorithms.

Niklas Paulig
Martin Waltz

Martin Waltz

Martin Waltz conducted his studies in Industrial Engineering and is now a research associate, with his main research focus being (Deep) Reinforcement Learning.

Ankit Anil Chaudhari

Ankit Chaudhari is currently working on "Enhancing Traffic-Flow Understanding by Two-Dimensional Microscopic Models". His research interests are traffic flow modelling, traffic simulation, mixed traffic flow, machine learning and reinforcement learning.

Ankit Anil Chaudhari
Dianzhao Li

Dianzhao Li

Dianzhao Li is a research assistant at RL-Dresden, focusing on the area of trajectory planning for autonomously driving vehicles with reinforcement learning algorithms. He now mixes the human driving datasets with RL in simulated environments to achieve better performance for the vehicles.

Paul Auerbach

Paul Auerbach is a research associate at the Barkhausen Institut and collaborates with RL-Dresden on the simulation and solving of traffic scenarios with the help of reinforcement learning. He aims to transfer the learned RL models to real world model cars.

Paul Auerbach
Gong Chen

Gong Chen

Gong Chen is a research associate within RL-Dresden, concentrating on applying reinforcement learning to simulate shipping traffic under shallow water conditions.

Saad Sagheer

Saad Sagheer is a Civil Engineer who is currently working as a research associate and collabrates with BAW (Bundesanstalt für Wasserbau). His main field of research is to conduct micro-traffic simulation of inland vessels in Rhine especially Middle Rhine with different water level conditions related to climate change.

Saad Sagheer


Former Members
Fabian Hart
Hadil Romdhane