A reinforcement learning framework to enhance the ramp merging capabilities of autonomous vehicles

While many automotive companies are now invested in the development of self-driving cars, the vehicles created so far have not yet attained the safety levels necessary for them to be deployed on a large-scale. For this to happen, the vehicles will need to be able to tackle a wide variety of challenges on the road both safely and effectively.

Researchers at Carnegie Mellon University have recently developed a reinforcement learning (RL)-based framework that could help to improve the performance of autonomous vehicles in ramp merging scenarios, instances where vehicles on a ramp road are deviated onto a main road. Their framework, presented in a paper pre-published on arXiv, could potentially help to improve the safety of autonomous vehicles in these particularly salient times, reducing the risk of accidents.

“Prof. John Dolan’s lab at CMU has been working on various autonomous driving applications for quite some time,” Soumith Udatha, one of the researchers who developed the model, told TechXplore. “The application that we focused on in this paper is that of freeway merging because of the challenges involved with vehicles at high speeds, drivers with various styles, and uncertainties involved.”

The overreaching objective of the research efforts by Udatha and his colleagues is to improve the safety of autonomous vehicles. In their recent paper, they specifically tried to devise a framework that could effectively capture ramp merging scenarios and plan a vehicle’s actions based on its analyses of any uncertainties and possible risks.

“RL models interact with an environment and collect data to optimize their rewards, but this data exploration encounters some problems when deployed in real-world settings,” Udatha explained. “This is partly because not all the states that the agent encounters are safe. We constrained our RL policy with control barrier functions (CBFs) to ensure safety at a specified distance. So, with the environmental constraints, we ignore the unsafe states and enhance a system’s ability to learn how to navigate.”

CBFs are a class of fairly new computational methods designed to enhance the safe control of autonomous systems. CBFs can be directly applied to different optimization problems, including ramp-merging. Despite their promising qualities, the optimizations they perform do not account for the data gathered by a system while it is exploring an environment. RL methods can help to fill this gap.

“We found that our algorithm can be extended to both offline and online RL environments,” Udatha said. “Since we have enormous amounts of data now for offline RL, training on offline datasets can eventually lead to better models. With our metrics, we also discovered that the inclusion of Probabilistic CBFs as constraints provides better safety, as it accounts for the driver uncertainty to an extent.”

Udatha and his colleagues tested their framework in a series of tests, using the online version of the CARLA simulator developed by a team of researchers at Intel Labs and the Computer Vision Center in Barcelona. In these simulations, their approach achieved remarkable results, highlighting its possible value for enhancing the safety of autonomous vehicles during ramp merging.

“We now plan to extend our research by training our model to merge an autonomous vehicle with multiple vehicles in a scene with driver uncertainties,” Udatha added. “We also found that a standard benchmark to compare various approaches for ramp-merging is currently missing, so we are concurrently trying to establish a ramp-merging benchmark for NGSIM, a freeway dataset released by NHTSA on the US I-80 and US 101 freeways.”

© 2022 Science X Network