How GenAI is helping drive vehicle autonomy

Even if it feels like an an easy task for humans, driving has proven to be one of the most difficult tasks ever for machines, given that there is an infinite set of new scenarios that may arise in the driving journey.

Already today, we can see autonomous vehicles taking to the roads in geographies such as San Francisco or Wuhan. Waymo, a subsidiary of Google, is now providing more than 200,000 paid robotaxi rides every week among their operations in Los Angeles, San Francisco and Phoenix. The year 2025 is also expected to be an important one for autonomous trucks, with several companies planning to launch commercial operations in the US.

Despite these first successes, various challenges for vehicle autonomy persist, from regulatory complexities to consumer acceptance. Technological hurdles also remain critical, and artificial intelligence (AI), including its newer generative AI (GenAI) capabilities, is emerging as a key enabler in overcoming some of these challenges.

So, how do vehicles think? And what is the role GenAI plays in the driving process? The following three points help clarify these questions.

Rule-based systems have been the basis of decision-making in automated vehicles. However, while they offer predictability and transparency, they fall short in handling the complexity of real-world driving.

AI is helping overcome these shortcomings, first by introducing AI in single components and now with the development of end-to-end (E2E) AI models. These E2E AI models combine perception, prediction and planning into a single neural network, enabling vehicles to learn and act with unprecedented speed and adaptability.

However, E2E AI models come with challenges, interpretability being the most pressing one due to the traditional black-box nature of these models. While the black-box nature results in safety concerns, recent breakthroughs seem to provide more interpretable and verifiable solutions.

Raquel Urtasun, Founder and CEO of self-driving technology company Waabi, states: “There is a massive leap in AI happening right now, and it is resulting in smarter end-to-end AI systems that can learn much more efficiently, are interpretable, and can generalize to every possible scenario on the road.

“These advances result in autonomous vehicles with superhuman capabilities that will enhance road safety and transform transportation as we know it.”

Developing AI systems for autonomous vehicles requires vast amounts of data, especially to account for edge cases such as complex road scenarios and unexpected emergencies As collecting real-world data for every conceivable scenario is, unfortunately, impossible, this is where synthetic data comes becomes invaluable.

AI-driven simulation platforms can generate synthetic datasets, providing scalable, diverse and targeted training scenarios. Such data enables developers to simulate millions of miles of driving while covering both routine and rare scenarios. For example, companies like Waymo, Waabi and insurtech Simulytic leverage synthetic data to train AI models for critical edge cases such as complex multimodal situations or sensor disruptions in extreme weather.

Andrea Kollmorgen, CEO of Simulytic, a Siemens venture, explains the broader implications: “Next to training the models, simulation allows us to create comprehensive risk profiles for autonomous vehicles, enabling insurers to develop fair and effective strategies.”

While synthetic data aids training by covering diverse scenarios, real-world data remains essential. Final validations on the road are crucial to ensuring performance and safety.

Fully autonomous vehicles are the goal, but assisted and automated driving will dominate the next decade, requiring effective human-machine collaboration. Driver monitoring systems (DMS) and human-machine interfaces (HMI) play a key role in ensuring seamless interaction. AI enhances both technologies.

DMS analyses eye movements and facial expressions to detect fatigue, stress, and attention levels, and GenAI can enhance the accuracy of these assessments. As to HMI, it encompasses how the driver interacts with the vehicle, and GenAI can improve such interactions, enabling drivers to manage functions via natural language commands, thereby reducing distractions.

“The ‘ABC’ of autonomous driving – AI, big data, and computing power – advances DMS and HMI systems, enabling more intuitive, collaborative and safe driving. Especially in situations like crisis management, AI-enhanced systems can play a pivotal role by providing immediate, context-specific interventions,” notes Erhan Köseoğlu, Executive Director of Growth and Smart Mobility at Ford Otosan.

AI not only shapes vehicle behaviour but also revolutionizes the development processes behind autonomous systems. It generates code snippets, automates testing, detects bugs and suggests optimized solutions, enhancing software quality, reducing costs and accelerating development cycles. These advancements are crucial in a field with the potential to make roads safer.

However, as AI-driven development scales, it also introduces new risks, including cybersecurity vulnerabilities, system reliability concerns and regulatory compliance challenges. Ensuring transparency in AI-driven models and aligning with evolving safety standards will be critical.

The transformation of software engineering by GenAI also underscores the need for an AI-ready workforce. One that is, among other things, able to teach models to make safe and ethical decisions.

Alwin Bakkenes, Volvo Car’s Head of Global Software Engineering, comments: “Through advanced AI applications, we believe we can take significant leaps in reducing collisions and get ever closer to our ambition of zero collisions.

“Success here will depend not just on technology but also on the people who design, implement, and manage it. We need to invest in talent development at every level, from UI/UX designers to AI research scientists, to ensure we stay at the forefront of this transformation.”

The path to vehicle autonomy remains complex. AI is critical in tackling the intricacies of autonomous systems, but it also raises new challenges, such as limited explainability, potential biases and randomness in decision-making.

Collaboration within the industry is essential to harness the potential of GenAI while addressing associated risks. Shinpei Kato, CEO of TIER IV and Chairman of the Autoware Foundation, highlights: “Collaboration is key to unlocking safe and scalable autonomous systems, and open-source software can accelerate the deployment of vehicle autonomy.”

Finally, the automotive and tech industries need to engage other stakeholders to ensure the safe and successful integration of GenAI advancements in vehicle autonomy. This involves fostering dialogue with regulators and policy-makers to enhance their understanding of the related capabilities and limitations.

The World Economic Forum also contributes to the goal of a safe and reliable software-defined-vehicle and vehicle autonomy transformation with its DRIVE-A: Vehicle Autonomy initiative.