Machine Learning Development Using Behavioral Analysis

A behavioral safety-driven use case for developing ML models for automated driving features guided by novel safety standards

Introduction

One of the most significant challenges in modern automotive safety is the safe development and deployment of systems that rely on non-deterministic Machine Learning (ML) components. As the industry moves toward automated driving functions that depend heavily on perception, planning, and control algorithms powered by ML, traditional safety approaches must evolve to ensure predictable and acceptable vehicle behavior across complex operational environments.

From a safety engineering perspective, the problem is not only verifying that software operates without faults, but also ensuring that learned behaviors remain safe under a wide variety of real-world conditions. This introduces challenges addressed by functional safety (ISO 26262), safety of the intended functionality (ISO 21448/SOTIF), and more recently AI-specific safety lifecycle guidance such as ISO/PAS 8800.

Based on practical experience integrating safety across multiple organizations deploying ML-enabled systems, this article describes a structured behavioral safety approach that bridges system definition, scenario analysis, model development, verification, and validation. The intent is to demonstrate how safety-driven scenario engineering can guide both ML training and validation while ensuring alignment with industry standards.

1. General Approach: Behavioral Safety as a Systems Engineering Framework

Behavioral safety focuses on defining and validating how a vehicle should behave safely across its Operational Design Domain (ODD) rather than only analyzing system malfunctions.

Traditional functional safety processes such as HARA (Hazard Analysis and Risk Assessment) focus on identifying hazards caused by failures. However, automated driving introduces risks even when systems operate as intended but insufficiently — a core concern of SOTIF and AI safety.

The proposed approach begins with:

  • Clear definition of the system’s ODD.

  • Identification of vehicle capabilities or behaviors (e.g., Adaptive Cruise Control, lane keeping, lane changes, collision avoidance).

  • Definition of safe behavioral expectations across operating contexts.

Behavioral safety therefore acts as a bridge between:

DisciplinePrimary FocusISO 26262 (FuSa)Malfunction-related hazardsISO 21448 (SOTIF)Performance limitations and insufficienciesISO/PAS 8800AI lifecycle and data-driven safetyBehavioral safety approachSafe vehicle behavior across scenarios

Rather than treating safety as an activity occurring after development, this framework embeds safety criteria into scenario definition, data selection, model training, and validation.

2. Scenario Definition Based on ODD and Vehicle Behaviors

Behavioral safety begins with structured scenario definition grounded in a rigorous Operational Design Domain (ODD) specification. Rather than defining scenarios ad hoc, the ODD must be systematically decomposed using established industry methodologies to ensure completeness, traceability, and consistency across engineering teams. Two frameworks are particularly useful for structuring this process, ISO 34503 and Pegasus Layer Model.

ISO 34503 provides a structured approach for defining the ODD through three primary categories

  • Scenery elements – including, static road and infrastructure characteristics road geometry, lane markings, signage, barriers

  • Environmental conditions – including  External environmental factors affecting operation weather, lighting, visibility, road surface conditions

  • Dynamic elements – including moving actors and traffic interactions vehicles, vulnerable road users (VRUs)

The PEGASUS project refines the ODD through a multi-layer approach:

  • Layer 1 – Road-level layout

  • Layer 2 – Traffic infrastructure

  • Layer 3 – Temporary modifications

  • Layer 4 – Dynamic objects

  • Layer 5 – Environmental conditions

  • Layer 6 – Digital information

Both of these layers are highly effective framework to decompose the ODD. Whichever is used, the most important thing to remember is to have a systematic approach to break down the ODD.

3. Scenario Analysis Using Safety Performance Indicators (SPIs)

Once scenarios are defined, behavioral safety analysis evaluates safe performance expectations using Safety Performance Indicators (SPIs).

Typical SPIs include:

  • Minimum time-to-collision (TTC)

  • Minimum distance to obstacle

  • Maximum acceleration/deceleration limits

  • Comfort and stability thresholds

  • Reaction timing constraints

The analysis answers:

  • When must the vehicle detect the hazard?

  • When must braking begin?

  • What trajectory constraints must be respected?

These behavioral expectations directly translate into:

  • Perception requirements (detection range, classification confidence)

  • Planning requirements (trajectory generation)

  • Control requirements (actuation limits)

4. Safe Behavior Definition Across the ODD

Aggregating scenario analyses produces a structured set of safe behavioral requirements, which includes functional, performance, and constraint requirements. These define what “safe operation” means beyond simply avoiding failures.

This resembles HARA (ISO 26262) and HIRE (ISO 21448) in methodology but differs in intent. Both HARA and HIRE focus assessing the risk of hazards caused by unsafe behavior, ISO 26262 HARA focuses on malfunctioning behavior at the vehicle level (caused by faults), whereas SOTIF’s HIRE focuses on hazardous behavior caused by functional insufficiencies. In the case of behavioral safety, we are focusing on what the safe behaviors are, not on identifying unsafe behaviors. Of course, in practicality, identifying the safe behaviors across different scenarios means that we are identifying the limits of the safe behaviors, and therefore where the unsafe behavior begins. Which is why, in my experience, all of these safety analysis benefit of feeding one another for consistency and efficiency.

The scenario image below describes a specific scenario relevant for our ODD. In this scenario, we analyze against SPIs for maximum deceleration and minimum distance to a stopped vehicle. So the questions we are answering by doing this analysis are

  • At what point does the vehicle need to start decelerating in order to stop at a safe distance from the stopped vehicle?

  • And the above leads to the question, at what point does the SDT need to detect the stopped vehicle such that it can start decelerating at the minimum distance to safely execute longitudinal deceleration to stop at a safe distance?

This approach is followed for all the safety relevant scenarios within our ODD across behavior capabilities, which gives us a set of relevant abstract scenarios for parametrization in simulation.

5. Scenario Parameterization for Simulation

High-level scenarios are converted into concrete, parameterized simulation cases. Again, these scenarios are created based on the systematic ODD characterization of industry leading standards like ISO 34503.

Parameters may include:

  • Speed ranges

  • Distances

  • Sensor noise levels

  • Traffic density

  • Environmental variability

Simulation environments (e.g., CarMaker, CARLA, proprietary tools) allow systematic exploration of scenario variations within the ODD.

This stage transforms safety analysis into executable engineering artifacts and satisfy guidelines set forth by ISO/PAS 8800 for requirement generation at different stages of the AI/ML lifecycle, including the data lifecycle, as well as the AI model lifecycle. The scenarios generated can be used at different stages including:

  • Training scenarios

  • Validation datasets

  • Verification test cases

  • Vehicle Validation test cases

6. Model Training Using Safety-Driven Simulation Scenarios

This framework is particularly effective for ML models involved in planning and prediction tasks. For trajectory generation models — such as planners that output spline-based representations of the expected vehicle path — safety-derived scenarios provide explicit guidance for what constitutes acceptable trajectory behavior. Instead of learning arbitrary driving patterns from uncontrolled datasets, imitation learning models can be trained on trajectories generated or validated against defined safety envelopes, including constraints on acceleration, deceleration, curvature, time-to-collision, and minimum distance margins. As a result, the learned spline trajectories inherently encode safe operational behavior consistent with ODD-specific requirements.

Similarly, ML models responsible for actor prediction benefit from safety-driven scenario generation. Prediction models estimate future trajectories or intent of surrounding agents, and their outputs directly influence planning decisions. By deriving training scenarios from behavioral safety analysis, prediction models can be exposed to safety-relevant interactions — such as cut-ins, sudden stops, merging maneuvers, or vulnerable road user behaviors — ensuring that learned predictions remain representative of the operational risks the system must manage. This improves downstream planning reliability by reducing mismatches between predicted actor behavior and safety expectation.

Furthermore, integrating safety into the full ML lifecycle significantly reduces development efforts since the behaviors are baked in from the data used for training and validating models, all the way to the ML model component integration at the vehicle level.

Image 2. Trajectory Generation for an autonomous vehicle

7. Verification and Validation Across the Lifecycle

Behavioral safety scenarios become the backbone of verification and validation:

StageUsageSimulation testingInitial validation of model performanceSoftware-in-the-loopFunctional validationHardware-in-the-loopIntegration testingClosed-course testingControlled real-world validationPublic road testingOperational validation

Each scenario includes measurable expected behavior, enabling objective pass/fail criteria.

8. Alignment with ISO/PAS 8800 (AI Safety Lifecycle)

This approach aligns closely with ISO/PAS 8800 as it places safety-driven scenario engineering at the center of the AI lifecycle, ensuring that model development is guided by safety intent rather than purely data availability. By deriving datasets from safety-relevant scenarios and ODD analysis, training, validation, and test data become traceable to hazard analysis and operational risks, supporting the standard’s expectations for data lifecycle management and demonstrable dataset representativeness. This establishes a clear link between safety requirements and the data used to train and evaluate ML models.

Additionally, embedding safety scenarios throughout the model development and deployment lifecycle supports ISO/PAS 8800’s emphasis on continuous verification. The same scenario framework can be reused to validate learned behavior during training, confirm performance after deployment to target hardware, and verify system-level behavior once integrated into the vehicle. This end-to-end traceability—from safety analysis to data, model behavior, and vehicle-level validation—provides structured evidence that AI components behave safely within the intended operational domain, which is a core principle of ISO/PAS 8800.

9. Benefits and Practical Motivation

A common industry challenge is allowing ML development to proceed independently from safety engineering, resulting in:

  • Misalignment between model behavior and safety expectations.

  • Late-stage redesign or retraining.

  • Increased development cost and risk.

Embedding behavioral safety into the development lifecycle:

  • Reduces iteration cycles.

  • Aligns system engineering and ML workflows.

  • Enables early identification of gaps between intended and learned behavior.

10. Applicability Beyond ML Training

This approach extends beyond ML development:

  • Risk assessment (ISO 26262 and ISO 21448).

  • Test scenario generation.

  • Safety case evidence.

  • Runtime monitoring design.

  • Performance benchmarking.

Conclusion

Behavioral safety provides a practical bridge between traditional automotive safety engineering and modern ML-driven automated driving systems. By defining safe behaviors through systematic scenario analysis and using these scenarios to drive model training and validation, organizations can integrate safety throughout the lifecycle — from data generation to vehicle deployment.

Aligning scenario engineering with ISO 26262, ISO 21448, and ISO/PAS 8800 ensures that safety is not an afterthought but a foundational element guiding system behavior, ML development, and verification.

Summary

  • Behavioral safety defines safe vehicle behavior across ODD scenarios rather than focusing only on failures.

  • Scenario analysis using safety performance indicators translates safety expectations into engineering requirements.

  • Parameterized simulation scenarios drive both ML training and verification.

  • The approach aligns strongly with ISO/PAS 8800 by ensuring ODD-representative datasets and traceable AI lifecycle processes.

  • Integrating safety early reduces development risk and improves ML deployment readiness.