RAD: Training an End-to-End Driving Policy via Large-Scale 3DGS-based Reinforcement Learning

🔼 RAD’s training process consists of three stages. First, perception pre-training uses ground truth map and agent data to train instance-level token encoders for scene understanding. Second, planning pre-training leverages large-scale driving demonstrations to initialize the action distribution of the autonomous driving policy. Finally, reinforced post-training uses reinforcement learning (RL) and imitation learning (IL) together to fine-tune the policy, combining the strengths of both approaches to achieve optimal performance. RL handles the causal relationships and the open-loop problem, while IL maintains human-like driving behavior.

read the caption

Figure 2: Overall framework of RAD. RAD takes a three-stage training paradigm. In the perception pre-training, ground-truths of map and agent are used to guide instance-level tokens to encode corresponding information. In the planning pre-training stage, large-scale driving demonstrations are used to initialize the action distribution. In the reinforced post-training stage, RL and IL synergistically fine-tune the AD policy.

🔼 The figure illustrates the post-training stage of the RAD model, where N parallel workers simultaneously interact with the 3DGS environment. Each worker generates a sequence of data, including states (s), actions (a), rewards (r), and next states (s), which is then collected in a buffer. This data, along with human driving demonstrations, are used to refine the driving policy using both reinforcement learning (RL) and imitation learning (IL) techniques, combining their strengths for improved performance.

read the caption

Figure 3: Post-training. N𝑁Nitalic_N workers parallelly run. The generated rollout data (st,at,rt+1,st+1,…)subscript𝑠𝑡subscript𝑎𝑡subscript𝑟𝑡1subscript𝑠𝑡1…(s_{t},a_{t},r_{t+1},s_{t+1},...)( italic_s start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT , italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT , italic_r start_POSTSUBSCRIPT italic_t + 1 end_POSTSUBSCRIPT , italic_s start_POSTSUBSCRIPT italic_t + 1 end_POSTSUBSCRIPT , … ) are recorded in a rollout buffer. Rollout data and human driving demonstrations are used in RL- and IL-training steps to fine-tune the AD policy synergistically.

🔼 Figure 4 illustrates the four reward components used in the reinforcement learning process of the autonomous driving system. Each component penalizes specific unsafe behaviors: 1. Dynamic Collision (𝑟dcr_{ dc}): A negative reward is given if the vehicle collides with a moving obstacle (e.g., another car or pedestrian). 2. Static Collision (𝑟scr_{ sc}): A negative reward is given if the vehicle collides with a stationary obstacle (e.g., a curb or barrier). 3. Positional Deviation (𝑟pdr_{ pd}): A negative reward is given if the vehicle deviates from the optimal trajectory by more than a threshold distance (dmax). 4. Heading Deviation (𝑟hdr_{ hd}): A negative reward is given if the vehicle’s heading angle deviates from the optimal trajectory by more than a threshold angle (ψmax).

read the caption

Figure 4: Example diagram of four types of reward sources. (1): Collision with a dynamic obstacle ahead triggers a reward rdcsubscript𝑟dcr_{\text{dc}}italic_r start_POSTSUBSCRIPT dc end_POSTSUBSCRIPT. (2): Hitting a static roadside obstacle incurs a reward rscsubscript𝑟scr_{\text{sc}}italic_r start_POSTSUBSCRIPT sc end_POSTSUBSCRIPT. (3): Moving onto the curb exceeds the positional deviation threshold dmaxsubscript𝑑maxd_{\text{max}}italic_d start_POSTSUBSCRIPT max end_POSTSUBSCRIPT, triggering a reward rpdsubscript𝑟pdr_{\text{pd}}italic_r start_POSTSUBSCRIPT pd end_POSTSUBSCRIPT. (4): Drifting toward the adjacent lane exceeds the heading deviation threshold ψmaxsubscript𝜓max\psi_{\text{max}}italic_ψ start_POSTSUBSCRIPT max end_POSTSUBSCRIPT, triggering a reward rhdsubscript𝑟hdr_{\text{hd}}italic_r start_POSTSUBSCRIPT hd end_POSTSUBSCRIPT.

🔼 This figure presents a qualitative comparison of the performance of an Imitation Learning (IL)-only policy versus the proposed RAD policy in handling complex driving scenarios within a 3DGS (3D Gaussian Splatting) simulated environment. The scenarios depicted showcase the challenges of yielding to pedestrians (Rows 1-2) and executing unprotected left turns (Rows 3-4). By visually comparing the behaviors of both policies under these conditions, the figure highlights RAD’s superior ability to navigate these complex situations safely and smoothly, while the IL-only approach demonstrates limitations in handling such challenging conditions.

read the caption

Figure 5: Closed-loop qualitative comparisons between the IL-only policy and RAD in 3DGS environments. Rows 1-2 correspond to Yield to Pedestrians. Rows 3-4 correspond to Unprotected Left-turn.

🔼 Figure A1 presents a qualitative comparison of the performance of the IL-only policy and RAD across diverse driving scenarios. The scenarios are displayed in rows, with each row showcasing a different driving situation. Rows 1-2 illustrate a detour maneuver; Rows 3-4 show navigation through dense traffic conditions; Rows 5-6 depict maneuvering through congested traffic; and Rows 7-8 demonstrate successful U-turn execution. For each scenario, the figure displays a sequence of images, allowing visual comparison between the IL-only policy’s and RAD’s performance in terms of smoothness, collision avoidance, and adherence to the driving lanes.

read the caption

Figure A1: More Qualitative Results. Comparison between the IL-only policy and RAD in various driving scenarios: Detour (Rows 1-2), Crawl in Dense Traffic (Rows 3-4), Traffic Congestion (Rows 5-6), and U-turn(Rows 7-8).

🔼 This table presents the results of an ablation study on the reward components used in the RAD model. Each row represents a different configuration, where certain reward components are either included or excluded. The impact of each component on various performance metrics (CR, DCR, SCR, DR, PDR, HDR, ADD, Long. Jerk, Lat. Jerk) is assessed. The goal is to determine the contribution of each reward component in achieving optimal performance, particularly in minimizing collisions and improving the smoothness of driving behavior.

read the caption

Table 2: Ablation on reward sources. The table shows the impact of different reward components on performance.

🔼 This ablation study analyzes the effects of each auxiliary objective (dynamic collision avoidance, static collision avoidance, positional deviation correction, and heading deviation correction) on the overall performance of the autonomous driving policy. It shows how each component contributes to the final results, highlighting their importance in collision avoidance, trajectory tracking, and driving smoothness.

read the caption

Table 3: Ablation on auxiliary objectives. The table shows the impact of different auxiliary objectives on performance.

🔼 This table presents a quantitative comparison of various end-to-end autonomous driving (AD) methods, specifically focusing on those that utilize Imitation Learning (IL). The comparison is based on performance metrics obtained from a rigorous evaluation benchmark involving dense traffic scenarios within a photorealistic 3DGS (3D Gaussian Splatting) simulated environment. The metrics assess collision rates (both dynamic and static), positional and heading deviations from expert trajectories, average deviation distance, and longitudinal and lateral jerk. These metrics provide a comprehensive evaluation of safety, accuracy, and smoothness of the AD policies in challenging driving conditions.

read the caption

Table 4: Closed-loop quantitative comparisons with other IL-based methods on the 3DGS dense-traffic evaluation benchmark.

🔼 This table lists the hyperparameters used in the Planning Pre-training stage of the RAD model. It includes the learning rate, learning rate schedule, optimizer, optimizer hyperparameters, weight decay, batch size, and the number of training steps. The dimensions of the planning head are also specified.

read the caption

Table A1: Hyperparameters used in RAD Planning Pre-Training stage.

🔼 This table lists the hyperparameters used during the reinforced post-training stage of the RAD model. It includes the learning rate and its decay schedule, the optimizer used (AdamW), optimizer hyperparameters (beta1, beta2, epsilon), weight decay, the number of RL workers, RL and IL batch sizes, the GAE parameter (gamma and lambda), clipping thresholds (epsilon_x and epsilon_y), the maximum deviation threshold (dmax and vmax), planning and value function dimensions. These settings are crucial for fine-tuning the model’s policy using both reinforcement and imitation learning.

read the caption

Table A2: Hyperparameters used in RAD Reinforced Post-Training stage.