GoalFlow: Goal-Driven Flow Matching for Multimodal Trajectories Generation in End-to-End Autonomous Driving

🔼 GoalFlow’s architecture is composed of three modules: the Perception Module, the Goal Point Construction Module, and the Trajectory Planning Module. The Perception Module takes image and LiDAR data as input, processing them to generate a Bird’s Eye View (BEV) feature representation, denoted as Fbev. This BEV feature is then passed to the Goal Point Construction Module. This module utilizes a Goal Point Vocabulary (a set of potential goal points) to identify the optimal goal point for trajectory generation, using a scoring mechanism. The selected goal point is then given to the Trajectory Planning Module, which uses a denoising process to generate multiple trajectory candidates starting from a Gaussian distribution and converging towards the target distribution. These candidates are then evaluated by a Trajectory Scorer, which selects the single best trajectory.

read the caption

Figure 2: Overview of the GoalFlow architecture. GoalFlow consists of three modules. The Perception Module is responsible for integrating scene information into a BEV feature Fb⁢e⁢vsubscript𝐹𝑏𝑒𝑣F_{bev}italic_F start_POSTSUBSCRIPT italic_b italic_e italic_v end_POSTSUBSCRIPT, the Goal Point Construction Module selects the optimal goal point from Goal Point Vocabulary 𝕍𝕍\mathbb{V}blackboard_V as guidance information, and the Trajectory Planning Module generates the trajectories by denoising from the Gaussian distribution to the target distribution. Finally, the Trajectory Scorer selects the optimal trajectory from the candidates.

🔼 This figure shows the detailed structure of the Goal Point Construction Module. It visually depicts how the module processes the BEV feature (Fbev), ego status (Fego), and the goal point vocabulary (V) to select the optimal goal point. The figure highlights the key components: the vocabulary encoder, the scorer decoder (using MLPs), and the final goal point selection process. The accompanying text (b) illustrates the distributions of distance score (dis), drivable area compliance score (dac), and final score (final). Warmer colors indicate higher scores.

read the caption

(a)

🔼 This figure shows the score distributions of the Distance Score (d_is), Drivable Area Compliance Score (d_ac), and the final score (f_final) for goal point selection. Warmer colors indicate higher scores. The Distance Score measures how close a goal point is to the ground truth. The Drivable Area Compliance Score checks if the goal point is within a drivable area. The final score combines both scores to select the best goal point.

read the caption

(b)

🔼 Figure 3 illustrates the Goal Point Construction Module, a key component of the GoalFlow architecture. Part (a) provides a detailed diagram of this module’s structure, showing how it uses a transformer-based scorer decoder to generate distance and drivable area compliance scores for each potential goal point. Part (b) visualizes the resulting score distributions (distance score, drivable area compliance score, and final score), using a color gradient to represent the magnitude of each score—warmer colors indicating higher scores. This visualization helps to understand how the module effectively selects the most appropriate goal point for trajectory generation.

read the caption

Figure 3: Goal Point Scorer. (a) shows the detailed structure of the Goal Point Construction Module, and (b) presents the score distributions of {δ^id⁢i⁢s}Nsuperscriptsubscriptsuperscript^𝛿𝑑𝑖𝑠𝑖𝑁\{\hat{\delta}^{dis}_{i}\}^{N}{ over^ start_ARG italic_δ end_ARG start_POSTSUPERSCRIPT italic_d italic_i italic_s end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT } start_POSTSUPERSCRIPT italic_N end_POSTSUPERSCRIPT, {δ^id⁢a⁢c}Nsuperscriptsubscriptsuperscript^𝛿𝑑𝑎𝑐𝑖𝑁\{\hat{\delta}^{dac}_{i}\}^{N}{ over^ start_ARG italic_δ end_ARG start_POSTSUPERSCRIPT italic_d italic_a italic_c end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT } start_POSTSUPERSCRIPT italic_N end_POSTSUPERSCRIPT, and {δ^if⁢i⁢n⁢a⁢l}Nsuperscriptsubscriptsuperscript^𝛿𝑓𝑖𝑛𝑎𝑙𝑖𝑁\{\hat{\delta}^{final}_{i}\}^{N}{ over^ start_ARG italic_δ end_ARG start_POSTSUPERSCRIPT italic_f italic_i italic_n italic_a italic_l end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT } start_POSTSUPERSCRIPT italic_N end_POSTSUPERSCRIPT, where points with higher scores are highlighted with warmer color.

🔼 This figure illustrates the architecture of the Rectified Flow model employed in GoalFlow for trajectory generation. It shows how the model takes different inputs including time step (t), the current state (xt), and features from the environment (Fenv) and ego vehicle (Fego) to generate a trajectory. These features are processed through various encoder and transformer layers to produce a shift vector (vt), used for transitioning the trajectory from a noise distribution to a target distribution. The various components such as the encoder, transformer layers, and positional encoding are clearly depicted to show the process of trajectory prediction using the rectified flow framework.

read the caption

Figure 4: The network architecture used in Rectified Flow.

🔼 This table presents an ablation study analyzing the impact of different components on the model’s performance. The base model (ℳ₀) uses rectified flow without goal point guidance, averaging all generated trajectories for the final output. Adding the distance score map (ℳ₁) improves the model by providing distance information for guidance. Further enhancement comes from incorporating the drivable area compliance (DAC) score map (ℳ₂), which adds constraints to keep trajectories within drivable areas. Finally, adding a trajectory scorer (ℳ₃) refines the selection process for the optimal trajectory among the candidates.

read the caption

Table 2: Ablation study on the influence of each component. ℳ0subscriptℳ0\mathcal{M}_{0}caligraphic_M start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT is the base model, which uses rectified flow without goal point guidance and averages all generated trajectories to produce the final output. ℳ1subscriptℳ1\mathcal{M}_{1}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT and ℳ2subscriptℳ2\mathcal{M}_{2}caligraphic_M start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT introduce the distance score map and DAC score map, respectively, to guide the rectified flow. ℳ3subscriptℳ3\mathcal{M}_{3}caligraphic_M start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT builds upon ℳ1subscriptℳ1\mathcal{M}_{1}caligraphic_M start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT by incorporating trajectory scorer.

🔼 This table presents an ablation study on the impact of varying the number of denoising steps (T) during inference on the performance of the GoalFlow model. It shows that the model’s performance metrics (SNC, SDAC, STTC, SCF, SEP, and SPDM) remain relatively stable across a wide range of denoising steps from 20 to 1, demonstrating the model’s robustness to this hyperparameter. This is a key finding, highlighting the efficiency of GoalFlow, as it requires only a few denoising steps to achieve excellent results. The inference time is also shown, demonstrating the significant reduction in computation time as the number of steps decreases.

read the caption

Table 3: Impact of different timesteps in inference. T𝑇Titalic_T denotes the number of denoising steps during inference. The results indicate that the model’s performance is robust to variations of denoising steps.

🔼 This table presents an ablation study analyzing the effect of varying the standard deviation (σ) of the initial noise distribution (x₀) on the performance of the GoalFlow model. The standard deviation (σ) controls the amount of noise added to the initial state of the trajectory generation process. The results demonstrate that model performance is relatively robust to changes in σ when σ is below 0.1. However, as σ increases beyond 0.1, the performance drops significantly, indicating that too much initial noise negatively impacts the model’s ability to generate high-quality trajectories.

read the caption

Table 4: Impact of different values of σ𝜎\sigmaitalic_σ on the initial noise distribution. σ𝜎\sigmaitalic_σ is the standard deviation of x0subscript𝑥0x_{0}italic_x start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT. The results show that performance drops significantly when σ𝜎\sigmaitalic_σ exceeds 0.1, but remains stable for values below 0.1.

🔼 This table presents an ablation study on the impact of scaling the transformer’s hidden dimension and changing the image backbone in the GoalFlow model. The results show that increasing the hidden dimension and using a larger backbone (V2-99 vs. ResNet-34) generally lead to performance improvements, as measured by various metrics like SNC, SDAC, STTC, SEP, and SPDM. The table isolates the effect of trajectory post-processing by comparing results to the performance of model M2, which includes all other components except the changes made for this experiment.

read the caption

Table 5: Impact of Scaling Model. We examine the impact of scaling the Transformer’s hidden dimension and changing the image backbone. Both increasing the hidden dimension and using a larger backbone improve performance. To isolate the effect of trajectory post-processing, we compare with ℳ2subscriptℳ2\mathcal{M}_{2}caligraphic_M start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT.