Mobile Video Diffusion

🔼 This figure demonstrates the impact of optimizing cross-attention mechanisms on mobile device performance. It displays the number of CPU cycles required for the four most computationally expensive operations (MatMul, Softmax, Gelu, and Conv2D) within the UNet architecture, using an input resolution of 128x128 pixels. A key finding highlighted is that removing unnecessary computations related to the similarity map in cross-attention layers leads to an approximately 80% reduction in the number of cycles needed for the Softmax operation, significantly improving efficiency on mobile hardware.

read the caption

Figure 2: Effect of optimized cross-attention for a mobile device. We show the number of cycles of the top-4 operations on mobile hardware for an input resolution of 128×128128128128\times 128128 × 128. Note that removing the no-op similarity map computation in cross-attention layers reduces cycles on softmax operations by roughly 80808080%.

🔼 This figure illustrates the concept of ‘channel funnels,’ a technique used to optimize the model’s efficiency. Channel funnels act as adaptors during training, reducing the model’s width (number of channels) and thus the number of parameters. This speeds up training without a significant impact on accuracy. Crucially, at inference time, the funnels are merged with the weight matrices, so there is no loss in accuracy or quality of the final output. The diagram visually shows the process, highlighting how the model’s width is reduced during training and restored during inference.

read the caption

Figure 3: Channel funnels. We show an example of channel funnels applied to a couple of layers within the model. At training time, funnels serve as adaptors reducing model width. At inference, they are merged with corresponding weight matrices without loss of quality.

🔼 This figure shows the architecture of temporal blocks in Stable Video Diffusion (SVD). Panel (a) illustrates how the original SVD model incorporates temporal blocks to handle the temporal dynamics of videos. Each temporal block receives inputs from the spatial blocks, processes them through temporal layers, and then integrates the result with the spatial block output via a weighted sum, controlled by the parameter ‘a’, allowing for variable emphasis on spatial or temporal features.

read the caption

(a) Temporal blocks in the original architecture of SVD.

🔼 During training, a random binary gate (0 or 1) is applied to each temporal block. This gate determines whether the temporal block is active (gate=1) or inactive (gate=0) during that specific training iteration. This technique is used to perform a learnable pruning of temporal blocks where the network learns which temporal blocks are more important for generating high-quality videos. At inference time, only the most important temporal blocks (those with the highest learned importance scores) are used, resulting in a smaller and more efficient model.

read the caption

(b) A zero-one gate multiplier is sampled to each temporal block during training.

🔼 This table shows the impact of various optimizations on the performance and efficiency of a Stable Video Diffusion model adapted for mobile devices. The optimizations include lowering resolution, using optimized cross-attention, adversarial finetuning, temporal multiscaling, temporal block pruning, and channel funneling. The table shows the number of function evaluations (NFE), Fréchet Video Distance (FVD) values at both 25 frames per second (FPS) and 7 FPS, TeraFLOPS (TFLOPS), and latency in milliseconds (ms) on both a GPU and a mobile phone. The bottom row represents the final, optimized Mobile Video Diffusion (MobileVD) model.

read the caption

Table 2: Effect of our optimizations. We successfully deployed the image-to-video model to a mobile device without significantly sacrificing the visual quality. FLOPs and latency are provided for a single function evaluation with batch size of 1. We call the model in the bottom row Mobile Video Diffusion, or MobileVD.

🔼 This table presents ablation study results on the impact of using multi-scale representations within the UNet architecture on mobile video diffusion model performance. It shows the effects of adding either spatial or temporal downsampling layers in the UNet, or both. The results are measured by FID (Fréchet Inception Distance) and TFLOPs, reflecting the trade-off between image quality and computational efficiency. The key takeaway is that while either spatial or temporal multi-scaling improves performance on mobile devices, combining both methods leads to a noticeable drop in image quality.

read the caption

Table 3: Effect of additional multiscaling layers in UNet. We observe that both temporal and spatial multiscaling has good impact on mobile latency without compromising much on FVD, while combining the two increases FVD by a noticeable amount.

🔼 This table investigates the impact of two hyperparameters—channel funnel initialization methods (Coupled Singular Initialization (CSI) vs. He initialization) and the funnel factor—on the Fréchet Video Distance (FVD) metric. The results demonstrate that using CSI is significantly better than He initialization for achieving lower FVD scores. Furthermore, decreasing the funnel factor below 0.5 leads to a decline in model performance.

read the caption

Table 4: Effect of funnel initialization and fun-factor. Initialization funnels with CSI is crucial to getting good FVD as He initialization [20] obtains roughly 200 FVD units more. Additionally, we see that reducing the fun-factor beyond 0.5 starts to affect the performance.

🔼 This table compares different methods for reducing the width of neural network layers, aiming to improve model efficiency. It contrasts channel funneling (the proposed method), finetuned low-rank approximation of attention layers using truncated singular value decomposition, and channel funneling applied to convolutional layers instead of attention layers. The ‘fun-factor’, representing the reduction rate, is indicated for channel funneling.

read the caption

Table 5: Comparison of model width reduction methods. We compare the proposed channel funneling (in grey) with finetuned low-rank approximation of individual attention layers with truncated singular decomposition. We additionally compare to Funnels applied to convolutions instead of attention. The reduction rate (referred to as fun-factor in case of funnels) is highlighted with r𝑟ritalic_r.

🔼 This table compares the performance of different temporal block pruning methods. Specifically, it shows how a learnable pruning technique outperforms L1 regularization, which lacks explicit control over the number of pruned blocks. The experiment starts with a model optimized up to the temporal block pruning stage. The results demonstrate the effectiveness of the proposed method in reducing computational cost while maintaining model performance.

read the caption

Table 6: Impact of temporal blocks pruning. Our pruning outperforms the L1−limit-fromsubscript𝐿1L_{1}-italic_L start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT - regularization which does not have explicit control over the number of removed blocks. We use the checkpoint, optimized up to the temporal block pruning stage, as the starting point.

🔼 This table compares the performance of the proposed MobileVD model with several state-of-the-art video generation models. It includes the number of function evaluations (NFE), Fréchet Video Distance (FVD), TeraFLOPs (TFLOPS), and latency (in milliseconds) on both GPU and mobile phone. The comparison is done for both high-resolution (1024x576) and low-resolution (512x256) settings. Note that FVD values marked with an asterisk (*) were obtained from an external source (Zhang et al., 2024), while the other performance metrics are based on the authors’ measurements of the UNet used in Stable Video Diffusion (SVD). To ensure consistency, the FVD for SVD and MobileVD was measured on the UCF-101 dataset at 7 frames per second.

read the caption

Table 7: Comparison with recent models. The set of optimizations proposed in our paper, can also be applied to high-resolution generation. FLOPs and latency are provided for a single function evaluation with batch size of 1. For rows marked with asterisk∗ FVD measurements were taken from Zhang et al. [71], while performance metrics are based on our measurements for UNet used by SVD. For consistency with these results, FVD for SVD and our models was measured on UCF-101 dataset at 7 frames per second.

🔼 This table presents a quantitative analysis of the impact of various optimization techniques on the performance of a mobile video diffusion model. It compares the original Stable Video Diffusion (SVD) model with several optimized versions, showcasing the trade-offs between computational cost (measured in FLOPs and latency on both GPU and mobile phone), and visual quality (evaluated using the Fréchet Video Distance (FVD) and JEDi metrics). The optimizations include: low-resolution finetuning, optimized cross-attention, adversarial finetuning, temporal multi-scaling, temporal block pruning, and channel funneling. The table highlights the significant reduction in computational cost achieved by the MobileVD model (the final optimized version) while maintaining relatively good visual quality, demonstrating its suitability for mobile deployment. A high-resolution variant, MobileVD-HD, is also included, showcasing the scalability of the optimizations.

read the caption

Table 8: Effect of our optimizations. We successfully deployed the image-to-video model to a mobile device without significantly sacrificing the visual quality. FLOPs and latency are provided for a single function evaluation with batch size of 1. We call the model in the bottom row Mobile Video Diffusion, or MobileVD. The model trained with the same hyperparameters but intended for high-resolution generations is referred to as MobileVD-HD.

🔼 This table compares the performance of two different decoders used to convert latent codes generated by the MobileVD model into actual RGB video frames. It shows that the TAESD decoder is significantly faster for on-device processing compared to the original decoder from SVD, while maintaining virtually identical visual quality as measured by the FVD and JEDi metrics. This indicates that the TAESD decoder is a suitable and more efficient alternative for mobile applications.

read the caption

Table 9: Impact of latent decoder. While being significantly faster on device, decoder from TAESD has little to no impact on visual quality as measured by FVD and JEDi.