VidTok: A Versatile and Open-Source Video Tokenizer

🔼 This figure presents a high-level overview of the typical architecture of a video tokenizer. The diagram illustrates the key components and the flow of information during the tokenization and reconstruction process. A video input is first fed into an encoder, which compresses the video into a compact representation within a latent space. Within this latent space, a regularizer is often applied to enforce certain properties or constraints, such as sparsity or smoothness. Finally, a decoder takes the regularized latent representation and reconstructs the original video. Depending on the specific model, the latent space can be either continuous or discrete, and the architecture may operate in a causal or non-causal manner.

read the caption

Figure 2: An overview of video tokenizers.

🔼 The figure presents the architecture of the video tokenizer, VidTok. It visually illustrates the flow of data through the model, highlighting the components responsible for temporal and spatial compression and decompression. The model takes a video input of size T×H×W (Time × Height × Width) and compresses it into a latent representation of smaller dimensions using an encoder with 2D convolutions and an AlphaBlender operator. It then reconstructs the original video from the latent representation using a decoder with a similar structure, including 2D convolutions and AlphaBlender. In the context of causal video processing, additional considerations are introduced. The first frame is processed differently to allow the model to function for single images as well. The diagram also depicts the specific modules used for temporal upsampling and downsampling. The improved model architecture uses a combination of 2D and 3D convolutional layers to reduce computational demands without compromising performance.

read the caption

Figure 3: The improved model architecture. In the context of a causal setting, consider an input with dimensions T×H×W=17×256×256𝑇𝐻𝑊17256256T\times H\times W=17\times 256\times 256italic_T × italic_H × italic_W = 17 × 256 × 256. Assuming a temporal compression factor of 4444 and a spatial compression factor of 8888, the intermediate latent representation is reduced to dimensions T×H×W=5×32×32𝑇𝐻𝑊53232T\times H\times W=5\times 32\times 32italic_T × italic_H × italic_W = 5 × 32 × 32.

🔼 The figure provides a visual comparison of two quantization methods used in the proposed video tokenizer model, VidTok. Vector Quantization (VQ), shown on the left, maps latent vectors to entries in a learned codebook. This is the method used in VQ-VAE (Vector Quantised-Variational AutoEncoder). Finite Scalar Quantization (FSQ), shown on the right, directly quantizes each scalar value in the latent space to predefined discrete levels and does not require learning a codebook. VidTok uses FSQ. FSQ enhances training stability and improves codebook utilization.

read the caption

Figure 4: Left: Vector Quantization (VQ) employed in Vector Quantised-Variational AutoEncoder (VQ-VAE) (Van Den Oord et al., 2017). Right: Finite Scalar Quantization (FSQ) (Mentzer et al., 2024) utilized in our model.

🔼 Qualitative comparison of VidTok with state-of-the-art video tokenizers, showcasing the effectiveness of VidTok in reconstructing video frames with high fidelity. The comparison includes various state-of-the-art models such as OmniTokenizer, Cosmos-DV, CV-VAE, Open-Sora-v1.2, Open-Sora-Plan-v1.2, and CogVideoX, across a range of video content. The reconstructed frames by each model are juxtaposed with the original frames, allowing a clear assessment of their performance.

read the caption

Figure 5: Qualitative comparison with the state-of-the-art video tokenizers.

🔼 This figure shows the impact of using different frame rates for the training data on the performance of a video tokenizer model. The top row displays the ground truth frames. The middle row shows the reconstructed frames when the model was trained with data at 8 frames per second (FPS). The bottom row shows the results when trained with data at 3 FPS. The comparison suggests that training with a lower frame rate (3 FPS) leads to better capturing of motion dynamics, resulting in improved reconstruction, especially noticeable in the fox’s legs and tail. This observation implies that a lower frame rate during training might encourage the model to focus more on the overall motion flow rather than fine-grained details within individual frames.

read the caption

Figure 6: The influence of different sample rates on model performance during training. The second row presents the test results obtained using training data with a sample rate of 8 FPS, while the third row shows the test results using training data with a sample rate of 3 FPS. The results demonstrate that employing training data with reduced frame rates enhances the model’s capacity to effectively capture motion dynamics.

🔼 This table presents an ablation study on the model architecture for video tokenization, comparing the performance of four different architectures. All models are trained using ‘KL - 4chn’ as regularizer. Variant 1 uses a fully 3D architecture for spatial and temporal sampling. Variant 2 separates spatial and temporal sampling using 2D and 1D convolutions but without AlphaBlender. Variant 3 replaces all 3D convolutions with 2D convolutions. The proposed Ours model utilizes a combination of 2D and 1D convolutions with the AlphaBlender operator for temporal upsampling and downsampling, while other parts leverage 3D convolutions for information fusion. The table compares the number of parameters (Param.), FLOPS (floating point operations per second), Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM), Learned Perceptual Image Patch Similarity (LPIPS), and Fréchet Video Distance (FVD). The results demonstrate that the proposed architecture strikes a balance between computational efficiency and reconstruction performance.

read the caption

Table 2: Ablation study on the model architecture. Variant 1: fully 3D architecture. Variant 2: w/o AlphaBlender. Variant 3: w/o 3D architecture. We use ‘KL - 4chn’ as regularizer for all settings.

🔼 This table shows the ablation study conducted on discrete techniques including VQ, LFQ, and FSQ, demonstrating FSQ’s significant advantages over existing discrete tokenization techniques in terms of codebook utilization, reconstruction quality, and training stability. The presence or absence of regularization loss is also considered during the ablation analysis.

read the caption

Table 3: Analysis of the impact of discrete techniques on model performance. R.L. denotes Regularization Loss, while U.R. represents Utilization Rate.

🔼 This table presents the ablation study on the two-stage training strategy proposed in the paper. The first stage involves training the full model on low-resolution videos (128x128), while the second stage involves fine-tuning only the decoder on high-resolution videos (256x256). The table compares different training settings, including single-stage training on high-resolution videos, two-stage training with or without fixed encoder, and varying sample rates. The results demonstrate that the two-stage training strategy with a fixed encoder achieves similar performance to single-stage training on high resolution but at roughly half the training time.

read the caption

Table 4: Ablation study on the proposed training strategy. To ensure a fair comparison, both stages use training data from Training Set 1. Across all configurations, the regularizer ‘KL - 4chn’ is employed. The training computational cost, measured in GPU hours, is evaluated using NVIDIA A100 GPUs.

🔼 This table provides a comprehensive overview of the different VidTok model configurations. It shows variations based on continuous vs. discrete tokenization, different video compression ratios (how much the video is compressed), whether the model is causal (processes information sequentially) or non-causal (processes information in any order), the latent space size, model parameters, and performance metrics (PSNR, SSIM, LPIPS, and FVD). This variety allows users to select the best model for their specific needs, whether it’s high-fidelity reconstruction or efficient compression.

read the caption

Table 5: Model summary. We offer a suite of models with diverse configurations, encompassing both continuous and discrete tokenization, various video compression ratios (VCR), and options for causal and non-causal scenarios. These configurations are designed to address the distinct requirements of various downstream tasks.