FlowTok: Flowing Seamlessly Across Text and Image Tokens

🔼 This figure compares two approaches to text-to-image generation. The top panel illustrates the conventional method using a diffusion process, where text acts as a conditioning signal to guide the generation from noise to a final image. The bottom panel shows the FlowTok approach. Instead of using diffusion, FlowTok projects both text and images into a shared, low-dimensional (1D) latent space. This allows for direct transformation between the text and image representations, resulting in more efficient and seamless generation of images from text or vice versa.

read the caption

Figure 2: Text as Conditions vs. Direct Flow between Modalities. Top: Conventional text-to-image generation relies on the diffusion process, where text serves as a conditioning signal to guide the denoising process. Bottom: The proposed FlowTok enables direct flow between text and image modalities by projecting both into a shared, compact 1D latent space, facilitating seamless generation of both.

🔼 The figure shows a comparison of the Fréchet Inception Distance (FID) score, a metric for evaluating image quality, against the training cost (measured in 8-A100 GPU days) for various models. It illustrates that FlowTok achieves comparable or better FID scores with significantly reduced training costs compared to other models, highlighting its efficiency.

read the caption

(a) FID vs. Training Costs.

🔼 This figure shows a comparison of different models’ performance on the COCO dataset, specifically focusing on the trade-off between image quality (measured by FID score) and inference speed (images per second). The graph visualizes how various models balance these two factors. Lower FID indicates better image quality, while higher inference speed implies faster image generation.

read the caption

(b) FID vs. Inference Speed.

🔼 Figure 3 displays a comparison of FlowTok’s performance against other state-of-the-art models on the COCO dataset. Subfigure (a) shows that FlowTok achieves comparable FID scores (a measure of image quality) to other models, but requires significantly less training time (measured in 8-A100 GPU days). Subfigure (b) demonstrates that FlowTok’s inference speed (images per second) is considerably faster than competing methods. The superior efficiency of FlowTok results from its novel design using compact 1D tokens for representing both text and images, enabling direct transformation between the two modalities and improving computational efficiency. Note that a competitor model, CrossFlow, utilized high-quality proprietary data, which may influence the results.

read the caption

Figure 3: COCO Results. FlowTok presents comparable performance to previous methods on COCO while significantly reducing training resource requirements (Fig. 3(a)) and achieving much faster sampling speed (Fig. 3(b)). This efficiency stems from its minimalist design centered around 1D tokens, which facilitates direct transformation between text and image modalities, leading to superior performance with enhanced computational efficiency. We note that the compared CrossFlow [53] uses high-quality proprietary data.

🔼 Figure 4 provides a detailed overview of the FlowTok framework. The top half illustrates the text-to-image generation process. Input text is first encoded into a high-dimensional vector representation (Tinit) using a CLIP text encoder. This vector is then projected into a lower-dimensional latent space, creating text tokens (ZT). These text tokens are transformed into image tokens (ZI) of the same dimensions via a flow matching process. Finally, a 1D Image VAE Decoder uses the image tokens to generate the final image. The bottom half demonstrates image-to-text generation. An input image is encoded into image tokens (ZI) by a 1D Image VAE Encoder. Through flow matching, these tokens are mapped to text tokens (ZT) which are then decoded by a text decoder to produce the final text. The figure highlights FlowTok’s efficiency compared to traditional methods, achieving a 3.3x compression rate by using a direct 1D transformation of both text and images, leading to reduced memory usage, faster training, and quicker inference.

read the caption

Figure 4: Overview of FlowTok. FlowTok is a minimal framework that facilitates seamless flow between 1D text tokens and image tokens for both text-to-image and image-to-text generation. Top: For text-to-image generation, the input text is encoded by the CLIP text encoder into 𝐓init∈ℝN×Csubscript𝐓initsuperscriptℝ𝑁𝐶\mathbf{T}_{\text{init}}\in\mathbb{R}^{N\times C}bold_T start_POSTSUBSCRIPT init end_POSTSUBSCRIPT ∈ blackboard_R start_POSTSUPERSCRIPT italic_N × italic_C end_POSTSUPERSCRIPT, projected into a low-dimensional latent space as text tokens 𝐙T∈ℝN×Dsubscript𝐙Tsuperscriptℝ𝑁𝐷\mathbf{Z}_{\text{T}}\in\mathbb{R}^{N\times D}bold_Z start_POSTSUBSCRIPT T end_POSTSUBSCRIPT ∈ blackboard_R start_POSTSUPERSCRIPT italic_N × italic_D end_POSTSUPERSCRIPT, then transformed into image tokens 𝐙I∈ℝN×Dsubscript𝐙Isuperscriptℝ𝑁𝐷\mathbf{Z}_{\text{I}}\in\mathbb{R}^{N\times D}bold_Z start_POSTSUBSCRIPT I end_POSTSUBSCRIPT ∈ blackboard_R start_POSTSUPERSCRIPT italic_N × italic_D end_POSTSUPERSCRIPT of the same shape through flow matching and decoded by a 1D Image VAE Decoder to generate the final image. Bottom: For image-to-text generation, an input image is encoded by a 1D Image VAE Encoder into 𝐙Isubscript𝐙I\mathbf{Z}_{\text{I}}bold_Z start_POSTSUBSCRIPT I end_POSTSUBSCRIPT, mapped to 𝐙Tsubscript𝐙T\mathbf{Z}_{\text{T}}bold_Z start_POSTSUBSCRIPT T end_POSTSUBSCRIPT through flow matching and decoded into text via a text decoder. Unlike conventional approaches that rely on 2D noise and image latents (e.g., 32×32×43232432\times 32\times 432 × 32 × 4 for 256-resolution images) with text as conditions, our direct 1D transformation (i.e., 77×16771677\times 1677 × 16) achieves a 3.3×\times× compression rate, significantly reducing memory costs, accelerating training, and enabling faster inference.

🔼 This figure shows the FID (Fréchet Inception Distance) scores and inference speeds of different models on the COCO dataset. Panel (a) compares the FID scores against the training costs (measured in 8-A100 GPU days), illustrating the efficiency of FlowTok which achieves comparable FID scores with significantly reduced training costs compared to other models like SD-2.1 and CrossFlow. Panel (b) presents the FID scores against the inference speed (images per second), highlighting FlowTok’s superior inference efficiency with throughputs exceeding those of Show-o and CrossFlow.

read the caption

(a)

🔼 This figure shows the comparison of FID scores versus inference speed for various models. FlowTok’s superior efficiency is highlighted by its placement at the top-left. Compared to other methods, FlowTok achieves comparable FID scores while showing significantly faster inference speeds, demonstrating a major improvement in efficiency.

read the caption

(b)

🔼 Table 2 presents a comparison of FlowTok’s performance in zero-shot text-to-image generation against other state-of-the-art models on two benchmark datasets: COCO and MJHQ-30K. The models are grouped into two categories based on their approach: those using text as conditions to guide image generation, and those directly learning the alignment between text and image distributions. The table also indicates whether models were trained solely on publicly available data (‘open-data’), the training cost in terms of 8 A100 GPU-days (‘T’), and inference speed (images per second) at 256px resolution using a single A100 GPU (‘I’). These metrics offer a comprehensive evaluation of both performance and efficiency.

read the caption

Table 2: Zero-Shot Text-to-Image Generation Results on COCO and MJHQ-30K. We compare FlowTok with state-of-the-art methods, categorized into two approaches: (1) text as conditions, where text tokens are used as conditions to guide the generation process, and (2) text as source distributions, where the model directly learns the alignment between text and image distributions. “open-data”: Models are trained exclusively with publicly available datasets. “T”: Model training cost, measured in 8 A100 days using float16 precision. “I”: Model inference throughput, measured at 256px resolution in samples per second on a single A100 with batch size 64 using float16 precision.

🔼 Table 3 presents a comparison of various image-to-text generation models, focusing on their performance on the COCO Karpathy split dataset. To ensure a fair comparison, only non-autoregressive models trained without CIDEr optimization are included in the results. The table shows that FlowTok’s performance is on par with state-of-the-art methods, demonstrating its effectiveness in this task.

read the caption

Table 3: Image-to-Text Generation Results on COCO. FlowTok achieves performance comparable to state-of-the-art methods on image-to-text generation, evaluated on the COCO Karpathy split. For a fair comparison, we restrict our evaluation to non-autoregressive methods trained without CIDEr optimization.

🔼 This table presents ablation studies on the text alignment loss function used in FlowTok, a text-to-image generation model. Three key aspects of the loss function are investigated: the alignment target, the choice of loss function (cosine similarity vs. contrastive loss), and the loss weight. The goal is to determine the optimal configuration for preserving semantic information and improving generation quality. The results are evaluated using the FID-30K metric on the COCO dataset with the FlowTok-B model, without classifier-free guidance (CFG).

read the caption

Table 4: Ablation Studies on Text Alignment Loss. We conduct comprehensive ablation studies on three key aspects of the text alignment loss ℒalignsubscriptℒalign\mathcal{L}_{\text{align}}caligraphic_L start_POSTSUBSCRIPT align end_POSTSUBSCRIPT: the alignment target (Tab. LABEL:tab:ablation_loss_target), the choice of loss function (Tab. LABEL:tab:ablation_loss_function), and the loss weight γ2subscript𝛾2\gamma_{2}italic_γ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT (Tab. LABEL:tab:ablation_loss_weight), aiming to identify the most effective strategy for preserving semantic information within FlowTok during text-to-image generation. For efficient verification, we report FID-30K on COCO using FlowTok-B, without applying the CFG indicator.

🔼 Table 5 details the datasets used to train the FlowTok model, specifically focusing on the filtering criteria applied to each dataset. These criteria include image resolution (aspect ratio less than 2 and the longer side of the image at least 256 pixels), aesthetic quality (images with predicted aesthetic scores above a certain threshold, specified in parentheses for each dataset), and the presence of watermarks (images with a predicted watermark probability above 0.5 were removed). The table notes that a re-captioned version of LAION-pop, provided by MaskGen [41], was used, incorporating improved captions.

read the caption

Table 5: Training Data Details. The filtering criteria applied include resolution (aspect ratio < 2 and longer side ≥\geq≥ 256), aesthetic score (predicted score exceeding the specified value in parentheses), and watermark detection (removal of images predicted to contain watermarks). ††\dagger†: We use the re-captioned version released by MaskGen [41], which contains improved captions.

🔼 This table details the hyperparameters used during the training process of the FlowTok model, providing specific values for both pre-training and fine-tuning phases. It includes settings for the optimizer (AdamW), optimizer parameters (beta1, beta2, weight decay), learning rate, learning rate scheduling, warmup steps, and batch size. The parameters are separated for pre-training and fine-tuning stages, reflecting potential differences in training strategies.

read the caption

Table 6: Training Hyper-parameters for FlowTok.