Flowing from Words to Pixels: A Framework for Cross-Modality Evolution

🔼 CrossFlow is a framework for cross-modal generation that directly maps one modality to another using flow matching without additional conditioning. The figure illustrates the architecture of CrossFlow for text-to-image generation. It uses a Text Variational Encoder (VE) to convert text embeddings from a language model into a latent representation (z0). This latent representation is then evolved using a standard flow matching model into an image latent space (z1) from which the final image is generated. This process bypasses the need for noise distributions and explicit conditioning mechanisms typically found in diffusion or flow-matching models.

read the caption

Figure 2: CrossFlow Architecture. CrossFlow enables direct evolution between two different modalities. Taking text-to-image generation as an example, our T2I model comprises two main components: a Text Variational Encoder and a standard flow matching model. At inference time, we utilize the Text Variational Encoder to extract the text latent z0∈ℝh×w×csubscript𝑧0superscriptℝℎ𝑤𝑐z_{0}\in\mathbb{R}^{h\times w\times c}italic_z start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ∈ blackboard_R start_POSTSUPERSCRIPT italic_h × italic_w × italic_c end_POSTSUPERSCRIPT from text embedding x∈ℝn×d𝑥superscriptℝ𝑛𝑑x\in\mathbb{R}^{n\times d}italic_x ∈ blackboard_R start_POSTSUPERSCRIPT italic_n × italic_d end_POSTSUPERSCRIPT produced by any language model. Then we directly evolve this text latent into the image space to generate image latent z1∈ℝh×w×csubscript𝑧1superscriptℝℎ𝑤𝑐z_{1}\in\mathbb{R}^{h\times w\times c}italic_z start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ∈ blackboard_R start_POSTSUPERSCRIPT italic_h × italic_w × italic_c end_POSTSUPERSCRIPT.

🔼 This figure compares the performance of CrossFlow and a standard flow matching model (baseline) for text-to-image generation. The experiment controls for the amount of training data used, model size (number of parameters), and number of training steps. The left panel shows how performance (measured by FID-30K) improves with larger model sizes for both methods, but CrossFlow demonstrates better scaling and higher performance at larger sizes. The right panel shows how performance changes as a function of training steps. CrossFlow requires more training steps to converge, but ultimately achieves better performance than the baseline model, again demonstrating superior scaling properties. In summary, this figure illustrates that CrossFlow is more efficient in terms of scaling than the baseline method.

read the caption

Figure 3: Performance vs. Model Parameters and Iterations. We compare the baseline of starting from noise with text cross-attention with CrossFlow, while controlling for data, model size and training steps. Left: Larger models are able to exploit the cross-modality connection better. Right: CrossFlow needs more steps to converge, but converges to better final performance. Overall, CrossFlow scales better than the baseline and can serve as the framework for future media generation models.

🔼 This figure demonstrates the smooth transitions achievable with CrossFlow when linearly interpolating between two text-based latent representations. The interpolation showcases seamless changes in various aspects of the generated images, including object orientation, color composition, shapes, backgrounds, and even object types. The visual effect highlights the model’s ability to navigate the latent space smoothly and predictably. Note that due to space constraints, only a subset of the interpolations are displayed here; the full set is included in Appendix C (Fig. 10 and Fig. 11).

read the caption

Figure 4: CrossFlow provides visually smooth interpolations in the latent space. We show images generated by linear interpolation between the first (left) and second (right) text latents. CrossFlow enables visually smooth transformations of object direction, composite colors, shapes, background scenes, and even object categories. Please zoom in for better visualization. For brevity, we display only 7 interpolating images here; additional interpolating images can be found in Appendix C (Fig. 10 and Fig. 11).

🔼 This figure demonstrates the novel capability of CrossFlow to perform arithmetic operations directly within the latent space of text embeddings. A Variational Encoder first maps input text into a latent representation (z0). Then, arithmetic operations (addition and subtraction) are performed on these latent vectors. The results of these operations are then used to generate new images, which reflect the semantic changes induced by the arithmetic. The images generated, along with their corresponding latent codes (z0), are shown to illustrate how simple arithmetic manipulations in the latent space lead to meaningful alterations in the generated images.

read the caption

Figure 5: CrossFlow allows arithmetic in text latent space. Using the Text Variational Encoder (VE), we first map the input text into the latent space z0subscript𝑧0z_{0}italic_z start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT. Arithmetic operations are then performed in this latent space, and the resulting latent representation is used to generate the corresponding image. The latent code z0subscript𝑧0z_{0}italic_z start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT used to generate each image is provided at the bottom.

🔼 This figure shows how CrossFlow directly evolves text into images for text-to-image generation. It contrasts with traditional methods that learn a complex mapping from Gaussian noise to the target image distribution. CrossFlow learns a direct mapping from the distribution of one modality (text) to the distribution of another (image), simplifying the process and eliminating the need for noise or additional conditioning mechanisms.

read the caption

(a)

🔼 This figure shows the various tasks that CrossFlow can perform, highlighting its versatility and generalizability. It demonstrates the ability of CrossFlow to directly evolve one modality (e.g., text) into another (e.g., image) without requiring task-specific architectures. The various tasks shown include text-to-image generation, image captioning, monocular depth estimation, and image super-resolution. Each example visually showcases the successful transformation between modalities using the CrossFlow framework.

read the caption

(b)

🔼 This figure shows the CrossFlow architecture. It is a general framework for cross-modal flow matching. The figure illustrates how CrossFlow directly evolves one modality (e.g., text) to another (e.g., image) using flow matching, without the need for cross-attention or a conditioning mechanism. A variational encoder processes the input data, ensuring the source distribution is compatible with the target distribution. A standard flow matching model, using a vanilla transformer, is then used to transform the source modality into the target modality. This architecture is shown to be efficient and effective across a range of tasks.

read the caption

(c)

🔼 This figure demonstrates the generalizability of the CrossFlow framework by showcasing its performance on various cross-modal and intra-modal tasks. These include text-to-image generation, image captioning, depth estimation, and image super-resolution. Each subfigure shows example results, illustrating that CrossFlow can handle diverse media types and achieve state-of-the-art or comparable performance without requiring task-specific architectures.

read the caption

(d)

🔼 This figure demonstrates the impact of different training strategies on the performance of the model. It compares three approaches: (1) jointly training the variational encoder (VE) and flow matching model from the start; (2) pre-training the VE and then training the flow matching model; and (3) pre-training the VE and then jointly fine-tuning both models. The results show that jointly training both components from the beginning yields the best performance, highlighted by the lowest FID score and highest CLIP score, indicating the best balance between visual fidelity and semantic alignment.

read the caption

(e)

🔼 Table 2 compares the performance of the proposed CrossFlow model against other state-of-the-art text-to-image (T2I) generation models. The comparison is based on two metrics: FID-30K (Frechet Inception Distance, a measure of image quality) and GenEval (a more holistic metric assessing various aspects of text-image alignment). The table highlights that CrossFlow, despite its simpler architecture, achieves comparable performance to significantly larger and more complex models. More detailed GenEval results broken down by specific subtasks are available in Section B.1 of the paper.

read the caption

Table 2: Comparison with recent T2I models. For GenEval, we report the overall score here and provide task-specific scores in Sec. B.1. CrossFlow achieves comparable performance with state-of-the-art T2I models by directly evolving text into images.

🔼 This ablation study analyzes the impact of different design choices in CrossFlow on text-to-image generation performance. Using the smallest model (70 million parameters), the study evaluates various components: the type of text encoder (Variational Encoder vs. standard encoder), the training objective function (reconstruction loss vs. contrastive loss), the application of Classifier-Free Guidance (CFG), different large language models (LLMs), and the training strategy (joint vs. separate training of encoder and flow matching model). Zero-shot FID-10K and CLIP scores are reported to quantify performance. The final configurations used in the CrossFlow model are highlighted.

read the caption

Table 3: Ablation study on Text Variational Encoder, training objective, CFG, language models, and training strategy. We conduct ablation study on our smallest model (70M), reporting zero-shot FID-10K and CLIP scores. Final settings used for CrossFlow are underlined. AG: Autoguidance. *: results without applying CFG.

🔼 Table 4 presents a comparison of different image captioning models on the COCO Karpathy dataset split. The table highlights the performance of CrossFlow, a novel approach that directly maps image data to text, against several state-of-the-art methods. To ensure a fair comparison, only non-autoregressive models trained without CIDEr optimization are included in the comparison. The results demonstrate that CrossFlow achieves comparable performance to existing top-tier models, highlighting its effectiveness in image captioning tasks.

read the caption

Table 4: Image captioning on COCO Karpathy split. CrossFlow directly evolves from image to text, achieving comparable performance to state-of-the-art models on image captioning. For a fair comparison, we only consider non-autoregressive methods that are trained without CIDEr optimization.

🔼 Table 5 presents a comparison of CrossFlow’s performance on monocular depth estimation against other state-of-the-art methods. The results are shown for two benchmark datasets: KITTI and NYUv2. Metrics used for evaluation include Absolute Relative Error (AbsRel) and the δi metric (percentage of pixels with depth error greater than i times the ground truth). Lower AbsRel and higher δi values indicate better performance. The table demonstrates that CrossFlow, despite its relatively simple architecture, achieves comparable accuracy to more complex models, showcasing its effectiveness in directly mapping images to depth.

read the caption

Table 5: Monocular depth estimation on KITTI and NYUv2. CrossFlow enables direct mapping from image to depth, achieving comparable performance to state-of-the-art models.

🔼 Table 6 presents a comparison of image super-resolution results on the ImageNet validation set. It contrasts the performance of a standard super-resolution (SR) method using flow matching with the authors’ novel ‘direct mapping’ approach. The results demonstrate that the direct mapping method proposed in the paper outperforms the standard flow-matching-based SR technique.

read the caption

Table 6: Image super-resolution on the ImageNet validation set. Compared with standard SR method with flow matching, our direct mapping method achieves better performance.

🔼 Table 7 presents a comparison of the CrossFlow model’s performance on the GenEval benchmark against several state-of-the-art text-to-image generation models, including LDM-XL and DALL-E 2. The results demonstrate that CrossFlow achieves comparable performance, showcasing its effectiveness and efficiency as a simpler and promising approach for high-quality media generation.

read the caption

Table 7: GenEval comparisons. Our model achieves comparable performance to state-of-the-art models such as LDM-XL and DALL·E 2, suggesting that CrossFlow is a simple and promising direction for state-of-the-art media generation.

🔼 This table presents a comparison of zero-shot depth estimation performance across different methods. The results are evaluated on five datasets (KITTI, NYUv2, ETH3D, ScanNet, DIODE), and the metrics used are Absolute Relative Error (AbsRel) and the δ1 (81) metric. The authors’ CrossFlow model is compared to several baselines, including results reported by Marigold [39], showing comparable or better performance with CrossFlow’s unified framework approach. The best, second-best, and third-best results for each dataset are highlighted.

read the caption

Table 8: Zero-shot depth estimation. Baseline results are reported by Marigold [39]. We follow Marigold and train our CrossFlow on the same datasets, i.e., Hypersim [72] and Virtual KITTI [9]. We highlight the best, second best, and third best entries. With just a unified framework, CrossFlow achieves comparable or even superior performance on complex zero-shot depth estimation, demonstrating the general-purpose nature of CrossFlow on various cross-modal tasks.

🔼 This table presents an ablation study comparing a standard text encoder against a variational text encoder in a text compression task. The goal was to determine if either method could effectively reduce the dimensionality of text embeddings from 77x768 to 1x1024 while preserving information. The results show that both approaches maintain high reconstruction accuracy despite the significant compression ratio of 14.4 times.

read the caption

Table 9: Ablation on text compression. Both text encoder and Text Variational Encoder preserve most of the input information, despite the large compression ratio (77×768→1×1024→777681102477\times 768\rightarrow 1\times 102477 × 768 → 1 × 1024, 14.4×14.4\times14.4 ×).