BrushEdit: All-In-One Image Inpainting and Editing

🔼 BrushEdit’s model architecture takes a mask and a masked image as input and outputs an inpainted image. It involves downsampling the mask, encoding the masked image with VAE, concatenating noisy latent, masked image latent, and downsampled mask as BrushEdit input, adding extracted BrushEdit features to pre-trained UNet layers (using a zero convolution block), denoising, and blending the generated and masked images with a blurred mask.

read the caption

Figure 2: Model overview. Our model outputs an inpainted image given the mask and masked image input. Firstly, we downsample the mask to accommodate the size of the latent, and input the masked image to the VAE encoder to align the distribution of latent space. Then, noisy latent, masked image latent, and downsampled mask are concatenated as the input of BrushEdit. The feature extracted from BrushEdit is added to pretrained UNet layer by layer after a zero convolution block[33]. After denoising, the generated image and masked image are blended with a blurred mask.

🔼 Figure 3 provides a visual overview of the benchmarks used for evaluating the BrushEdit model. The top two rows (I and II) showcase examples from BrushBench, featuring both natural and artificially generated images alongside their corresponding masks and captions. These examples cover various scenarios, including humans, animals, indoor, and outdoor settings. Each image in these rows is accompanied by two masks: one for ‘inside’ inpainting, where the inner part of the image is masked, and one for ‘outside’ inpainting, where the outer area is masked. The bottom row (III) displays samples from EditBench, another benchmark used for testing inpainting performance. It includes both generated images and natural images with their respective masks and captions. All images shown are randomly selected from their respective benchmarks.

read the caption

Figure 3: Benchmark overview. I and II separately show natural and artificial images, masks, and caption of BrushBench. (a) to (d) show images of humans, animals, indoor scenarios, and outdoor scenarios. Each group of images shows the original image, inside-inpainting mask, and outside-inpainting mask, with an image caption on the top. III show image, mask, and caption from EditBench [32], with (e) for generated images and (f) for natural images. The images are randomly selected from both benchmarks.

🔼 This figure visually compares existing image editing methods (Prompt-to-Prompt, MasaCtrl, pix2pix-zero, and Plug-and-Play with DDIM and PnP inversion) with BrushEdit. It demonstrates various editing operations like removing, adding, modifying, and swapping objects on both natural and synthetic images. Specifically, row I shows removing a flower from a dog’s mouth and a laptop from a table. Row II presents adding a collar to a dog and flowers to a woman’s hair. Row III depicts changing a jacket to a blouse and a wreath to a crown. Lastly, row IV showcases replacing dumplings with sushi and a car with a motorcycle. BrushEdit exhibits better coherence between edited and unedited regions, closer adherence to editing instructions, smoother transitions at mask boundaries, and higher overall consistency.

read the caption

Figure 4: Comparison of previous editing methods and BrushEdit on natural and synthetic images, covering image editing operations such as removing objects (I), adding objects (II), modifying attributes (III), and swapping objects (IV).

🔼 Figure 5 compares the output of BrushEdit and other image inpainting models on various inpainting tasks and masks, including random and segmentation masks. Each example includes the original masked image and outputs from multiple methods: BLD, SDI, HDP, PP, CNI, BrushNet, and BrushEdit. The figure showcases BrushEdit’s ability to produce more coherent and contextually appropriate inpainted regions compared to other state-of-the-art inpainting models across various scenes, objects, and masking conditions.

read the caption

Figure 5: Performance comparisons of BrushEdit and previous image inpainting methods across various inpainting tasks: (I) Random Mask Inpainting (II) Segmentation Mask Inpainting. Each group of results contains 7777 inpainting methods: (b) Blended Latent Diffusion (BLD) [27], (c) Stable Diffusion Inpainting (SDI) [5], (d) HD-Painter (HDP) [30], (e) PowerPaint (PP) [29], (f) ControlNet-Inpainting (CNI) [33], (g) Our Previous BrushNet and (h) Ours.

🔼 Figure 6 demonstrates the integration of BrushEdit with five popular community-fine-tuned Stable Diffusion 1.5 models: DreamShaper, epiCRealism, Henmix_Real, MeinaMix (specifically designed for anime), and Realistic Vision. The figure showcases how BrushEdit, through its flexible design, allows these diverse models to perform image editing tasks while preserving background details and adhering to the textual instructions.

read the caption

Figure 6: Integrating BrushEdit to community fine-tuned diffusion models. We use five popular community diffusion models fine-tuned from stable diffusion v1.5: DreamShaper (DS) [99], epiCRealism (ER) [100], Henmix_Real (HR) [101], MeinaMix (MM) [102], and Realistic Vision (RV) [103]. MM is specifically designed for anime images.

🔼 Figure 7 demonstrates how BrushEdit’s control scale parameter, w, allows adjustment of the balance between preserving the original content of the unmasked regions and applying the edits specified in the text prompt. Subfigures (b) through (h) show the output images as w decreases from 1.0 to 0.2. A larger w value prioritizes preserving the original image, leading to more subtle edits. Conversely, a smaller w value emphasizes the edits, potentially at the cost of altering more of the original content. This demonstrates that w acts as a slider for how strongly the edits are applied, giving users fine-grained control over the editing process.

read the caption

Figure 7: Flexible control scale of BrushEdit. (a) shows the given masked image, (b)-(h) show adding control scale w𝑤witalic_w from 1.01.01.01.0 to 0.20.20.20.2. Results show a gradually diminishing controllable ability from precise to rough control.

🔼 This table presents a quantitative comparison of BrushEdit with various other image editing methods using the PnPBench dataset. The comparison includes inversion-based methods like Prompt-to-Prompt, MasaCtrl, Pix2Pix-Zero, and Plug-and-Play, each evaluated with two inversion techniques (DDIM and PnP). The metrics used for evaluation are PSNR, LPIPS, MSE, SSIM, and CLIP Similarity. The best results are highlighted in red, and the second-best results are highlighted in blue, demonstrating BrushEdit’s superior performance in preserving unedited regions and ensuring accurate text alignment in edited areas.

read the caption

TABLE II: Comparison of BrushEdit with various editing methods in PnpBench. For editing methods Prompt-to-Prompt (P2P)[8], MasaCtrl[9], Pix2Pix-Zero (P2P-Zero)[9], and Plug-and-Play (PnP)[66], we evaluate two inversion techniques, DDIM Inversion (DDIM)[2] and PnP Inversion (PnP)[11], to establish stronger baselines. Red stands for the best result, Blue stands for the second best result.

🔼 This table compares the inference time of BrushEdit with other inversion-based editing methods on image editing tasks. It demonstrates that BrushEdit achieves comparable or better results while using significantly less inference time. Specifically, BrushEdit takes 3.57 seconds per image edit, while other methods range from 18.22 to 382.98 seconds.

read the caption

TABLE III: Comparison of inference time between our inpainting-based BrushEdit  and other inversion-based methods, including Negative-Prompt Inversion (NP), Edit Friendly Inversion (EF), AIDI[98], EDICT, Null-Text Inversion (NT), and Style Diffusion added with Prompt-to-Prompt. BrushEdit achieves better editing results with far less inference time than all inversion-based methods.

🔼 Comparison of BrushEdit and other diffusion-based inpainting models on BrushBench for inside- and outside-inpainting tasks, evaluating background fidelity and text alignment using Stable Diffusion v1.5. Metrics include PSNR, MSE, LPIPS, SSIM, and CLIP Similarity. Higher values indicate better performance for PSNR, SSIM, CLIP Similarity. Lower values indicate better performance for MSE, LPIPS.

read the caption

TABLE IV: Quantitative comparisons between BrushEdit and other diffusion-based inpainting models in BrushBench: Blended Latent Diffusion (BLD)[27], Stable Diffusion Inpainting (SDI)[5], HD-Painter (HDP)[30], PowerPaint (PP)[29], ControlNet-Inpainting (CNI)[33], and our previous Segmentation-based BrushNet-Seg[22]. The table shows metrics on background fidelity and text alignment (Text Align) for both inside- and outside-inpainting. All models use Stable Diffusion V1.5 as the base model. Red indicates the best result, while Blue indicates the second-best result.

🔼 This table presents a quantitative comparison of various image inpainting models using the EditBench dataset with random masks. The comparison includes Blended Latent Diffusion (BLD), Stable Diffusion Inpainting (SDI), HD-Painter (HDP), PowerPaint (PP), ControlNet-Inpainting (CNI), BrushNet-Ran, and the proposed method BrushEdit. The table evaluates the models based on background fidelity metrics (PSNR, MSE, LPIPS, SSIM) and text alignment using CLIP Similarity. The results demonstrate that BrushEdit significantly improves the preservation of masked regions, outperforming other methods, thanks to its novel dual-branch framework and combined mask training strategy.

read the caption

TABLE V: Quantitative comparisons among BrushEdit and other diffusion-based inpainting models, Random-mask-based BrushNet-Ran in EditBench. A detailed explanation of compared methods and metrics can be found in the caption of Tab. IV. Red stands for the best result, Blue stands for the second best result.

🔼 This table presents an ablation study on BrushEdit’s dual-branch vs. single-branch design for image inpainting. Stable Diffusion Inpainting (SDI) uses a single-branch design where the entire UNet is fine-tuned. Two dual-branch models were evaluated: one with the base UNet fine-tuned and one with the base UNet frozen. Results show superior performance with the dual-branch design, with freezing the base UNet providing the best balance of performance and model flexibility.

read the caption

TABLE VI: Ablation on dual-branch design. Stable Diffusion Inpainting (SDI) use single-branch design, where the entire UNet is fine-tuned. We conducted an ablation analysis by training a dual-branch model with two variations: one with the base UNet fine-tuned, and another with the base UNet forzened. Results demonstrate the superior performance achieved by adopting the dual-branch design. Red is the best result.

🔼 This table presents the ablation study results for different model architectures on image inpainting tasks. It investigates the impact of varying components such as the image encoder (using a randomly initialized convolutional layer or a VAE), mask inclusion, the presence of cross-attention layers, the method of UNet feature addition (full, half, or like ControlNet), and blending operation (none, direct pasting, or blurred blending).

read the caption

TABLE VII: Ablation on model architecture. We ablate on the following components: the image encoder (Enc), selected from a random initialized convolution (Conv) and a VAE; the inclusion of mask in input (Mask), chosen from adding (w/) and not adding (w/o); the presence of cross-attention layers (Attn), chosen from adding (w/) and not adding (w/o); the type of UNet feature addition (UNet), selected from adding the full UNet feature (full), adding half of the UNet feature (half), and adding the feature like ControlNet (CN); and finally, the blending operation (Blend), chosen from not adding (w/o), direct pasting (paste), and blurred blending (blur). Red is the best result.