Upload QwenImageLayeredModularPipeline

README.md

@@ -0,0 +1,93 @@
+---
+library_name: diffusers
+tags:
+- modular-diffusers
+- diffusers
+- qwenimage-layered
+- text-to-image
+---
+This is a modular diffusion pipeline built with 🧨 Diffusers' modular pipeline framework.
+
+**Pipeline Type**: QwenImageLayeredAutoBlocks
+
+**Description**: Auto Modular pipeline for layered denoising tasks using QwenImage-Layered.
+
+This pipeline uses a 4-block architecture that can be customized and extended.
+
+## Example Usage
+
+[TODO]
+
+## Pipeline Architecture
+
+This modular pipeline is composed of the following blocks:
+
+1. **text_encoder** (`QwenImageLayeredTextEncoderStep`)
+   - QwenImage-Layered Text encoder step that encode the text prompt, will generate a prompt based on image if not provided.
+   - *resize*: `QwenImageLayeredResizeStep`
+     - Image Resize step that resize the image to a target area (defined by the resolution parameter from user) while maintaining the aspect ratio.
+   - *get_image_prompt*: `QwenImageLayeredGetImagePromptStep`
+     - Auto-caption step that generates a text prompt from the input image if none is provided.
+   - *encode*: `QwenImageTextEncoderStep`
+     - Text Encoder step that generates text embeddings to guide the image generation.
+2. **vae_encoder** (`QwenImageLayeredVaeEncoderStep`)
+   - Vae encoder step that encode the image inputs into their latent representations.
+   - *resize*: `QwenImageLayeredResizeStep`
+     - Image Resize step that resize the image to a target area (defined by the resolution parameter from user) while maintaining the aspect ratio.
+   - *preprocess*: `QwenImageEditProcessImagesInputStep`
+     - Image Preprocess step. Images needs to be resized first.
+   - *encode*: `QwenImageVaeEncoderStep`
+     - VAE Encoder step that converts processed_image into latent representations image_latents.
+   - *permute*: `QwenImageLayeredPermuteLatentsStep`
+     - Permute image latents from (B, C, 1, H, W) to (B, 1, C, H, W) for Layered packing.
+3. **denoise** (`QwenImageLayeredCoreDenoiseStep`)
+   - Core denoising workflow for QwenImage-Layered img2img task.
+   - *input*: `QwenImageLayeredInputStep`
+     - Input step that prepares the inputs for the layered denoising step. It:
+   - *prepare_latents*: `QwenImageLayeredPrepareLatentsStep`
+     - Prepare initial random noise (B, layers+1, C, H, W) for the generation process
+   - *set_timesteps*: `QwenImageLayeredSetTimestepsStep`
+     - Set timesteps step for QwenImage Layered with custom mu calculation based on image_latents.
+   - *prepare_rope_inputs*: `QwenImageLayeredRoPEInputsStep`
+     - Step that prepares the RoPE inputs for the denoising process. Should be place after prepare_latents step
+   - *denoise*: `QwenImageLayeredDenoiseStep`
+     - Denoise step that iteratively denoise the latents. 
+   - *after_denoise*: `QwenImageLayeredAfterDenoiseStep`
+     - Unpack latents from (B, seq, C*4) to (B, C, layers+1, H, W) after denoising.
+4. **decode** (`QwenImageLayeredDecoderStep`)
+   - Decode unpacked latents (B, C, layers+1, H, W) into layer images. 
+
+## Model Components
+
+1. image_resize_processor (`VaeImageProcessor`)
+2. text_encoder (`Qwen2_5_VLForConditionalGeneration`)
+3. processor (`Qwen2VLProcessor`)
+4. tokenizer (`Qwen2Tokenizer`): The tokenizer to use
+5. guider (`ClassifierFreeGuidance`)
+6. image_processor (`VaeImageProcessor`)
+7. vae (`AutoencoderKLQwenImage`)
+8. pachifier (`QwenImageLayeredPachifier`)
+9. scheduler (`FlowMatchEulerDiscreteScheduler`)
+10. transformer (`QwenImageTransformer2DModel`)  ## Input/Output Specification
+
+**Inputs:**
+
+- `image` (`Image | list`): Reference image(s) for denoising. Can be a single image or list of images.
+- `resolution` (`int`, *optional*, defaults to `640`): The target area to resize the image to, can be 1024 or 640
+- `prompt` (`str`, *optional*): The prompt or prompts to guide image generation.
+- `use_en_prompt` (`bool`, *optional*, defaults to `False`): Whether to use English prompt template
+- `negative_prompt` (`str`, *optional*): The prompt or prompts not to guide the image generation.
+- `max_sequence_length` (`int`, *optional*, defaults to `1024`): Maximum sequence length for prompt encoding.
+- `generator` (`Generator`, *optional*): Torch generator for deterministic generation.
+- `num_images_per_prompt` (`int`, *optional*, defaults to `1`): The number of images to generate per prompt.
+- `latents` (`Tensor`, *optional*): Pre-generated noisy latents for image generation.
+- `layers` (`int`, *optional*, defaults to `4`): Number of layers to extract from the image
+- `num_inference_steps` (`int`, *optional*, defaults to `50`): The number of denoising steps.
+- `sigmas` (`list`, *optional*): Custom sigmas for the denoising process.
+- `attention_kwargs` (`dict`, *optional*): Additional kwargs for attention processors.
+- `**denoiser_input_fields` (`None`, *optional*): conditional model inputs for the denoiser: e.g. prompt_embeds, negative_prompt_embeds, etc.
+- `output_type` (`str`, *optional*, defaults to `pil`): Output format: 'pil', 'np', 'pt'.
+
+**Outputs:**
+
+- `images` (`list`): Generated images.