Upload folder using huggingface_hub

.gitattributes

@@ -53,8 +53,7 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 "/audio_vae_nf4.safetensors" filter=lfs diff=lfs merge=lfs -text
 "/minimax-h3-fl2va-nf4.safetensors" filter=lfs diff=lfs merge=lfs -text
 "/minimax-h3-ref2va-nf4.safetensors" filter=lfs diff=lfs merge=lfs -text
-"/minimax-h3-text-encoder-nf4.safetensors" filter=lfs diff=lfs merge=lfs -textaudio_vae_nf4.safetensors filter=lfs diff=lfs merge=lfs -text
-minimax-h3-fl2va-nf4.safetensors filter=lfs diff=lfs merge=lfs -text
-minimax-h3-ref2va-nf4.safetensors filter=lfs diff=lfs merge=lfs -text
-minimax-h3-text-encoder-nf4.safetensors filter=lfs diff=lfs merge=lfs -text
-video_vae_nf4.safetensors filter=lfs diff=lfs merge=lfs -text
+"/minimax-h3-text-encoder-nf4.safetensors" filter=lfs diff=lfs merge=lfs -text
+"/minimax-h3-fl2va-pruned-nf4.safetensors" filter=lfs diff=lfs merge=lfs -text
+"/minimax-h3-ref2va-pruned-nf4.safetensors" filter=lfs diff=lfs merge=lfs -textminimax-h3-fl2va-pruned-nf4.safetensors filter=lfs diff=lfs merge=lfs -text
+minimax-h3-ref2va-pruned-nf4.safetensors filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,15 +1,9 @@
 ---
 license: apache-2.0
-base_model:
-- MiniMaxAI/MiniMax-H3
-frameworks:
-- ""
-base_model_relation: quantized
 ---
-
 # MiniMax-H3-NF4
 
-This model is the **NF4 quantized version** of the video generation model [MiniMax-H3](https://modelscope.cn/models/MiniMax/MiniMax-H3). It utilizes the `bitsandbytes` 4-bit quantization scheme and is designed to be used with [DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio), enabling model inference on devices with limited VRAM and RAM.
+This model is the **NF4 quantized version** of the video generation model [MiniMax-H3](https://modelscope.cn/models/MiniMax/MiniMax-H3), using the `bitsandbytes` 4-bit quantization method. When used with [DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio), it enables model inference on devices with limited GPU memory and system RAM.
 
 ## Environment Setup
 
@@ -21,11 +15,11 @@ pip install -e ".[all]"
 
 ## Inference Code
 
-### Enable VRAM Management
+### Enable Memory Management
 
-Run the following code to perform inference using DiffSynth-Studio. VRAM management will be automatically enabled. The actual VRAM usage depends on the available VRAM on your GPU; a minimum of 8GB VRAM is required to run.
+Run the following code to perform inference using DiffSynth-Studio. Memory management will be automatically enabled. The actual GPU memory usage depends on available GPU memory, and the model can run with as little as 8GB of GPU memory.
 
-#### FL2VA (Text-to-Video/Audio):
+FL2VA:
 
 ```python
 import torch
@@ -44,6 +38,7 @@ vram_config = {
     "computation_dtype": torch.bfloat16,
     "computation_device": "cuda",
 }
+
 pipe = MiniMaxH3Pipeline.from_pretrained(
     torch_dtype=torch.bfloat16,
     device="cuda",
@@ -68,11 +63,11 @@ write_video_audio(
 )
 ```
 
-#### Ref2VA (Reference-to-Video/Audio):
+Ref2VA:
 
 <details>
 
-<summary>Expand Code</summary>
+<summary>Expand code</summary>
 
 ```python
 import torch
@@ -127,7 +122,7 @@ pipe = MiniMaxH3Pipeline.from_pretrained(
 # Text + Reference Image -> Video + Audio
 dataset_snapshot_download(dataset_id="DiffSynth-Studio/diffsynth_example_dataset", local_dir="data/diffsynth_example_dataset", allow_file_pattern="minimax_h3/MiniMax-H3-Ref2VA/*")
 ref_image = Image.open("data/diffsynth_example_dataset/minimax_h3/MiniMax-H3-Ref2VA/0.png").convert("RGB")
-prompt = "A website page, website UI design, website animation, video showing smooth webpage scrolling effect. A highly explosive and dynamic product official website style product landing page UI/UX demo video, the core display subject is product image 1. The page uses bold, powerful, tilted oversized sans-serif fonts for flamboyant typography. The background features dynamic light and shadow with extreme speed sense, dark carbon fiber or sports breathable mesh textures interweaving and changing. The video shows a tight-paced, powerful webpage downward scrolling effect, as well as strong visual zoom and color inversion UI interaction actions when hovering the mouse."
+prompt = "A webpage, website UI design, web animation. The video demonstrates a smooth scrolling-down effect. A highly dynamic and energetic product homepage-style landing page UI/UX demo video, with the main focus being product image 1. The page features bold, slanted, oversized sans-serif typography for striking layout. The background contains fast-moving light effects combined with dynamically shifting dark carbon fiber or sport breathable mesh textures. The video shows a tightly paced, powerful scrolling-down animation, along with strong UI interactions such as visual zoom-in and color inversion on mouse hover."
 video, audio = pipe(
     prompt=prompt,
     height=480, width=832, num_frames=124, num_inference_steps=50, seed=42,
@@ -158,9 +153,152 @@ write_video_audio(
 
 </details>
 
+[Comfy-Org/MiniMax-H3](https://www.modelscope.cn/models/Comfy-Org/MiniMax-H3) provides a pruned variant of MiniMax-H3, which replaces the DiT's timestep embedding MLP with a lookup table, reducing the input dimension of `adaln_proj.linear` from 2688 to 8 and bringing the total number of model parameters down to approximately 20B. We also provide NF4 quantized weights for this version, with each DiT file being around 9.8 GB.
+
+Usage is identical to the above; simply replace the DiT's `origin_file_pattern` with the pruned version. The text encoder and both VAEs still reuse the same set of weights.
+
+FL2VA:
+
+<details>
+
+<summary>Expand code</summary>
+
+```python
+import torch
+from diffsynth.pipelines.minimax_h3_audio_video import MiniMaxH3Pipeline, ModelConfig
+from diffsynth.utils.data.audio_video import write_video_audio
+from modelscope import dataset_snapshot_download
+from PIL import Image
+
+vram_config = {
+    "offload_dtype": "disk",
+    "offload_device": "disk",
+    "onload_dtype": torch.bfloat16,
+    "onload_device": "cpu",
+    "preparing_dtype": torch.bfloat16,
+    "preparing_device": "cuda",
+    "computation_dtype": torch.bfloat16,
+    "computation_device": "cuda",
+}
+pipe = MiniMaxH3Pipeline.from_pretrained(
+    torch_dtype=torch.bfloat16,
+    device="cuda",
+    model_configs=[
+        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="minimax-h3-fl2va-pruned-nf4.safetensors", **vram_config),
+        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="minimax-h3-text-encoder-nf4.safetensors", **vram_config),
+        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="video_vae_nf4.safetensors", **vram_config),
+        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="audio_vae_nf4.safetensors", **vram_config),
+    ],
+    processor_config=ModelConfig(model_id="MiniMax/MiniMax-H3", origin_file_pattern="FL2VA/processor/"),
+    vram_limit=torch.cuda.mem_get_info("cuda")[1] / (1024 ** 3) - 4,
+)
+```
+
+prompt = "A girl is very happy, she is speaking in english: “I enjoy working with Diffsynth-Studio, it's a perfect framework.”"
+video, audio = pipe(
+    prompt=prompt,
+    height=480, width=832, num_frames=124, num_inference_steps=50, seed=0,
+)
+write_video_audio(
+    video=video, audio=audio,
+    output_path="t2va.mp4", fps=24, audio_sample_rate=32000,
+)
+```
+
+</details>
+
+Ref2VA:
+
+<details>
+
+<summary>Expand code</summary>
+
+```python
+import torch
+from PIL import Image
+from diffsynth.pipelines.minimax_h3_audio_video import MiniMaxH3Pipeline, ModelConfig
+from diffsynth.utils.data.audio_video import write_video_audio
+from diffsynth.utils.data.audio import read_audio
+from diffsynth.utils.data import VideoData
+from modelscope import dataset_snapshot_download
+
+def align_frame_count(frame_count):
+    current = max(int(frame_count), 1)
+    while current % 17 != 5:
+        current += 1
+    return current
+
+def read_video_with_fps(path, num_out_frames, height, width, fps=24):
+    video = VideoData(path, height=height, width=width)
+    frames = video.raw_data()
+    src_fps = float(video.data.reader.get_meta_data()["fps"])
+    out = []
+    for k in range(num_out_frames):
+        idx = int(round(k * src_fps / fps))
+        if idx >= len(frames):
+            break
+        out.append(frames[idx])
+    return out
+
+vram_config = {
+    "offload_dtype": "disk",
+    "offload_device": "disk",
+    "onload_dtype": torch.bfloat16,
+    "onload_device": "cpu",
+    "preparing_dtype": torch.bfloat16,
+    "preparing_device": "cuda",
+    "computation_dtype": torch.bfloat16,
+    "computation_device": "cuda",
+}
+pipe = MiniMaxH3Pipeline.from_pretrained(
+    torch_dtype=torch.bfloat16,
+    device="cuda",
+    model_configs=[
+        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="minimax-h3-ref2va-pruned-nf4.safetensors", **vram_config),
+        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="minimax-h3-text-encoder-nf4.safetensors", **vram_config),
+        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="video_vae_nf4.safetensors", **vram_config),
+        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="audio_vae_nf4.safetensors", **vram_config),
+    ],
+    processor_config=ModelConfig(model_id="MiniMax/MiniMax-H3", origin_file_pattern="Ref2VA/processor/"),
+    vram_limit=torch.cuda.mem_get_info("cuda")[1] / (1024 ** 3) - 5,
+)
+
+# Text + Reference Image -> Video + Audio
+dataset_snapshot_download(dataset_id="DiffSynth-Studio/diffsynth_example_dataset", local_dir="data/diffsynth_example_dataset", allow_file_pattern="minimax_h3/MiniMax-H3-Ref2VA/*")
+ref_image = Image.open("data/diffsynth_example_dataset/minimax_h3/MiniMax-H3-Ref2VA/0.png").convert("RGB")
+prompt = "A website page, web UI design, website motion effects. The video demonstrates a smooth scrolling-down effect on a webpage. An intense, dynamic product landing page UI/UX demonstration video with strong visual impact, centered around Product Image 1. The layout features bold, slanted, oversized sans-serif typography. The background includes dynamic lighting effects conveying speed, along with evolving dark carbon fiber or athletic mesh textures. The video showcases a fast-paced, powerful scrolling animation, as well as strong UI interactions such as dramatic visual zooming and color inversion on mouse hover."
+video, audio = pipe(
+    prompt=prompt,
+    height=480, width=832, num_frames=124, num_inference_steps=50, seed=42,
+    references=[{"type": "image", "image": Image.open("data/diffsynth_example_dataset/minimax_h3/MiniMax-H3-Ref2VA/0.png").convert("RGB")}]
+)
+write_video_audio(
+    video=video, audio=audio,
+    output_path="ti2va.mp4", fps=24, audio_sample_rate=32000,
+)
+
+# Text + Reference Audio + Reference Video -> Video + Audio
+ref_video = read_video_with_fps("data/diffsynth_example_dataset/minimax_h3/MiniMax-H3-Ref2VA/video.mp4", 124, 480, 832)
+ref_audio, sample_rate = read_audio("data/diffsynth_example_dataset/minimax_h3/MiniMax-H3-Ref2VA/voice.mp3", duration=len(ref_video) / 24, resample=True, resample_rate=pipe.audio_vae.sample_rate)
+prompt = "subject_definitions:\n<Subject 1> is the young man with short wavy blonde hair, wearing a bright pink suit jacket, matching pink trousers, an unbuttoned white shirt, and silver rings, holding a small black lamb in his arms in <Video 1>.\n<Video 1> is the source video for the editing task.\n<Audio 1> is the synchronized audio track of <Video 1>, providing the background music.\n<Audio 2> is the voice timbre reference for <Subject 1>'s voice, containing a spoken male voiceover.\n\nsummary:\n[video editing + audio reference + audio reuse] The target video is an edited version of <Video 1>. <Subject 1>, wearing a bright pink suit and holding a black lamb, stands in a grassy field with other white lambs in the background. The edit animates <Subject 1>'s face to speak the user-provided dialogue. <Audio 1> is partially reused as the continuous background music, while the target references the calm male voice timbre of <Audio 2> for <Subject 1>'s spoken lines.\n\nretention_analysis:\n<Subject 1> (appears in [Shot 1]): fully_preserved - the man retains his identity, wavy blonde hair, pink suit, white shirt, accessories, and the black lamb he holds, with his mouth newly animated to speak.\n<Video 1> (source video editing): fully_preserved - the original camera framing, warm golden hour lighting, grassy hill setting, and background white lambs are maintained while the central character is edited.\n<Audio 1>: partially_copy - the atmospheric background music from <Audio 1> is reused in the target video, mixed beneath the newly added spoken dialogue.\n<Audio 2>: reference - the target audio references the male voice timbre from <Audio 2> to generate <Subject 1>'s spoken dialogue.\n\ndetailed_description:\nThe target video is in realistic photographic style.\n[Shot 1] The shot begins from the source <Video 1>, showing <Subject 1>, a young man with short wavy blonde hair, wearing a bright pink suit jacket, matching pink trousers, and a casually unbuttoned white shirt. He stands confidently in a sunlit green pasture, gently holding a small black lamb securely in his arms. The warm, golden hour lighting casts soft shadows across his face and the bright pink fabric of his suit. Behind him, several white lambs stand and graze on the rolling grassy hill against a clear, pale blue sky. The atmospheric background music from <Audio 1> plays continuously throughout the scene. <Subject 1> physically speaks, his mouth movements naturally syncing to the new dialogue, with his voice timbre referencing the calm male delivery from <Audio 2>. Looking thoughtfully forward, <Subject 1> (S1) speaks softly, <d>[English] Follow the wind, live free.</d> As he delivers the line, he subtly shifts his weight, cradling the resting black lamb while the camera slowly pushes in. <Subject 1> (S1) continues his thought, <d>[English] Leave worries behind, enjoy the moment.</d> Exactly as his voice stops, his lips meet in a relaxed, peaceful smile, and his jaw ceases speaking motion. He then turns his gaze slightly away toward the horizon, gently stroking the black lamb's fleece with his fingers as the camera holds on this tranquil, sunlit state through the end of the video.\n\noverall_soundscape:\nThe soundscape consists of the continuous, atmospheric background music from <Audio 1>, overlaid with the clear, calm male dialogue spoken by the main character, referencing the voice timbre of <Audio 2>.\n\nnon_diegetic_music:\nThe atmospheric, sustained background music from <Audio 1> is reused as the continuous score, playing quietly beneath the spoken dialogue."
+video, audio = pipe(
+    prompt=prompt,
+    height=480, width=832, num_frames=124, num_inference_steps=50, seed=42,
+    references=[
+        {"type": "video", "video": ref_video},
+        {"type": "audio", "audio": ref_audio, "sample_rate": sample_rate},
+    ],
+)
+write_video_audio(
+    video=video, audio=audio,
+    output_path="tav2va.mp4", fps=24, audio_sample_rate=32000,
+)
+
+</details>
+
 ### Extreme Hardware Optimization
 
-If your computing device has extremely limited performance, we support enabling direct disk-to-VRAM loading. With this configuration, tensors in the model are loaded from disk to VRAM one by one according to the computation order. This allows the model to run with only 8GB of RAM:
+If your computing hardware has extremely limited performance, we support enabling direct disk-to-GPU-memory access. With this configuration, tensors in the model will be loaded sequentially from disk into GPU memory as needed for computation, allowing the model to run with only 8GB of RAM:
 
 ```diff
 vram_config = {
@@ -182,7 +320,7 @@ pipe = MiniMaxH3Pipeline.from_pretrained(
 )
 ```
 
-We also support running model inference on Mac M-series chips, although this is not recommended:
+If you wish to run model inference on Mac M-series chips, we also provide support—although this is not recommended:
 
 ```diff
 vram_config = {
@@ -206,15 +344,15 @@ pipe = MiniMaxH3Pipeline.from_pretrained(
 
 ## Training Code
 
-This quantized model supports LoRA training. Please follow the steps below to start the training program.
+This quantized model supports LoRA training. Follow the steps below to launch the training job.
 
-Download the sample dataset:
+Download the example dataset:
 
 ```shell
 modelscope download --dataset DiffSynth-Studio/diffsynth_example_dataset --include "minimax_h3/MiniMax-H3-FL2VA/*" --local_dir ./data/diffsynth_example_dataset
 ```
 
-**Training configuration suitable for Data Center GPUs (e.g., Nvidia H20):** Run the following script to start the LoRA training program. Requires 48GB VRAM.
+Training configuration suitable for data center GPUs (e.g., Nvidia H20): Run the following script to start LoRA training, which requires 48GB of GPU memory.
 
 ```shell
 accelerate launch examples/minimax_h3/model_training/train.py \
@@ -238,7 +376,7 @@ accelerate launch examples/minimax_h3/model_training/train.py \
   --find_unused_parameters
 ```
 
-**Training configuration suitable for Consumer GPUs (e.g., Nvidia RTX 4090):** Run the following scripts to start two-stage split training with gradient checkpointing offload. Requires 24GB VRAM.
+Training configuration suitable for consumer-grade GPUs (e.g., Nvidia RTX 4090): Run the above script to start a two-stage split training with gradient checkpointing and offload enabled, requiring 24GB of VRAM.
 
 ```shell
 accelerate launch examples/minimax_h3/model_training/train.py \
@@ -289,4 +427,4 @@ accelerate launch examples/minimax_h3/model_training/train.py \
 * DiffSynth-Studio Documentation: [Minimax-H3](https://diffsynth-studio-doc.readthedocs.io/en/latest/Model_Details/MiniMax-H3.html)
 * DiffSynth-Studio Documentation: [VRAM Management](https://diffsynth-studio-doc.readthedocs.io/en/latest/Pipeline_Usage/VRAM_management.html)
 * DiffSynth-Studio Documentation: [Two-Stage Split Training](https://diffsynth-studio-doc.readthedocs.io/en/latest/Training/Split_Training.html)
-* DiffSynth-Studio Documentation: [Low VRAM Training](https://diffsynth-studio-doc.readthedocs.io/en/latest/Pipeline_Usage/Model_Training.html#low-vram-training)
+* DiffSynth-Studio Documentation: [Low VRAM Training](https://diffsynth-studio-doc.readthedocs.io/en/latest/Pipeline_Usage/Model_Training.html#low-vram-training)

README_from_modelscope.md

@@ -1,51 +1,32 @@
----
+---	
 base_model:
+ 
 - MiniMax/MiniMax-H3
 frameworks:
 - ""
 license: Apache License 2.0
 base_model_relation: quantized
 ---
-# MiniMax-H3-NF4
-
-MiniMax-H3 多模态音视频生成模型的 **NF4 量化版本**（通过 `bitsandbytes` 4-bit 量化），配合 [DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio) 使用，可在显存/内存受限的机器上进行「文本/图像/视频/音频 → 视频 + 音频」的联合生成。
-
-## 文件说明
-
-| 文件 | 大小 | 作用 | 是否共用 |
-|---|---|---|---|
-| `minimax-h3-fl2va-nf4.safetensors` | ~16 GB | **FL2VA** 任务的 DiT 主干（文本 / 首尾关键帧 → 视频+音频） | FL2VA 专用 |
-| `minimax-h3-ref2va-nf4.safetensors` | ~16 GB | **Ref2VA** 任务的 DiT 主干（参考图像/视频/音频 → 视频+音频） | Ref2VA 专用 |
-| `minimax-h3-text-encoder-nf4.safetensors` | ~15 GB | Qwen3-VL 文本/视觉编码器 | 两任务共用 |
-| `video_vae_nf4.safetensors` | ~1.6 GB | 视频 VAE 解码器 | 两任务共用 |
-| `audio_vae_nf4.safetensors` | ~271 MB | 音频 VAE 解码器 | 两任务共用 |
 
-> 说明：DiT 按任务二选一，其余三个（text_encoder / video_vae / audio_vae）在两种任务下通用。加载时框架会根据文件 hash 自动识别组件类型并套用对应的量化配置（含对少数量化敏感层的 bf16 保留），无需手动指定量化参数。
-
-## 环境要求
-
-- CUDA GPU（NF4 反量化依赖 `bitsandbytes` 的 CUDA kernel）
-- processor / tokenizer 需从原始仓库 `MiniMax/MiniMax-H3` 获取（下面 `processor_config`）
+# MiniMax-H3-NF4
 
-### 安装 DiffSynth-Studio
+本模型是视频生成模型 [MiniMax-H3](https://modelscope.cn/models/MiniMax/MiniMax-H3) 的 **NF4 量化版本**，采用 `bitsandbytes` 4-bit 量化方案，配合 [DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio) 使用，可以在显存和内存有限的设备上运行模型推理。
 
-从源码安装（推荐，可获得最新的 MiniMax-H3 支持），并直接带上 NF4 量化依赖：
+## 环境安装
 
-```bash
+```shell
 git clone https://github.com/modelscope/DiffSynth-Studio.git
 cd DiffSynth-Studio
-pip install -e ".[quant]"
+pip install -e ".[all]"
 ```
 
-## 使用（Disk offload，低显存和内存占用）
+## 推理代码
 
-权重存放磁盘、推理时按层流式加载到 GPU，显存占用最低。**纯文本生成视频+音频（t2v）最低约 6 GB 显存即可运行。**
+### 启用显存管理
 
-> `vram_limit`（单位 GB）是显存占用阈值，调小可降低显存占用（代价是更慢）。
+运行以下代码，使用 DiffSynth-Studio 进行推理，显存管理将会自动启用，实际的显存占用量取决于 GPU 上的可用显存，最低 8G 显存即可运行。
 
-> 若 CPU 内存充足，可把 `offload_device` / `offload_dtype` 改为 `"cpu"` / `torch.bfloat16`（即 CPU offload），权重常驻内存、不走磁盘，速度更快；其余代码不变。
-
-### FL2VA — 文本 / 首尾关键帧 → 视频+音频
+FL2VA:
 
 ```python
 import torch
@@ -64,6 +45,7 @@ vram_config = {
     "computation_dtype": torch.bfloat16,
     "computation_device": "cuda",
 }
+
 pipe = MiniMaxH3Pipeline.from_pretrained(
     torch_dtype=torch.bfloat16,
     device="cuda",
@@ -74,10 +56,9 @@ pipe = MiniMaxH3Pipeline.from_pretrained(
         ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="audio_vae_nf4.safetensors", **vram_config),
     ],
     processor_config=ModelConfig(model_id="MiniMax/MiniMax-H3", origin_file_pattern="FL2VA/processor/"),
-    vram_limit=torch.cuda.mem_get_info("cuda")[1] / (1024 ** 3) - 2,
+    vram_limit=torch.cuda.mem_get_info("cuda")[1] / (1024 ** 3) - 4,
 )
 
-# Text -> Video + Audio
 prompt = "A girl is very happy, she is speaking in english: “I enjoy working with Diffsynth-Studio, it's a perfect framework.”"
 video, audio = pipe(
     prompt=prompt,
@@ -87,34 +68,157 @@ write_video_audio(
     video=video, audio=audio,
     output_path="t2va.mp4", fps=24, audio_sample_rate=32000,
 )
+```
+
+Ref2VA:
+
+<details>
+
+<summary>展开代码</summary>
+
+```python
+import torch
+from PIL import Image
+from diffsynth.pipelines.minimax_h3_audio_video import MiniMaxH3Pipeline, ModelConfig
+from diffsynth.utils.data.audio_video import write_video_audio
+from diffsynth.utils.data.audio import read_audio
+from diffsynth.utils.data import VideoData
+from modelscope import dataset_snapshot_download
+
+def align_frame_count(frame_count):
+    current = max(int(frame_count), 1)
+    while current % 17 != 5:
+        current += 1
+    return current
 
-# Text + First Frame + Last Frame -> Video + Audio
-dataset_snapshot_download(dataset_id="DiffSynth-Studio/diffsynth_example_dataset", local_dir="data/diffsynth_example_dataset", allow_file_pattern="minimax_h3/MiniMax-H3-FL2VA/*")
-first_frame = Image.open("data/diffsynth_example_dataset/minimax_h3/MiniMax-H3-FL2VA/first.png")
-last_frame = Image.open("data/diffsynth_example_dataset/minimax_h3/MiniMax-H3-FL2VA/last.png")
-prompt = "室内���庭争吵短剧场景，竖屏短剧质感，真实真人表演，中式家庭/小饭馆室内环境，暖色灯光，背景有红色装饰和书法字幅，浅景深，情绪强烈，剪辑节奏紧凑。表演要求：真实短剧表演风格，不要夸张舞台腔。男人的语气是愤怒、委屈、急切的反驳，他说“你到底想干什么？”；中老年女性的语气是尖锐、强势、咄咄逼人的质问，她说“你必须赔钱！”。两人之间有强烈对峙感，节奏逐步升级。画面风格：竖屏9:16，手机短剧质感，真人实拍感，浅景深，室内暖光，中近景为主，频繁正反打剪辑，背景保持生活化，不要科幻、不要古装、不要动画感。画面中不要出现任何字幕、文字、平台水印或贴片。 "
+def read_video_with_fps(path, num_out_frames, height, width, fps=24):
+    video = VideoData(path, height=height, width=width)
+    frames = video.raw_data()
+    src_fps = float(video.data.reader.get_meta_data()["fps"])
+    out = []
+    for k in range(num_out_frames):
+        idx = int(round(k * src_fps / fps))
+        if idx >= len(frames):
+            break
+        out.append(frames[idx])
+    return out
+
+vram_config = {
+    "offload_dtype": "disk",
+    "offload_device": "disk",
+    "onload_dtype": torch.bfloat16,
+    "onload_device": "cpu",
+    "preparing_dtype": torch.bfloat16,
+    "preparing_device": "cuda",
+    "computation_dtype": torch.bfloat16,
+    "computation_device": "cuda",
+}
+pipe = MiniMaxH3Pipeline.from_pretrained(
+    torch_dtype=torch.bfloat16,
+    device="cuda",
+    model_configs=[
+        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="minimax-h3-ref2va-nf4.safetensors", **vram_config),
+        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="minimax-h3-text-encoder-nf4.safetensors", **vram_config),
+        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="video_vae_nf4.safetensors", **vram_config),
+        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="audio_vae_nf4.safetensors", **vram_config),
+    ],
+    processor_config=ModelConfig(model_id="MiniMax/MiniMax-H3", origin_file_pattern="Ref2VA/processor/"),
+    vram_limit=torch.cuda.mem_get_info("cuda")[1] / (1024 ** 3) - 5,
+)
+
+# Text + Reference Image -> Video + Audio
+dataset_snapshot_download(dataset_id="DiffSynth-Studio/diffsynth_example_dataset", local_dir="data/diffsynth_example_dataset", allow_file_pattern="minimax_h3/MiniMax-H3-Ref2VA/*")
+ref_image = Image.open("data/diffsynth_example_dataset/minimax_h3/MiniMax-H3-Ref2VA/0.png").convert("RGB")
+prompt = "一个网站页面，网站页面UI设计，网站动效，视频展示了流畅的网页向下滚动效果。一个极具爆发力与动感的产品官网风格产品落地页 UI/UX 演示视频，核心展示主体是该产品图片1。页面采用粗犷有力、倾斜的超大号无衬线字体进行张扬的排版。背景有极具速度感的动态光影、暗色碳纤维或运动透气网眼纹理在交织变换。视频展示了节奏紧凑、充满力量感的网页向下滚动效果，以及鼠标悬停时强烈的视觉放大与颜色反转等 UI 交互动作。"
+video, audio = pipe(
+    prompt=prompt,
+    height=480, width=832, num_frames=124, num_inference_steps=50, seed=42,
+    references=[{"type": "image", "image": Image.open("data/diffsynth_example_dataset/minimax_h3/MiniMax-H3-Ref2VA/0.png").convert("RGB")}]
+)
+write_video_audio(
+    video=video, audio=audio,
+    output_path="ti2va.mp4", fps=24, audio_sample_rate=32000,
+)
+
+# Text + Reference Audio + Reference Video -> Video + Audio
+ref_video = read_video_with_fps("data/diffsynth_example_dataset/minimax_h3/MiniMax-H3-Ref2VA/video.mp4", 124, 480, 832)
+ref_audio, sample_rate = read_audio("data/diffsynth_example_dataset/minimax_h3/MiniMax-H3-Ref2VA/voice.mp3", duration=len(ref_video) / 24, resample=True, resample_rate=pipe.audio_vae.sample_rate)
+prompt = "subject_definitions:\n<Subject 1> is the young man with short wavy blonde hair, wearing a bright pink suit jacket, matching pink trousers, an unbuttoned white shirt, and silver rings, holding a small black lamb in his arms in <Video 1>.\n<Video 1> is the source video for the editing task.\n<Audio 1> is the synchronized audio track of <Video 1>, providing the background music.\n<Audio 2> is the voice timbre reference for <Subject 1>'s voice, containing a spoken male voiceover.\n\nsummary:\n[video editing + audio reference + audio reuse] The target video is an edited version of <Video 1>. <Subject 1>, wearing a bright pink suit and holding a black lamb, stands in a grassy field with other white lambs in the background. The edit animates <Subject 1>'s face to speak the user-provided dialogue. <Audio 1> is partially reused as the continuous background music, while the target references the calm male voice timbre of <Audio 2> for <Subject 1>'s spoken lines.\n\nretention_analysis:\n<Subject 1> (appears in [Shot 1]): fully_preserved - the man retains his identity, wavy blonde hair, pink suit, white shirt, accessories, and the black lamb he holds, with his mouth newly animated to speak.\n<Video 1> (source video editing): fully_preserved - the original camera framing, warm golden hour lighting, grassy hill setting, and background white lambs are maintained while the central character is edited.\n<Audio 1>: partially_copy - the atmospheric background music from <Audio 1> is reused in the target video, mixed beneath the newly added spoken dialogue.\n<Audio 2>: reference - the target audio references the male voice timbre from <Audio 2> to generate <Subject 1>'s spoken dialogue.\n\ndetailed_description:\nThe target video is in realistic photographic style.\n[Shot 1] The shot begins from the source <Video 1>, showing <Subject 1>, a young man with short wavy blonde hair, wearing a bright pink suit jacket, matching pink trousers, and a casually unbuttoned white shirt. He stands confidently in a sunlit green pasture, gently holding a small black lamb securely in his arms. The warm, golden hour lighting casts soft shadows across his face and the bright pink fabric of his suit. Behind him, several white lambs stand and graze on the rolling grassy hill against a clear, pale blue sky. The atmospheric background music from <Audio 1> plays continuously throughout the scene. <Subject 1> physically speaks, his mouth movements naturally syncing to the new dialogue, with his voice timbre referencing the calm male delivery from <Audio 2>. Looking thoughtfully forward, <Subject 1> (S1) speaks softly, <d>[English] Follow the wind, live free.</d> As he delivers the line, he subtly shifts his weight, cradling the resting black lamb while the camera slowly pushes in. <Subject 1> (S1) continues his thought, <d>[English] Leave worries behind, enjoy the moment.</d> Exactly as his voice stops, his lips meet in a relaxed, peaceful smile, and his jaw ceases speaking motion. He then turns his gaze slightly away toward the horizon, gently stroking the black lamb's fleece with his fingers as the camera holds on this tranquil, sunlit state through the end of the video.\n\noverall_soundscape:\nThe soundscape consists of the continuous, atmospheric background music from <Audio 1>, overlaid with the clear, calm male dialogue spoken by the main character, referencing the voice timbre of <Audio 2>.\n\nnon_diegetic_music:\nThe atmospheric, sustained background music from <Audio 1> is reused as the continuous score, playing quietly beneath the spoken dialogue."
 video, audio = pipe(
     prompt=prompt,
-    height=832, width=480, num_frames=124, num_inference_steps=50, seed=0,
-    keyframes=[first_frame, last_frame], keyframe_indices=[0, -1],
+    height=480, width=832, num_frames=124, num_inference_steps=50, seed=42,
+    references=[
+        {"type": "video", "video": ref_video},
+        {"type": "audio", "audio": ref_audio, "sample_rate": sample_rate},
+    ],
 )
 write_video_audio(
     video=video, audio=audio,
-    output_path="fl2va.mp4", fps=24, audio_sample_rate=32000,
+    output_path="tav2va.mp4", fps=24, audio_sample_rate=32000,
 )
 ```
 
-### Ref2VA — 参考图像/视频/音频 → 视频+音频
+</details>
+
+[Comfy-Org/MiniMax-H3](https://www.modelscope.cn/models/Comfy-Org/MiniMax-H3) 提供了 MiniMax-H3 的 pruned 变体，它把 DiT 的时间步嵌入 MLP 替换为一张查找表，使 `adaln_proj.linear` 的输入维度从 2688 降到 8，模型参数量降至约 20B。我们对其同样提供了 NF4 量化权重，单个 DiT 文件约 9.8 GB。
 
-支持四种参考类型，可在一个列表里组合（`video` 无声，带声视频用 `video_audio`）：
+用法与上文完全一致，只需把 DiT 的 `origin_file_pattern` 换成 pruned 版本，文本编码器与两个 VAE 仍然复用同一套权重。
+
+FL2VA:
+
+<details>
+
+<summary>展开代码</summary>
 
 ```python
-{"type": "image",       "image": PIL.Image}
-{"type": "video",       "video": list[PIL.Image]}                            # 无声
-{"type": "audio",       "audio": Tensor[C, L], "sample_rate": int}
-{"type": "video_audio", "video": list[PIL.Image], "audio": Tensor[C, L], "sample_rate": int}
+import torch
+from diffsynth.pipelines.minimax_h3_audio_video import MiniMaxH3Pipeline, ModelConfig
+from diffsynth.utils.data.audio_video import write_video_audio
+from modelscope import dataset_snapshot_download
+from PIL import Image
+
+vram_config = {
+    "offload_dtype": "disk",
+    "offload_device": "disk",
+    "onload_dtype": torch.bfloat16,
+    "onload_device": "cpu",
+    "preparing_dtype": torch.bfloat16,
+    "preparing_device": "cuda",
+    "computation_dtype": torch.bfloat16,
+    "computation_device": "cuda",
+}
+pipe = MiniMaxH3Pipeline.from_pretrained(
+    torch_dtype=torch.bfloat16,
+    device="cuda",
+    model_configs=[
+        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="minimax-h3-fl2va-pruned-nf4.safetensors", **vram_config),
+        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="minimax-h3-text-encoder-nf4.safetensors", **vram_config),
+        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="video_vae_nf4.safetensors", **vram_config),
+        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="audio_vae_nf4.safetensors", **vram_config),
+    ],
+    processor_config=ModelConfig(model_id="MiniMax/MiniMax-H3", origin_file_pattern="FL2VA/processor/"),
+    vram_limit=torch.cuda.mem_get_info("cuda")[1] / (1024 ** 3) - 4,
+)
+
+prompt = "A girl is very happy, she is speaking in english: “I enjoy working with Diffsynth-Studio, it's a perfect framework.”"
+video, audio = pipe(
+    prompt=prompt,
+    height=480, width=832, num_frames=124, num_inference_steps=50, seed=0,
+)
+write_video_audio(
+    video=video, audio=audio,
+    output_path="t2va.mp4", fps=24, audio_sample_rate=32000,
+)
 ```
 
+</details>
+
+Ref2VA:
+
+<details>
+
+<summary>展开代码</summary>
+
 ```python
 import torch
 from PIL import Image
@@ -156,7 +260,7 @@ pipe = MiniMaxH3Pipeline.from_pretrained(
     torch_dtype=torch.bfloat16,
     device="cuda",
     model_configs=[
-        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="minimax-h3-ref2va-nf4.safetensors", **vram_config),
+        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="minimax-h3-ref2va-pruned-nf4.safetensors", **vram_config),
         ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="minimax-h3-text-encoder-nf4.safetensors", **vram_config),
         ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="video_vae_nf4.safetensors", **vram_config),
         ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="audio_vae_nf4.safetensors", **vram_config),
@@ -197,20 +301,137 @@ write_video_audio(
 )
 ```
 
-## 常用参数
+</details>
+
+### 极端硬件优化
 
-- `height` / `width`：分辨率，如 `480x832`（横）或 `832x480`（竖）。
-- `num_frames`：帧数，需满足 `num_frames % 17 == 5`（如 124）。
-- `num_inference_steps`：去噪步数，示例用 50。
-- `keyframes` / `keyframe_indices`：FL2VA 首尾帧控制（`[0, -1]` 表示首帧和尾帧）。
-- `references`：Ref2VA 参考列表，元素为 `{"type": "image|video|audio|video_audio", ...}`。
-- 输出：`write_video_audio(video, audio, output_path, fps=24, audio_sample_rate=32000)`。
+如果你的计算设备性能极为有限，我们支持开启硬盘到显存的直连，在这样的配置下，模型中的张量将会按照计算顺序逐个从硬盘加载到显存，只需 8G 内存即可运行：
 
-## 参考示例脚本
+```diff
+vram_config = {
++   "offload_dtype": "disk",
++   "offload_device": "disk",
++   "onload_dtype": "disk",
++   "onload_device": "disk",
++   "preparing_dtype": "disk",
++   "preparing_device": "disk",
++   "computation_dtype": torch.bfloat16,
++   "computation_device": "cuda",
+}
+pipe = MiniMaxH3Pipeline.from_pretrained(
+    torch_dtype=torch.bfloat16,
+    device="cuda",
+    model_configs=...,
+    processor_config=...,
++   vram_limit=0,
+)
+```
+
+如果你希望在 Mac M 系列芯片上运行模型推理，我们也是支持的，尽管这并不推荐：
+
+```diff
+vram_config = {
++   "offload_dtype": "disk",
++   "offload_device": "disk",
++   "onload_dtype": "disk",
++   "onload_device": "disk",
++   "preparing_dtype": "disk",
++   "preparing_device": "disk",
++   "computation_dtype": torch.bfloat16,
++   "computation_device": "mps",
+}
+pipe = MiniMaxH3Pipeline.from_pretrained(
+    torch_dtype=torch.bfloat16,
++   device="mps",
+    model_configs=...,
+    processor_config=...,
++   vram_limit=0,
+)
+```
+
+## 训练代码
+
+本量化模型支持 LoRA 训练，请按照以下步骤启动训练程序。
+
+下载样例数据集：
+
+```shell
+modelscope download --dataset DiffSynth-Studio/diffsynth_example_dataset --include "minimax_h3/MiniMax-H3-FL2VA/*" --local_dir ./data/diffsynth_example_dataset
+```
+
+适合数据中心 GPU（例如 Nvidia H20）的训练配置：运行以下脚本，启动 LoRA 训练程序，需要 48G 显存。
+
+```shell
+accelerate launch examples/minimax_h3/model_training/train.py \
+  --dataset_base_path data/diffsynth_example_dataset/minimax_h3/MiniMax-H3-FL2VA \
+  --dataset_metadata_path data/diffsynth_example_dataset/minimax_h3/MiniMax-H3-FL2VA/metadata.csv \
+  --data_file_keys "video,input_audio" \
+  --extra_inputs "input_audio" \
+  --height 480 \
+  --width 832 \
+  --num_frames 124 \
+  --dataset_repeat 100 \
+  --model_id_with_origin_paths "DiffSynth-Studio/MiniMax-H3-NF4:minimax-h3-text-encoder-nf4.safetensors,DiffSynth-Studio/MiniMax-H3-NF4:minimax-h3-fl2va-nf4.safetensors,DiffSynth-Studio/MiniMax-H3-NF4:video_vae_nf4.safetensors,DiffSynth-Studio/MiniMax-H3-NF4:audio_vae_nf4.safetensors" \
+  --learning_rate 1e-4 \
+  --num_epochs 5 \
+  --remove_prefix_in_ckpt "pipe.dit." \
+  --output_path "./models/train/MiniMax-H3-T2VA-nf4" \
+  --lora_base_model "dit" \
+  --lora_target_modules "qkv_proj,out_proj" \
+  --lora_rank 32 \
+  --use_gradient_checkpointing \
+  --find_unused_parameters
+```
+
+适合消费级 GPU（例如 Nvidia RTX 4090）的训练配置：运行以下脚本，启动两阶段拆分训练与 gradient checkpointing offload，需要 24G 显存。
+
+```shell
+accelerate launch examples/minimax_h3/model_training/train.py \
+  --dataset_base_path data/diffsynth_example_dataset/minimax_h3/MiniMax-H3-FL2VA \
+  --dataset_metadata_path data/diffsynth_example_dataset/minimax_h3/MiniMax-H3-FL2VA/metadata.csv \
+  --data_file_keys "video,input_audio" \
+  --extra_inputs "input_audio" \
+  --height 480 \
+  --width 832 \
+  --num_frames 124 \
+  --dataset_repeat 1 \
+  --model_id_with_origin_paths "DiffSynth-Studio/MiniMax-H3-NF4:minimax-h3-text-encoder-nf4.safetensors,DiffSynth-Studio/MiniMax-H3-NF4:video_vae_nf4.safetensors,DiffSynth-Studio/MiniMax-H3-NF4:audio_vae_nf4.safetensors" \
+  --learning_rate 1e-4 \
+  --num_epochs 1 \
+  --remove_prefix_in_ckpt "pipe.dit." \
+  --output_path "./models/train/MiniMax-H3-T2VA-nf4-split-cache" \
+  --lora_base_model "dit" \
+  --lora_target_modules "qkv_proj,out_proj" \
+  --lora_rank 32 \
+  --use_gradient_checkpointing \
+  --use_gradient_checkpointing_offload \
+  --task "sft:data_process"
+
+accelerate launch examples/minimax_h3/model_training/train.py \
+  --dataset_base_path "./models/train/MiniMax-H3-T2VA-nf4-split-cache" \
+  --data_file_keys "video,input_audio" \
+  --extra_inputs "input_audio" \
+  --height 480 \
+  --width 832 \
+  --num_frames 124 \
+  --dataset_repeat 100 \
+  --model_id_with_origin_paths "DiffSynth-Studio/MiniMax-H3-NF4:minimax-h3-fl2va-nf4.safetensors" \
+  --learning_rate 1e-4 \
+  --num_epochs 5 \
+  --remove_prefix_in_ckpt "pipe.dit." \
+  --output_path "./models/train/MiniMax-H3-T2VA-nf4" \
+  --lora_base_model "dit" \
+  --lora_target_modules "qkv_proj,out_proj" \
+  --lora_rank 32 \
+  --use_gradient_checkpointing \
+  --use_gradient_checkpointing_offload \
+  --find_unused_parameters \
+  --task "sft:train"
+```
 
-仓库内完整可运行脚本（本 README 的示例即取自这些脚本）：
+## 参考资料
 
-- `examples/minimax_h3/model_inference_low_vram/MiniMax-H3-NF4-FL2VA.py`
-- `examples/minimax_h3/model_inference_low_vram/MiniMax-H3-NF4-Ref2VA.py`
-- `examples/minimax_h3/model_inference/MiniMax-H3-NF4-FL2VA.py`（CPU offload）
-- `examples/minimax_h3/model_inference/MiniMax-H3-NF4-Ref2VA.py`（CPU offload）
+* DiffSynth-Studio 文档：[Minimax-H3](https://diffsynth-studio-doc.readthedocs.io/en/latest/Model_Details/MiniMax-H3.html)
+* DiffSynth-Studio 文档：[显存管理](https://diffsynth-studio-doc.readthedocs.io/en/latest/Pipeline_Usage/VRAM_management.html)
+* DiffSynth-Studio 文档：[两阶段拆分训练](https://diffsynth-studio-doc.readthedocs.io/en/latest/Training/Split_Training.html)
+* DiffSynth-Studio 文档：[低显存训练](https://diffsynth-studio-doc.readthedocs.io/en/latest/Pipeline_Usage/Model_Training.html#low-vram-training)

minimax-h3-fl2va-pruned-nf4.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7c7b5080f6ee907a85cb44caa233ce121e8fa84de9dbac1e24d964c2232baea4
+size 10483868433

minimax-h3-ref2va-pruned-nf4.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7f32e780571a10e770676fa2c07ecc7fd0e22b35458243e456a80882c5693018
+size 10483868417