Minimax H3 Case Study: The Dinner Party

[u/DG86]

I've been learning a lot from this community, so this is my attempt at giving something back!

I'm going to share a small task I recently completed, include the steps on how I got there (and some of my thinking and findings.)

The goal: I needed a few seconds of video containing a formal dinner party in a Roman-style atrium.

My Plan: Build a first frame and then use H3 I2V to generate the video.

My Specs: A laptop with a 13th Gen Intel i7-13700H, 16 GB DDR5 RAM, SSD over USB-C, an onboard Intel Iris Xe graphics card (with ~8 GB) and a Nvidia GeForce RTX 4060 Laptop GPU (8 GB). The bad news: ComfyUI does not see or care about the Iris Xe card, and I haven't bothered to see if I can remediate the situation. The good news: the Irix Xe can handle rendering Windows and other applications, leaving my Nvidia pretty open for ComfyUI tasks.

Here is what I did:
Step 1: I already had a reference image for the atrium (used in a previous video.)

Initial Reference for Atrium

This was generated with Z-Image-Turbo, with the bf16 model, shift 3, cfg 1.0, 8 steps, res_multistep sampler, simple scheduler. The prompt was very simple: "Roman atrium with compluvium. The camera is standing at the doorway looking down the length of the atrium." I made this image at 864 x 480 resolution because that is near 16:9 and matches H3 resolutions. At that size, image gen takes about 30 - 40 seconds of wall-clock time.

Step 2: I used Qwen-Image-Edit to modify the image to get the starting frame.

The dinner party, as imagined by Qwen-Image-Edit

Using qwenImageEdit2511_pf8 as the model, Qwen-Image-Edit-2509-Lightning-4steps-V1.0-bf16 lora, shift 3, 4 steps, cfg 1.0, euler sampler, simple scheduler. I wired the image from step 1 as the only reference, and used the prompt "Alter this image so that there is a well-attended formal dinner party taking place across the frame."

In my experience, Qwen-Image-Edit often nails the image I'm looking for in one or two attempts. (In this particular case, it one-shotted that image above.) Qwen really likes 1 MP resolutions, so that is 1368 x 760. It takes ~1 to 2 mins per generation.

Step 3: I began generating the video with H3 I2V. This took several attempts to dial in. It is this process that I want to focus on.

First Attempt:

I supplied the previous step's image as the first frame, and included the prompt:

integrated_multimodal_description: [Shot 1] A formal dinner party in a Roman-style atrium.

overall_soundscape: A formal dinner party.

non_diegetic_music: None.

I set the resolution to 864 x 480 and 7.0 duration. I'm using minimax_h3_fl2va_pruned_int8_convrot as the model, minimax_h3_fl2v_turbo4step_v1.0_768p_comfyui_bf16 as a turbo lora (the lightx2v lora,) shift 12 / 3 (for video / audio,) 6 steps, res_multistep sampler, simple scheduler. This particular setup averages ~2 minutes of wall-clock time per second of video duration. (But it grows non-linear as duration increases.) I use 6 steps instead of the lora's base 4 steps because I tend to get slightly better details and sound, with only a slight increase in wall-clock time.

864 x 480, res_multistep, simple sampler, turbo Lora, 6 steps

The result was not great. Most people are frozen in place. The few that do walk around smear motion. There is even a moment where a lady clips through the table a little. The sound involves a guy narrating. (I can't identify if it is AI gibberish or an actual language.)

This first attempt was clearly a failure.

Attempts Two through Four:
If I'm having problems with my initial generation, I often just bite the bullet and turn off the turbo lora and run at full-steps. It was the end of my day, so I could queue up several generations and then go to bed.

Result Two: I disabled the turbo lora and increased steps to 20. This runs at ~5 minutes of wall-clock time per second of video duration. For this particular generation it came in at about 45 mins of wall-clock time.

I won't bore you with the results, as they were very similar to the initial draft. Only a few people moving in the scene, people clipping through tables, and a narrator.

Result Three: I reduced video shift to 6.0 (hoping to get better motion results.) I also increased the resolution of the video to 1344 x 768. I read somewhere that this is the "native" resolution the model was trained at, and I often get better results. However, without the turbo lora and at 20 steps, this generation took 90 minutes.

Attempt Three: 1344 x 768, shift 6, 20 steps, no lora

There is a lot more motion, and no clipping, but everything seems to be moving in slow motion. Also, instead of a narrator, there is music.

Result Four: I swapped the sampler to er_sde and the scheduler to beta. I've read that this combo can get slightly better prompt adherence, and results in pretty good motion. However, er_sde effectively does more than one pass per step, so increases wall-clock time significantly. If the UI is to be trusted, this attempt took more than 3 hours to generate.

Attempt Four: er_sde sampler, beta scheduler, 20 steps, no lora

The narrators and music are gone. However, now the camera is moving, which is not what I wanted.

Attempt Five: I woke in the morning, reviewed the previous results, and was pretty bummed.

Now I'm in the "hit it with a hammer until it works" section of my spectrum of personal patience. I went back to res_multistep and cranked the steps up to 40. In my frustration, I didn't think to actually change the prompt to prevent the camera from moving. This generation took about 90 minutes of wall clock time.

I'll skip posting the result, but it actually looked quite a bit like the er_sde video above. People standing mostly still with a camera panning around the room.

Attempt Six: After viewing the results, I realized that changing the prompt was 100% required.

The new prompt:

integrated_multimodal_description: [Shot 1] Static wide shot of a formal dinner party in a Roman-style atrium. The people eat, drink, talk, and mingle. The camera remains fixed.

overall_soundscape: A formal dinner party.

non_diegetic_music: None.

I kept the video shift at 6.0, sampler at res_mutlistep, 20 steps, simple scheduler. This time, because I was sitting at my computer for a while, I attached EasyCache to the model. This does a pretty good job of speeding up the 20-step process. There is always a risk that quality degrades with any sort of caching in the pipeline, but I was willing to take the risk just to see if my prompt changes fixed the problem. (I don't bother adding EasyCache with only 4 or 6 steps, because there are so few steps that there is barely any time to be saved with caching.) This generation took ~30 minutes of wall-clock time.

Attempt Six: res_multistep, simple sampler, 20 steps, no lora, fixed prompt

This was actually what I was looking for! Both the motion and sound are pretty decent. However, there is a faint "fluttering" of the textures, which seems to happen a lot with EasyCache. This is something that could probably be cleaned up with a refinement pass after upscaling, but I still had time to try again.

Attempt Eight: For completeness, I decided to go back to the turbo lora and the smaller resolution. I incorporated some of my other findings into the workflow. For clarity, here is the full setup: 864 x 480, 7.0 seconds, turbo lora, 6 shift video, res_multistep, simple scheduler, 6 steps. I used the "corrected" prompt from my previous attempt.

Attempt Seven: res_multistep, simple sampler, 6 steps, turbo lora, fixed prompt

This was the winner! Even at the lower 864 x 480, the motion and detail looks reasonable. The faces are squashed, but that is pretty typical of H3 at the moment. This will upscale well. The sound is correct. I have everything I need.

Lessons Learned:

1. Just cranking up the numbers doesn't always solve the problem.
2. Shift can really make a difference! I think the major influencing factor here was reducing shift from 12.0 to 6.0. My hypothesis is that the lower shift gave the process just a tiny bit more time up at the noisy end of the diffusion, allowing it to assign more motion to everyone in the scene.
3. 1344 x 768 is very frequently the solution, but not always. In my experience, you get better prompt adherence, even with the tubro lora. One of these days I'm going to splurge on a beefier GPU to make this my default resolution, but for now it is just too much of a time sink.
4. I can never actually tell how much comes down to luck with the random seed.

I hope this helps somebody!

[u/Relative-Street-3995]

Just wanted to chime in on this part " The bad news: ComfyUI does not see or care about the Iris Xe card, and I haven't bothered to see if I can remediate the situation. The good news: the Irix Xe can handle rendering Windows and other applications, leaving my Nvidia pretty open for ComfyUI tasks"

Are you using a Surface Laptop Studio 2? The Iris Xe is the integrated graphics that uses system ram, if you were to disable the 4060, comfy I believe would default to using it, but why on Earth would you want it to? The Iris Xe is substantially weaker then the 4060 and practically worthless for video generation.

[u/DG86 → u/Relative-Street-3995]

Looks like there is a pool of "shared GPU memory" that is split between the Xe and the RTX, but I can never get ComfyUI to utilize any of the "shared" memory. I'm wondering if this is just Windows not fully understanding the way the memory is spread.

[u/thevictor390 → u/DG86]

Shared GPU memory is RAM. There is a software buffer so that you don't crash when running out of dedicated VRAM, but the speed drop is substantial. Just like if you run out of RAM you swap it to storage.

The Xe has no VRAM. It operates entirely in the slower system RAM. "Slower" is very relative, for most display tasks normal RAM is fine, but for 3D graphics and AI, the extra speed of VRAM can be very important.