How to train Qwen Image on 4090, getting non stop OOM no matter what I set up the config to.

[E10H1M]

I’m unable to train Qwen-Image on a 24GB GPU — it always runs into CUDA OOM.

Steps I already tried:

• export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
• --enable_fp8_training
• --use_gradient_checkpointing
• --gradient_accumulation_steps 2
• --max_pixels 262144

Even with these, the model immediately runs out of memory on 24GB card

[Zeepat]

Did you manage to find a solution? I am running into the same issue even when I train on the lowest settings possible on a double 48gb vram gpu.

[E10H1M]

@Zeepat nope but I found this other repo:

https://github.com/FlyMyAI/flymyai-lora-trainer

which seems to work but is a RAM hog.
though, with dual 48GB vram, it might not be as demanding on you.
i say you check it out, i'm digging into it...
two heads is always better than one.

...least until someone from modelscope circles back.
otherwise, we'll have reverse engineered the training scripts by then 🥱😉

[Zeepat]

If you're looking to train a lora I recommend the OSTRIS ai-toolkit UI, which supports qwen-image and qwen-image-edit lora training. But full fine-tune or ControlNet training I can't find any alternatives to.

[E10H1M]

@Zeepat I want to actually understand ML and Model Architecture. He's just wrapping:
https://github.com/kohya-ss/musubi-tuner

I'd rather use Musubi tuner at that point but I didn't cause again: I want to actually understand model architecture and Musubi tuner uses Comfy weights, not the original weights. That's why I'm waiting for someone from Modelscope to reply.

Intellectual Independence is more important than convenience.
Especially in an extremely competitive field that demands extremely low level understanding to be taken even the least bit serious.

https://github.com/FlyMyAI/flymyai-lora-trainer is relevant because it actually trains on the original Qwen weights. The dev did the work to open source empowering information as he built scripts specifically for training on 24 GB of VRAM.

Soon I will be able to get back in the cloud, God willing.

[hwade]

Most of model trainning process has 2 key factors to affect the vram usage:

• dtype of loading params: quantize to lower accurate dtype (e.g. fp16 -> fp8)
• device to store params: store params in memory (offload to cpu), onload to gpu when inference

If you dive into the QwenImage training pipeline in Diffsynth studio framework , we can find the setting of model params are hard coded in the pipeline.

1. "module_map" stores the mapping of the patch of fp8 opperation implementation (e.g. nn.Linear patch to AutoWrappedLinear to implement fp8 linear mapping). The patched opers are implemented in some specific layers (e.g. linear, conv, emb), rather than all nn.Modules (e.g. QwenImageResidualBlock、Sequential). In other words, we can decrease the vram by adding more fp8 operation map of the net layer.

2. "model_config" contains the specific type and loading place of params. Try to offload the params from 'cuda' to 'cpu' and transfer loading type to fp8, which can save lots of vram usage (do not set computation device to 'cpu', or it turns to inference by cpu device)

[Deng-Xian-Sheng]

#1101

Supports int8-quanto quantization fine-tuning. Qwen-Image-Edit-2509, only 48GB of VRAM is needed for fine-tuning 🎉