Added Video Generation support for generating images #974

ParamThakkar123 · 2025-11-29T05:33:32Z

No description provided.

…ab-app into add/video-gen

ParamThakkar123 · 2025-11-29T05:36:23Z

I added support for video generation models starting with Wan2.1 models will later extend to more models if this works, currently I added support for WanPipeline in supported model architectures in Image_diffusion plugin but it gives an error where its trying to find /workspace/jobs/None which doesn't exist

codecov-commenter · 2025-11-29T05:37:36Z

Codecov Report

❌ Patch coverage is 2.15054% with 182 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
api/transformerlab/plugins/image_diffusion/main.py	0.00%	153 Missing ⚠️
api/transformerlab/routers/experiment/diffusion.py	12.12%	27 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!

deep1401 · 2025-12-02T21:10:14Z

I added support for video generation models starting with Wan2.1 models will later extend to more models if this works, currently I added support for WanPipeline in supported model architectures in Image_diffusion plugin but it gives an error where its trying to find /workspace/jobs/None which doesn't exist

There was an issue with running diffusion models, could you update and check if that issue is resolved now? The issue was caused by our new system with the workspace under orgs

…ab-app into add/video-gen

…nsformerlab-app into add/video-gen

deep1401 · 2025-12-03T15:14:54Z

I added support for video generation models starting with Wan2.1 models will later extend to more models if this works, currently I added support for WanPipeline in supported model architectures in Image_diffusion plugin but it gives an error where its trying to find /workspace/jobs/None which doesn't exist

There was an issue with running diffusion models, could you update and check if that issue is resolved now? The issue was caused by our new system with the workspace under orgs

I figure this is still a WIP but I would also suggest not adding things to the existing image_diffusion plugin and creating a new video diffusion plugin.

…ab-app into add/video-gen

deep1401

Added some comments, we might have to restructure the PR and break it down into 2 and test these

deep1401 · 2025-12-05T20:05:35Z

api/transformerlab/plugins/video_diffusion_plugin/index.json

+  "url": "",
+  "files": ["main.py", "video_diffusion_worker.py", "setup.sh"],
+  "model_architectures": [
+    "DiffusionPipeline",


There is no architecture called DiffusionPipeline, this should not be here

Also our image_diffusion plugin has no architectures in there so lets also not put any in here.

deep1401 · 2025-12-05T20:07:17Z

api/transformerlab/plugins/image_diffusion/setup.sh

@@ -1,0 +1,3 @@
+#!/bin/bash
+uv pip install git+https://github.com/huggingface/diffusers transformers accelerate safetensors


What version of diffusers is required? The latest release on pypi is 0.35.2 and I assumed Wan2.1 would work with that but maybe you'd know better

deep1401 · 2025-12-05T20:08:06Z

api/transformerlab/plugins/image_diffusion/index.json

-    "amd"
-  ],
+  "files": ["main.py", "diffusion_worker.py", "setup.sh"],
+  "model_architectures": ["DiffusionPipeline", "WanPipeline"],


I dont think adding architectures here works. Did you try running the model?
You'd need to add architecture in the massive list in the diffusion router

deep1401 · 2025-12-05T20:09:52Z

api/transformerlab/plugins/video_diffusion_plugin/main.py

This wouldnt work because we've not made any frontend changes right? Maybe lets split up PRs and you have a separate WIP for video diffusion. You could implement API and the plugin and add a video for a working model in that PR.

Lets not add anything without testing since diffusion has a lot of things which might break by minor changes here or there

Also lets make this PR image_diffusion changes only and also add the ZTurbo model additions in here

deep1401 · 2025-12-05T20:11:26Z

api/transformerlab/plugins/video_diffusion_plugin/video_diffusion_worker.py

I dont think this works since the structure of the wan model is different. Reference: https://huggingface.co/Wan-AI/Wan2.1-T2V-14B/tree/main

This file in image diffusion was added specifically for FLUX as that required sharding. Lets try to test everything we add :)

Yeah. Currently testing all the things I added. The models are quite huge :)

okay no worries I converted the PR to draft for now and we can split it up and test things.
Let me know if you need my help testing anything or if you dont get something on diffusion about how it works, I can help you figure out!

I mentioned a text to video reference model below but you can look at WanAI's collections and maybe select smaller models for testing. They usually put the small ones in their diffusers collections

Sure! Thanks deep. I will try this out and get it done earliest 🫡

deep1401 · 2025-12-05T20:11:47Z

api/transformerlab/plugins/video_diffusion_plugin/setup.sh

@@ -0,0 +1,3 @@
+#!/bin/bash
+uv pip install git+https://github.com/huggingface/diffusers transformers accelerate safetensors


Same comment as above as to why we need dev version and for what model

deep1401

Sorry I forgot to add this earlier but lets use this as a reference model for Wan support. This feels small enough to run on our hardware and diffusers required version is 0.33.0.dev so just updating to current diffusers prod version should work
https://huggingface.co/Wan-AI/Wan2.1-T2V-1.3B-Diffusers

…nsformerlab-app into add/video-gen

…ab-app into add/video-gen

ParamThakkar123 added 2 commits November 29, 2025 11:01

Added Video Generation support

6bd8f24

Merge branch 'main' of https://github.com/transformerlab/transformerl…

f33c5a6

…ab-app into add/video-gen

ParamThakkar123 marked this pull request as draft November 29, 2025 05:34

Merge branch 'main' into add/video-gen

9020496

ParamThakkar123 added 2 commits December 3, 2025 11:20

Merge branch 'main' of https://github.com/transformerlab/transformerl…

f748884

…ab-app into add/video-gen

Merge branch 'add/video-gen' of https://github.com/transformerlab/tra…

90f77f0

…nsformerlab-app into add/video-gen

ParamThakkar123 added 4 commits December 5, 2025 19:31

Merge branch 'main' of https://github.com/transformerlab/transformerl…

c85ed54

…ab-app into add/video-gen

Separate plugin for video diffusion

9dfc70a

Merge branch 'main' of https://github.com/transformerlab/transformerl…

9b73396

…ab-app into add/video-gen

Separate plugin for video diffusion

1f444e8

ParamThakkar123 marked this pull request as ready for review December 5, 2025 18:19

ParamThakkar123 and others added 2 commits December 5, 2025 23:51

Separate plugin for video diffusion

2a15194

Merge branch 'main' into add/video-gen

d45b7b3

deep1401 requested changes Dec 5, 2025

View reviewed changes

deep1401 marked this pull request as draft December 5, 2025 20:17

ParamThakkar123 added 4 commits December 13, 2025 19:19

Rebase

5988526

Merge branch 'add/video-gen' of https://github.com/transformerlab/tra…

2859589

…nsformerlab-app into add/video-gen

Added Support for generating images using Wan

353878d

Updates

fbba69c

ParamThakkar123 changed the title ~~Added Video Generation support~~ Added Video Generation support for generating images Dec 17, 2025

ParamThakkar123 marked this pull request as ready for review December 17, 2025 10:15

ParamThakkar123 added 4 commits December 17, 2025 15:45

Merge branch 'main' of https://github.com/transformerlab/transformerl…

97e43db

…ab-app into add/video-gen

Updates

e5a8922

Rebase

7490496

Rebase

eea03bd

ParamThakkar123 and others added 3 commits December 18, 2025 19:04

Rebase

18e8d54

Rebase

863987b

Merge branch 'main' into add/video-gen

09a9a83

		@@ -1,0 +1,3 @@
		#!/bin/bash
		uv pip install git+https://github.com/huggingface/diffusers transformers accelerate safetensors

		@@ -0,0 +1,3 @@
		#!/bin/bash
		uv pip install git+https://github.com/huggingface/diffusers transformers accelerate safetensors

Uh oh!

Added Video Generation support for generating images #974

Are you sure you want to change the base?

Added Video Generation support for generating images #974

Conversation

ParamThakkar123 commented Nov 29, 2025

Uh oh!

ParamThakkar123 commented Nov 29, 2025

Uh oh!

codecov-commenter commented Nov 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

deep1401 commented Dec 2, 2025

Uh oh!

deep1401 commented Dec 3, 2025

Uh oh!

deep1401 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deep1401 Dec 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deep1401 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov-commenter commented Nov 29, 2025 •

edited

Loading

deep1401 Dec 5, 2025 •

edited

Loading