AI Video Generation - General Discussion, Tips, Tricks, Frustrations and Showcases

If the LoRA you're after is of a specific looking (or just specific) person, just my opinion, but I would opt for working on the LoRA with an image model. That way I could use it to place that person in any environment I wanted via prompts, and then send the images through a video model like Wan or LTX. I think it is harder and more hardware intense to try to train a video LoRA for a specific person.

Am I understanding that to be your goal? Train a video LoRA to produce a specific subject?
 
Yes, you're correct, my goal is to train a character LoRA to use with, say, wan 2.2 14B. I'm not familiar with any image models, but I'll look into that. I didn't even think about using an image gen AI, then using the resulting pictures with a video gen AI, but yeah, you're definitely right that training a LoRA for an image model should be less resource intensive. I'll report back once I have something to show!

(Edit: I completely misunderstood your previous post. I thought that you meant that training a LoRA on videos was not possible with my rig, but training one on images was, but now I get it. You didn't mean training a LoRA on videos or images for a video gen AI, but training one on images for an image gen AI.)
 
Last edited:
  • Like
Reactions: Casshern2
I think starting with an image LoRA is definitely a good way to get familiar with the whole process before jumping into video LoRA training. Once you have a character LoRA working well, you can use it to generate a consistent set of images and then experiment with feeding those into a video model like Wan 2.2.
 
  • Like
Reactions: Casshern2
I finally found an upscale workflow that didn't have OOM crashes. And it was a very unassuming video from an equally unassuming YT channel. Although, not sure this qualifies as a true upscale because the node that does the size increase is a simple resize node. But it works.

This was also a good test of a Flux 2 Klein workflow to place different subjects in the same space, so all these lovely ladies were creates separately then placed on this street.

Tokyo Street Walking 1
 
  • Like
Reactions: yuumi_fan
I think I'm done. I have made two LoRAs, neither worked, then I tweaked it, and now it would take forever to train one, so I put that aside for now to get wan 2.2 14B running. I eventually got it to run, chose one of the screenshots that I wanted to train a LoRA with, but the videos were not at all what I prompted. The first one was abysmal; I tweaked the workflow and the second one was better, but still not what I asked for. I linked the AI-upscaled photo and the two videos, you can view/play them in your browser. The prompt was "A young woman with blonde hair looks to her side and notices a penis." For the second one I changed that to "an erect penis", but that didn't help. This will be up for 3 days:


Also, while the videos were generating I caught a whiff of a burnt smell. The fans were not working at max speed, the GPU temperature was around 65 Celsius, so I'm not sure what it was (it could have been simply dust according to ChatGPT), but I'm not sure if I want to risk it being something serious. Unless this is normal?

So yeah, I think I'll just wait until they make a retard-proof software for people like me unless somebody here has advice on how to do better and whether or not I should be concerned about my hardware.

Edit: tried again with a different photo, this time only wanted nudity. The prompt was: "A young woman with blonde hair reveals her boobs." But man... this is nightmare fuel. I'm guessing it didn't look at the entire photo, just the preview that you can see on the site as well before clicking on the photo, but I don't know. I don't know how to deal with this. If anybody's got any advice, let me know, because I'm at the end of my rope here.

 
Last edited:
Hey, pal! I was out of town until today and just missed the 3 day mark on your first ones. I was able to grab your second one. The first thing is your image is a non-standard vertical size, not even true 16:9 (or here not even 9:16 since it is vertical) and your video size is 640x480 which is not 16:9 or vertical either. Your image should always track with your video in terms of aspect ratio. This is why it centered (the default for the node) and crops off the entire top and bottom, so it had no idea what her face looked liked, just that she was blonde which is also primarily the only reason the video pans up, because the model was trying to show the blonde hair you prompted.

I've learned that Wan likes to have descriptive prompts. Something too small like "A young woman with blonde hair reveals her boobs" isn't enough to get any meaningful results you had in mind, and it forces the model to just make something up. Your video did what you asked, to the model, she is revealing as in "showing" or "displaying" her boobs, but because you didn't mention "bare boobs" or "naked boobs" or even just mention her nipples, she wasn't generated to actually show them. Now, this stuff being what it is and how it works, it's possible different passes with different seeds would eventually show them.

Here I've taken your image, cropped off the bottom to create a true-ish 9:16 image to feed Wan 2.2. One trick I've used and used here is to ask Google to create a prompt for Wan 2.2 with replacement language. I asked it:

Take this simple prompt of a woman trying to bring emphasis to her necklace by touching it and holding it for the viewer. This is for a Wan 2.2 I2V prompt so it has to be descriptive: A young woman with blonde hair reveals her necklace

Whatever it spits out I replaced necklace and any other words that need to change to mean her boobs. Here was the original result:

A close-up shot of a young woman with blonde hair gracefully raising her hand to her chest. Her fingers gently pinch and lift a delicate silver necklace away from her collarbone, holding it up toward the camera to present it to the viewer. The camera holds focus tightly on the shimmering pendant of the necklace. Her expression is warm and intentional, with a soft smile, ensuring the viewer's attention is entirely drawn to the jewelry. The motion is smooth, deliberate, and elegant, with subtle, natural micro-movements in her hair and shoulders.

I changed it to:

A close-up shot of a young woman with blonde hair lifts her blouse above her chest exposing her bare breasts and nipples. Her fingers gently touch her breasts, holding them up toward the camera to present them to the viewer. The camera holds focus tightly on her breasts. Her expression is warm and intentional, with a soft smile, ensuring the viewer's attention is entirely drawn to the jewelry. The motion is smooth, deliberate, and elegant, with subtle, natural micro-movements in her hair and shoulders.

Here is the image and the video that came it produced on the first try, which may have been luck, BUT it certainly didn't have her lift her shirt, she destroyed it LOL. But, you get the idea.

01.jpg

 
Hey, pal! I was out of town until today and just missed the 3 day mark on your first ones. I was able to grab your second one. The first thing is your image is a non-standard vertical size, not even true 16:9 (or here not even 9:16 since it is vertical) and your video size is 640x480 which is not 16:9 or vertical either. Your image should always track with your video in terms of aspect ratio. This is why it centered (the default for the node) and crops off the entire top and bottom, so it had no idea what her face looked liked, just that she was blonde which is also primarily the only reason the video pans up, because the model was trying to show the blonde hair you prompted.

I've learned that Wan likes to have descriptive prompts. Something too small like "A young woman with blonde hair reveals her boobs" isn't enough to get any meaningful results you had in mind, and it forces the model to just make something up. Your video did what you asked, to the model, she is revealing as in "showing" or "displaying" her boobs, but because you didn't mention "bare boobs" or "naked boobs" or even just mention her nipples, she wasn't generated to actually show them. Now, this stuff being what it is and how it works, it's possible different passes with different seeds would eventually show them.

Here I've taken your image, cropped off the bottom to create a true-ish 9:16 image to feed Wan 2.2. One trick I've used and used here is to ask Google to create a prompt for Wan 2.2 with replacement language. I asked it:

Take this simple prompt of a woman trying to bring emphasis to her necklace by touching it and holding it for the viewer. This is for a Wan 2.2 I2V prompt so it has to be descriptive: A young woman with blonde hair reveals her necklace

Whatever it spits out I replaced necklace and any other words that need to change to mean her boobs. Here was the original result:

A close-up shot of a young woman with blonde hair gracefully raising her hand to her chest. Her fingers gently pinch and lift a delicate silver necklace away from her collarbone, holding it up toward the camera to present it to the viewer. The camera holds focus tightly on the shimmering pendant of the necklace. Her expression is warm and intentional, with a soft smile, ensuring the viewer's attention is entirely drawn to the jewelry. The motion is smooth, deliberate, and elegant, with subtle, natural micro-movements in her hair and shoulders.

I changed it to:

A close-up shot of a young woman with blonde hair lifts her blouse above her chest exposing her bare breasts and nipples. Her fingers gently touch her breasts, holding them up toward the camera to present them to the viewer. The camera holds focus tightly on her breasts. Her expression is warm and intentional, with a soft smile, ensuring the viewer's attention is entirely drawn to the jewelry. The motion is smooth, deliberate, and elegant, with subtle, natural micro-movements in her hair and shoulders.

Here is the image and the video that came it produced on the first try, which may have been luck, BUT it certainly didn't have her lift her shirt, she destroyed it LOL. But, you get the idea.

View attachment 3856451


Here are the two videos created based on a different image from my first try at this if you are curious:


I tried to recreate your success by copying your image and prompt, but now ComfyUI just gets stuck at the first KSampler. I've been trying to troubleshoot it for the past hour or so, but no success yet, so I think I'll just give this thing up. I know that the thread has "frustrations" in the title, and boy, could I write a short story about my trials and tribulations, but I was willing to power through all of that to have a chance at creating the content that I would like, But the nail in the coffin came with this post of yours: the video doesn't keep the resemblance of the girl on the screenshot. This is, of course, no surpsise, since it doesn't use a LoRA. Even if I get the image LoRA working, I can't count on wan to keep the video consistent with that character, since wan doesn't have that LoRA. And as I said earlier, the resemblance is really important for me. So yeah, I'll throw in the towel here.

In any case, thank you for your help, Casshern, and keep the cool stuff coming!
 
  • Sad
Reactions: Casshern2
I'm not familiar with that performer in your screenshot so I don't know the consistency of her look. Keep in mind if you throw a "bad" or fuzzy/blurry or grainy image at Wan as a starting point I'd imagine there will be some significant drift since it doesn't have details to keep things as clean as possible. But that might be another issue if you're trying to use existing footage screenshots or some from a streaming site (which usually don't have great quality videos). If that's all you'll have as starting sources you likely won't get acceptable results like you want. :(

I'm about to start looking into training a small/light LoRA (if that's even possible) for Wan 2.2 I2V. According to AI, it's possible. If I can successfully do so with my low VRAM, I'll share the workflow.
 
Yeah, the base video is 854 x 480 and low quality; AI told me to scale up the screenshots, so that's what I did. And you're right, it's far from ideal, but that's the best that I could do. I would have been curious to see how closely a LoRA for wan 2.2 would have resembled the girl on the screenshots, but like I said, that's definitely a no go with my setup, although with you saying that it might be possible, I guess it might be possible. Again, I'm a retard when it comes to tech. I also couldn't get the ComfyUI workflow working again, tried 640 x 480 again, then reverted the ComfyUI version to an earlier snapshot when it still worked, but nothing. So I'm pretty much done until I can get a better rig, which won't be anytime soon... But I'll keep an eye on this thread to see how AI is progressing!
 
Oh, I don't have it anymore, I have already uninstalled ComfyUI and got rid of everything, but it was just the basic wan 2.2 14B I2V workflow, nothing extra was added.
 
Wow you really did a purge off this stuff. :eek:

The default workflow uses the full 14B high and low models, I've been using the light GGUF UNET models, that cuts the generation time down a lot, especially when using the lightning loras. Slight quality loss, but not all that noticeable.
 
Yeah, unfortunately, I lost heart, but sometimes I feel tempted to give training a LoRA for the 5B model a try, AI says it might work. Just might...

And yes, I also used the GGUF versions (Q4_K_M for both high and low), but not the lightning LoRAs (though AI mentioned those as a possibility). But apart from that, it was just the basic workflow, no LoRAs at all. The workflow stopped working after I changed the resolution to fit 9:16, but ComfyUI had an update right before that. Like I said, I tried everything, I couldn't get it running again. Maybe if I reinstall it it will work, I don't know, but before that, I would just try to get a LoRA, because that is the main thing for me by far. I just need to somehow summon the will to put myself through all the tech stuff again, though this time around it would be easier. Until that, I'll keep an eye on this thread and your progress. Let me know if you succeed with your LoRA and of course, I'm always happy to check out some videos!

Edit: tried creating a LoRA with AI Ostris Toolkit for wan 2.2 5B, but the GPU can't take it. So yeah, this is definitely it for me.
 
Last edited:
I'm going to try to train a LoRA using Kohya_ss. For my use case, though, it seems because I want to train a LoRA for the model I like using, gonzalomoXLFluxPony_v40UnityXLDMD, the training should be very successful since the subject (woman) I'm using was created using that model. It would likely be a bit more difficult, I suppose, if I'm training on a subject/image just out in the wild. I'll see how it goes!
 
It's funny that the first one on your image there always has good bounce/physics but the rest not so much. :) And I guess the settings were the same for all of them. Things like this would probably drive me crazy if I wanted to do something like your JAV trailer.
 
  • Haha
Reactions: Casshern2
I just cursed myself to waste all my free time, at least for a bit. I had an idea and I know I'm not the first out there, but I thought about using some of the things I've learned on the way here to do something interesting. Something fun yet playfully sinister. I thought I'd try inviting some of my creations into the real world to play with me. Sounds odd? Sounds ugly? Sounds stupid? It was trippy, let me tell you.

The Plan!

Generate video clips of some of my lovely ladies, ahem...doing things to me...POV style. I know what you're thinking (probably not) but you'd be wrong. This would work and it would be kind of cool to see. Obviously this can incorporate more than just things you've generated, but that's another conversation. But that's what I set myself yesterday evening to figure out; to star in my own short 5-10 second clips of AI goodness.

The Workflow!
(not a standard ComfyUI Workflow, but a workflow as in steps)

I did a quick and dirty listing of the steps likely required to do this.
  1. take a picture of my pathetic self exposed to the world
  2. generate an image of a woman that suits my needs, or use an existing generation
  3. for close quarters POV crop out just her bust to her head
  4. use flux 2 klein and prompt a number of passes to place her in my photo, matching the lighting, positioned where I want her
  5. use my current favorite video generator, Wan 2.2, to prompt away and generate some video clip
The Results!

Now, obviously good ole' Casshern2 is not about to expose his pathetic self to you dear readers, so after I got the concept and steps down and made a good number of clips with different women I've generated, I then set out to do the show and tell, but showing someone else.

What I did was grab a screenshot from an adult website for Step 1. My actual photo was similar but in my bedroom on my bed, basic same POV view. For Step 2 I used an existing honey you might recognize. I did Step 3. Step 4 for this demo was a bit more involved. I used flux 2 klein to first remove the starlet already in the screenshot, then used flux 2 klein again to simulate putting her in my photo where it is just me. the true Step 4. Once I settled on an image that placed her well and didn't change her face all that much (flux 2 klein can definitely do that when you don't want it to). Step 5 wasn't that bad, I got the prompting pretty good by now. Not a book like some say are needed, but fleshed out enough to make things happen.

I think the results will give you an idea if you did this yourself....with yourself...it is pretty neat. Definitely personal, since I see my body in my bedroom in my house with all my (and my wife's) stuff in the background. Morally iffy, I guess haha! It's like I'm alone with them when nobody is home. That sinister part, or part of it!

The screenshot (I blacked out her eyes; they are not blacked when I prompted to remove her)
4610330.jpg

After she's removed
20261015_Flux2Klein_00011_.jpg

The bust I used (actual size)
04.png

The flux 2 klein generated images

20261015_Flux2Klein_00014_.jpg 20261015_Flux2Klein_00016_.jpg

I used this one
20261015_Flux2Klein_00015_.jpg

Here are some passes just linked together (no sound):