demobook

ComfyUI: Character replacement with Wan Animate

ComfyUITry it →Watch full video →pixaroma ·

Demo summary

The creator demonstrates using Wan Animate and SAM 3 to mask a character in a reference video and replace them with a new character from a static image while maintaining the original motion.

Step-by-step

  1. Download and place SAM 3, body detection, and object detection models into their respective folders within the models directory.
  2. Download the Wan Animate FP8 or full diffusion model and place it in the diffusion_models/wan_2.1 folder.
  3. Download the five required LoRAs (Relight, Lighting, Pusura, Reward, and Fast) and place them in the loras/wan_2.1 folder.
  4. Download and save the VAE, Clip Vision, and Text Encoder models into their specific subfolders.
  5. Use the ComfyUI Manager to 'Install Missing Custom Nodes' to ensure all workflow components are present.
  6. Upload a source image and a reference video with matching aspect ratios and similar compositions.
  7. Adjust the width and height in the resolution node to match your media's ratio.
  8. Specify the subject type in the masking node (e.g., 'person') and click the run button.

Options

  • Use the full model instead of the FP8 version if you have high VRAM
  • Install nodes manually via Custom Nodes Manager instead of using 'Install Missing Custom Nodes'
  • Change the masking prompt if the subject is not a person

Watch out for

  • You must create specific subfolders (SAM3, detection, wan_2.1) if they do not already exist.
  • The body detection model requires a specific .bin file in the same folder to function.
  • High VRAM and RAM are required to run the five simultaneous LoRAs.
  • The reference video and source image must have the same ratio and similar composition for good results.

Tips

  • Use the FP8 version of the animation model to save on VRAM.
  • Follow the suggested resolution sizes provided in the workflow for each aspect ratio.
  • Don't worry about understanding the complex node connections; focus on the input and output nodes.
  • Use the Relight LoRA to maintain consistent and improved lighting across the video.

Highlights

“you don't need to know everything that is happening inside a car to be able to drive that car.”

All demos from “ComfyUI Video Models: InfiniteTalk + Wan 2.2 + SCAIL + LTX-2 (Ep06)”

  1. 5:244:57Image-to-video generation with Wan 2.2The user demonstrates how to use the Wan 2.2 GGUF model in ComfyUI to animate a static image of a woman based on a detailed text prompt.ComfyUIImage to Video
  2. 13:071:21Applying LoRA to Wan 2.2 video generationThe video shows how to integrate a LoRA (Low-Rank Adaptation) into a Wan 2.2 workflow to achieve specific cinematic movements like a face zoom.ComfyUIAI Animation Generator
  3. 15:344:59Character replacement with Wan AnimateCurrentThe creator demonstrates using Wan Animate and SAM 3 to mask a character in a reference video and replace them with a new character from a static image while maintaining the original motion.ComfyUIVideo to Video
  4. 21:072:43Cartoon animation with Wan SCAILThe user shows how to animate a cartoon ballerina using a real-person video as a motion reference via the Wan SCAIL workflow in ComfyUI.ComfyUIVideo to Video
  5. 25:063:04Create talking avatars with InfiniteTalkThe demonstration shows how to sync a static character image with an audio file to create a talking avatar using the InfiniteTalk model and Wan 2.1.ComfyUIAI Avatar Video Generator
  6. 31:133:44Text-to-video with LTX-2The video walks through setting up the LTX-2 model in ComfyUI to generate high-resolution video clips from text prompts and images.ComfyUIAI Animation Generator
  7. 36:000:39Singing characters with LTX-2 and custom audioThe creator demonstrates how to use LTX-2 to make a character sing by providing a custom audio file and a specific singing prompt.ComfyUIAI Lip Sync Generator
  8. 37:281:08Upscale video with Seed-V2The user demonstrates upscaling a low-resolution AI-generated video to Full HD using the Seed-V2 workflow in ComfyUI for improved sharpness.ComfyUIAI Video Upscaler
  9. 39:075:27Cloud-based ComfyUI on RunPod/RunHubThe video shows how to run complex video workflows in the cloud using RunHub AI, demonstrating the interface and execution of InfiniteTalk and Wan 2.2 without local hardware.ComfyUIAI Animation Generator
  10. 44:340:52Frame interpolation for smoother motionThe user demonstrates a workflow to double the frame rate of a 16fps video to 32fps to create smoother motion in AI-generated clips.ComfyUIAI Video Interpolation
  11. Watch “ComfyUI Video Models: InfiniteTalk + Wan 2.2 + SCAIL + LTX-2 (Ep06)” →

Video to Video

  1. 0:000:54Mocha character replacement showcaseThe video demonstrates Mocha's ability to replace characters in existing videos with new characters from reference images while maintaining original movements and lighting.AI Search
  2. 2:280:44Swap human character for 3D Pixar model in MochaThe creator demonstrates swapping a real person in a video with a 3D Pixar-style character using Mocha, showing the transfer of lip movements and hand gestures.AI Search
  3. 20:110:27Generating final character swap videoThe video shows the K-Sampler process and the final rendered output of a character replacement task within ComfyUI.AI Search
  4. 15:344:59Character replacement with Wan AnimateCurrentThe creator demonstrates using Wan Animate and SAM 3 to mask a character in a reference video and replace them with a new character from a static image while maintaining the original motion.pixaroma
  5. 21:072:43Cartoon animation with Wan SCAILThe user shows how to animate a cartoon ballerina using a real-person video as a motion reference via the Wan SCAIL workflow in ComfyUI.pixaroma
  6. 8:150:26Using Keyframes to guide video generationThe demo shows how to slide an image to a later point in the timeline to use it as a target keyframe, allowing the model to generate the action leading up to that specific visual.What Dreams Cost
  7. 15:352:08Final video generation with Wan 2.1The demonstration shows how to use the Wan 2.1 V2V model to combine the generated 3D motion and a reference character image into a final, high-quality AI video animation.Benji’s AI Playground
  8. 1:003:01Generate character-consistent video with Wan 2.6 R2VThe creator demonstrates a ComfyUI workflow using the Wan 2.6 API node to transform a 5-second reference video of himself into a new scene while maintaining character consistency and native audio lip-sync.Sebastian Kamph
  9. 4:031:00Generating Guide Videos with BFS NodeThe demo shows how to use the BFS Node extension to combine a reference face image and a driving video into a green-screen guide video required for the LTX sampling process.Veteran AI
  10. 5:030:47Injecting Latent Constraints with Add Guide MultiThe video demonstrates using the 'Add Guide Multi' node from the KJ Node extension to inject the guide video constraints and VAE-encoded latents into the sampling pipeline.Veteran AI
  11. 15:031:04Apply camera movement from reference videoThe creator demonstrates copying camera motion and pose dynamics from a reference video (like a dance clip) and applying it to a still character image.Yaroflasher
  12. 1:551:00Adding reference images for character consistencyThe video shows how to use the MSR custom node to input up to four reference images and a background image to maintain character facial consistency across video frames.bigboss97
demobook

Show, don’t tell.

Categories

Video & AnimationImage Generation & EditingMusic & AudioSales TechEmail Marketing

About

SubmitAboutPrivacyTermsContact

© demobook.co