Overview
Fine-tuned a generative image model to help webtoon artists create secondary objects and supporting visual content that would otherwise require additional drawing time. I selected Stable Diffusion 3.5 Medium based on image quality and the available training hardware, then fine-tuned it on a webtoon dataset prepared from company-owned artwork. After validating the base model, I added an automated character LoRA workflow so users could create character-specific LoRA weights from a small set of reference images with minimal manual setup.
Objectives
- Adapt a general diffusion model to the visual domain and composition patterns of webtoon artwork.
- Build a scalable image-caption dataset suitable for diffusion fine-tuning.
- Enable character-specific adaptation from roughly ten reference images through a simple one-click workflow.
- Find stable fine-tuning and generation settings that produced usable results without requiring users to tune training parameters manually.
My Contribution
- Created a training dataset of approximately 80,000 webtoon image crops from company-owned content.
- Used CogVLM to generate descriptive captions and implemented automated checks to identify low-quality image-caption pairs for review.
- Compared captioning approaches and selected the caption format that produced more useful conditioning for diffusion training.
- Fine-tuned Stable Diffusion 3.5 Medium on the prepared image-caption dataset.
- Evaluated model checkpoints qualitatively using a fixed set of prompts and seeds to compare visual quality and webtoon-domain consistency.
- Built an automated character LoRA workflow in which users supplied a small reference set and received character-specific LoRA weights.
- Ran repeated experiments on learning rate, training steps, and related LoRA settings to identify robust defaults for the one-click workflow.
- Defined practical generation rules and prompt constraints to improve consistency when producing images from only a few character references.
Architecture
- Company webtoon images → automated cropping and quality filtering.
- Filtered images → CogVLM caption generation → automated caption checks and manual spot verification.
- Image-caption dataset → Stable Diffusion 3.5 Medium domain fine-tuning.
- Fixed prompts and seeds → qualitative checkpoint comparison and base-model selection.
- User uploads a small character image set → preprocessing → automated LoRA fine-tuning → reusable character LoRA weights.
- Character LoRA + generation rules → character-focused webtoon image generation.
Technical Stack
- Python
- PyTorch
- Stable Diffusion 3.5
- LoRA
- CogVLM
- SimpleTuner
- Diffusion Fine-Tuning
- Dataset Engineering
Main Technical Challenges
- Selecting a base diffusion model that provided a good balance between image quality, prompt following, and available training hardware.
- Generating sufficiently descriptive captions at scale without allowing caption errors to degrade the training dataset.
- Finding LoRA learning rates and training durations that worked reliably with a very small number of character images.
- Reducing the amount of manual parameter tuning required from users while maintaining consistent character-generation quality.
Outcome
The domain-adapted diffusion model and automated character LoRA workflow were used internally, enabling users to create character-specific LoRA weights from a small image set without manually configuring the fine-tuning process.
Demo
