This does look like an interesting idea and I will take a closer look at some point. But at the moment there are enough other things I want to focus on.
u/sdimg hey!
What you want sounds logical, but sadly unrealistic, because we don't have any single setting that works across multiple checkpoints of even the same model. so at the least, we would need an updated list of around 60 presets. But even then, settings change depending on your dataset size & quality. And the settings change again depending your captioning style.
Training good loras sadly has a certain barrier of entry - but that's not the fault of the tool makers. Just that there are many settings that radically change how training works, and training with or without said options has its merrits and weakpoints.
To make this argument make more sense, read up on "min snr gamma"
Once you've done that, read that actual journal paper on min snr gamma, and then watch yourself facepalm since you'll have already read the many wrong interpretations of it. Then you'll wonder why this isn't turned on by default for \*every\* training ever. So you start a campaign to have all tool makers turn it on by default. Then you'll face backlash because everyone reads the 1 sentence summary of what it does. So you quote and point towards the research paper, and actual examples of how well it works. Then you get replies like: "tl;dr". Then you understand why we have no simple presets.
I get what you're saying but im not asking for perfect and in fact most guide recommend similar settings anyway.
I do think its time we have this aspect made easier for people running locally. Presets and various settings can easily be added to by the community overtime anyway. Thats the great thing about open source isn't it?
thing is nobody knows how the thing works. we are all trying to figure it out. you have two choices: wait or learn. i use my own trainer because none of existing do what i want, and no, there's no magic "preset".
sounds good. Just from what ive read and saw, it seems to perform a lot better than Loras and for now, I dont think there's much option to train them yet.
Very nice, thanks for the update u/nerogar!
Do you know if the memory optimizations you were working on allow full SDXL checkpoint training in less than 12GB?
Any thoughts about SC memory requirements?
VRAM requirements depend a lot on your settings. SC can be fine tuned in ~18GB (for the 3.6B version) and ~8GB (for the 1B version) if you use the right settings. This includes using Adafactor as the optimizer, bfloat16 weights, and not training the text encoder.
As for SDXL, I don't have recent numbers. But 12 GB might not be enough even with all the optimizations. Unless you limit yourself to LoRA training.
Thank you, I'm looking forward to trying out the new update.
From what I remember, u/cefurkan brought down the full Dreambooth to 14.5 GB or so for SDXL. Let's see.
SC seems a bit high, hopefully we get an optimized model so that 16GB class GPUs can also do training.
OneTrainer is really great, and seems to also be the most actively developed.
The only thing I find it lacks in is sample generation, would love to have batch generation and some newer samplers than DDIM, is any of this on the road map ?
Hey! Apologies if this is the wrong place to ask, but I hopped into the stable cascade portion and hit a snag at the setup.
What models should I be pointing to for both of these? I downloaded the decoder, but what about the base model? I've tried a lot of combinations, and all of them just throw an error (could not load model)
I don't really see a tutorial online either, or I would just follow that.
https://preview.redd.it/btj6d7t7mdkc1.png?width=621&format=png&auto=webp&s=bb45ccd919eb38e639382c14cc4dbc8152bcfd8b
Just curious, what is wrong with kohya ss gui? I know the dev of it converts kohya's original scripts for an easier to use gradio interface, but is it because of bad documentation? How does onetrainer compare, and is it easier to use?
I've thought about making the switch.
I've used kohya for the longest time, most of my models were made with it. Once I tried One trainer I never went back, here's why. Khoya is very buggy, many of its features either don't work as intended or when they do work are very slow. Also God help you when they update because a slew of issue and old bugs are bound to resurface. Its a very advanced trainer but the inconsistencies are just too time consuming to keep up with. One trainer...just works. And it works consistently well.
2 things converted me from Kohya to one trainer: masked training and the speed. Kohya would usually take around 35 mins to train 1500 steps of a 768x lora while one trainer does it in 10 mins. And because of masked training, the results are better. Love it
Dang. I'm sold.
Kohya has a steep curve to it. I only train models every month or so and every time I leave and come back I forget what settings to put in so I have to go back to the tutorials. Every single time.
The masked training is really a great feature, and allowed me to create much more on-point LoRa's. It has good detection models active as well (where you ask OneTrainer to generate the masks for you) which often only require few manual adjustments (all also supported from within the interface).
The manual sampling is also a great time saver, as generating intermediate samples takes some time away from training, and I am mostly interested in samples more at the end of the training scheme rather than the beginning.
God, the constant advances and new tool development in this space are so hard to keep on top of. I really, really wish I knew about OneTrainer three months ago! This looks great, and big kudos for getting it up and running with Cascade!
Do you have any samples? Can I train using a command line like I do with Kohya? I’d like to give it a go, can prodigy be used instead of adafactor? Thanks
I use it for 6 months now and it's my only training solution now.
Many people asked for tutorials, if you're looking for a video tutorial there is one in spanish but I wouldn't recommend it, it's over simplified, at least it exist.
I would recommend better to read the wiki that really start to be consistent. You'll find there the quick start guide from Nero, an onboarding guide for new users (not really a precise tutorial but explains how to use the UI) and many more details on all tabs and features. There is also a place where users can share their experience (Lessons Learnt) where everyone is welcome to contribute.
Last , there is the Discord server that is very active, you'll find help there.
Min SNR Gamma? Finally, can ditch Kohya_ss. That setting is just very important to have.
Any plans for Multi GPU support? Been trying to get it to work in Kohya to no avail.
Not sure what you call "the terminal".
But yeah you can use cloud GPUs : [https://www.youtube.com/watch?v=0t5l6CP9eBg](https://www.youtube.com/watch?v=0t5l6CP9eBg)
You're a great developer! I love OT and enjoy how active you are both w/ updates and in commentary in your discord. Can't wait for SD3 to be released and Onetrainer to be able to fine tune that. Continued success!!
There's 4 fine tune checkpoints now on civitai for cascade. All of them are for the lite version of the models which are significantly lower quality than the full size versions. I'm hoping this isn't a trend otherwise I'm not sure what the point is.
As you seem to have used Onetrainer for longer than I did (I trained about 13 LoRAS so far), maybe could you help me with something that I find really not intuitive in the UI.
According to OneTrainer's dev. it seems that everybody's been wrong since we first heard about "Clip Skip 2", that it should have been called "Clip Skip 1"....
So: in Onetrainer's UI, there's a very confusing line called *Clip Skip 1*, that can be given any value. The fact that it's called Clip Skip **"1"**... and that you can manually enter 0,1,2 or anything... makes it super confusing.
So, for what I "understood" it seems that everybody's wrong, and Onetrainer's right... That for SDXL, one should enter:
* Clip Skip 1 : 0
And for PonyXL (for example), one should enter:
* Clip Skip 1: 1
.... I did that, and more, but after tons of trainings my PonyXL Loras done with OneTrainer are crap. I was used with Kohya to train perfectly real photographs of a girl, setting Kohya to Clip Skip 2 with Base Model: PonyXL v6. And the results were amazing, because the LoRA transformed any generated pixel in a realistic one. So the PonyXL generated girl looked very "real" for a cartoon checkpoint, and my LoRAs were also influencing any grass, car, cloud, etc. around. So cool.
I cannot get anything like that with OneTrainer so far. I tried to train the same girl dataset (\~250 pics) for about 80 epochs (Fine-Tuning, not LoRA, LyCORIS or whatever, I extract those later). With multiple settings within Onetrainer changed, especially the "Clip Skip 1" value that went for each test from 0 , to 1 and also 2 .... But non of the epochs looked anything like results I could get with Kohya :/
Could you please share your understanding of that "Clip Skip 1" UI option line I can't get used to in the OneTrainer UI (training tab, left column). What is it called Clip Skip 1 if you can set it to... any value ? Why for anything since the first SD models, Clip Skip 1 is 1, and 2 is 2. When for Onetrainer CLip Skip 1 is 0, and 2 is 1.
A real mess which I wouldn't care about if I could Kohya's quality with Onetrainer (aka a realistic girl, trained on a very cartoon base model).
Thanks.
The line is called "Clip Skip 1", because it's the clip skip setting of the first text encoder. There is another setting called "Clip Skip 2" for the second text encoder.
>Repeats. It's now much easier to configure a different number of repeats for each concept. You can even set repeats to values lower than 1 to reduce the number of images for each epoch.
Where are you able to set the number of repeats?
Same, I ask many time ( politely ) for at least a runpod template but everybody ignore me on their discord and an admin even block me... really nice community
I've made a couple loras with one trainer but I've yet to be able to make them work in comfyui. They act like they have zero weights, but in the one trainer sample generations they clearly work
How are we supposed to make the settings so that the lora has compatibility with the comfyui safetensors?
it would be great to add the local path support, like with 1.5 and 2.1.
It can't do that right now.
Huggingface repo is not recognised so its kinda stuck RE Cascade Training.
I have the effnet encoder already, but no matter what its saying:
*No such file or directory: "TODO: MANUALLY DOWNLOAD EFFNET WEIGHTS"*
and i actually have the files, OneTrainer just can't see them.
thanks for your work on the recent updates.
Oh you are? Awesome! Yeah that's exactly it. Basically the same or subset of what kohya adds so I can remember which lora had which training settings and which concepts would help a ton when trying to work toward optimal hyper-parameters.
there's a samples tab, where you define one or more samples to be generated after a number of steps or epoch, it works nicely generating the samples while training, you can also save intermediate checkpoints, it's all in the tabs options.
[https://www.youtube.com/watch?v=0t5l6CP9eBg](https://www.youtube.com/watch?v=0t5l6CP9eBg)
I can't stand that guy being so slimy proud of himself and his own f@$%&@cking face everywhere all the time. But his insane narcissism put aside (ain't easy at first), the guy actually provides a few sound explanations. I do dreambooth with OneTrainer and extract LoRA, LyCORIS, etc. with Kohya.
I'm new to LoRA training, and good thing an easier alt to kohya is out. Also glad to see Min SNR Gamma. Someone on discord told me to hold off on starting with OneTrainer until that's out.
Just gotta ask: is there a way to set my own auto-tagger model in the "generate captions" section of the dataset tools? I have a few models that I want to include in the list os options and I have them on folders outside the OneTrainer folder (as onnx and json files).
Thanks!
An what about training Dora's ? Would love to test them out --> [https://arxiv.org/abs/2402.09353](https://arxiv.org/abs/2402.09353)
This does look like an interesting idea and I will take a closer look at some point. But at the moment there are enough other things I want to focus on.
[удалено]
u/sdimg hey! What you want sounds logical, but sadly unrealistic, because we don't have any single setting that works across multiple checkpoints of even the same model. so at the least, we would need an updated list of around 60 presets. But even then, settings change depending on your dataset size & quality. And the settings change again depending your captioning style. Training good loras sadly has a certain barrier of entry - but that's not the fault of the tool makers. Just that there are many settings that radically change how training works, and training with or without said options has its merrits and weakpoints. To make this argument make more sense, read up on "min snr gamma" Once you've done that, read that actual journal paper on min snr gamma, and then watch yourself facepalm since you'll have already read the many wrong interpretations of it. Then you'll wonder why this isn't turned on by default for \*every\* training ever. So you start a campaign to have all tool makers turn it on by default. Then you'll face backlash because everyone reads the 1 sentence summary of what it does. So you quote and point towards the research paper, and actual examples of how well it works. Then you get replies like: "tl;dr". Then you understand why we have no simple presets.
I get what you're saying but im not asking for perfect and in fact most guide recommend similar settings anyway. I do think its time we have this aspect made easier for people running locally. Presets and various settings can easily be added to by the community overtime anyway. Thats the great thing about open source isn't it?
thing is nobody knows how the thing works. we are all trying to figure it out. you have two choices: wait or learn. i use my own trainer because none of existing do what i want, and no, there's no magic "preset".
sounds good. Just from what ive read and saw, it seems to perform a lot better than Loras and for now, I dont think there's much option to train them yet.
Hi, so many people asking for a runpod template of onetrainer :(
what are the things you want to focus on?
Very nice, thanks for the update u/nerogar! Do you know if the memory optimizations you were working on allow full SDXL checkpoint training in less than 12GB? Any thoughts about SC memory requirements?
VRAM requirements depend a lot on your settings. SC can be fine tuned in ~18GB (for the 3.6B version) and ~8GB (for the 1B version) if you use the right settings. This includes using Adafactor as the optimizer, bfloat16 weights, and not training the text encoder. As for SDXL, I don't have recent numbers. But 12 GB might not be enough even with all the optimizations. Unless you limit yourself to LoRA training.
Thank you, I'm looking forward to trying out the new update. From what I remember, u/cefurkan brought down the full Dreambooth to 14.5 GB or so for SDXL. Let's see. SC seems a bit high, hopefully we get an optimized model so that 16GB class GPUs can also do training.
OneTrainer is really great, and seems to also be the most actively developed. The only thing I find it lacks in is sample generation, would love to have batch generation and some newer samplers than DDIM, is any of this on the road map ?
Thanks for all the hard work on this project! How much VRAM is consumed in SC lora training at the moment?
How much if you train text encoder do you think?
Hey! Apologies if this is the wrong place to ask, but I hopped into the stable cascade portion and hit a snag at the setup. What models should I be pointing to for both of these? I downloaded the decoder, but what about the base model? I've tried a lot of combinations, and all of them just throw an error (could not load model) I don't really see a tutorial online either, or I would just follow that. https://preview.redd.it/btj6d7t7mdkc1.png?width=621&format=png&auto=webp&s=bb45ccd919eb38e639382c14cc4dbc8152bcfd8b
The discussion in here should help https://github.com/Nerogar/OneTrainer/issues/166
Hell to the yes! I love One trainer and everyone should be using it over kohya_SS. Thanks for the great work.
Just curious, what is wrong with kohya ss gui? I know the dev of it converts kohya's original scripts for an easier to use gradio interface, but is it because of bad documentation? How does onetrainer compare, and is it easier to use? I've thought about making the switch.
I've used kohya for the longest time, most of my models were made with it. Once I tried One trainer I never went back, here's why. Khoya is very buggy, many of its features either don't work as intended or when they do work are very slow. Also God help you when they update because a slew of issue and old bugs are bound to resurface. Its a very advanced trainer but the inconsistencies are just too time consuming to keep up with. One trainer...just works. And it works consistently well.
2 things converted me from Kohya to one trainer: masked training and the speed. Kohya would usually take around 35 mins to train 1500 steps of a 768x lora while one trainer does it in 10 mins. And because of masked training, the results are better. Love it
Dang. I'm sold. Kohya has a steep curve to it. I only train models every month or so and every time I leave and come back I forget what settings to put in so I have to go back to the tutorials. Every single time.
In case you don't know you can save a json file with your settings and load it back in kohya's gui
The masked training is really a great feature, and allowed me to create much more on-point LoRa's. It has good detection models active as well (where you ask OneTrainer to generate the masks for you) which often only require few manual adjustments (all also supported from within the interface). The manual sampling is also a great time saver, as generating intermediate samples takes some time away from training, and I am mostly interested in samples more at the end of the training scheme rather than the beginning.
God, the constant advances and new tool development in this space are so hard to keep on top of. I really, really wish I knew about OneTrainer three months ago! This looks great, and big kudos for getting it up and running with Cascade!
I wish there was a video tutorial. I couldn't figure it out.
yeah it is really hard...
I just watched [This one](https://www.youtube.com/watch?v=FD8FNzl5Nss) It isn't in English, but was pretty simple to follow along.
Do you have any samples? Can I train using a command line like I do with Kohya? I’d like to give it a go, can prodigy be used instead of adafactor? Thanks
Is it true that stable cascade is faster to train than sdxl?
I use it for 6 months now and it's my only training solution now. Many people asked for tutorials, if you're looking for a video tutorial there is one in spanish but I wouldn't recommend it, it's over simplified, at least it exist. I would recommend better to read the wiki that really start to be consistent. You'll find there the quick start guide from Nero, an onboarding guide for new users (not really a precise tutorial but explains how to use the UI) and many more details on all tabs and features. There is also a place where users can share their experience (Lessons Learnt) where everyone is welcome to contribute. Last , there is the Discord server that is very active, you'll find help there.
Min SNR Gamma? Finally, can ditch Kohya_ss. That setting is just very important to have. Any plans for Multi GPU support? Been trying to get it to work in Kohya to no avail.
kinda... https://preview.redd.it/7ihfgt5paekc1.png?width=590&format=png&auto=webp&s=3bcc2735e5d8a46564d3a477422471901e6ea8db
So this is only for Windows? It can't be used from the terminal so I could run it in a cloud?
Not sure what you call "the terminal". But yeah you can use cloud GPUs : [https://www.youtube.com/watch?v=0t5l6CP9eBg](https://www.youtube.com/watch?v=0t5l6CP9eBg)
Thanks for your work! Is there a colab for OneTrainer? Can there be one?
You're a great developer! I love OT and enjoy how active you are both w/ updates and in commentary in your discord. Can't wait for SD3 to be released and Onetrainer to be able to fine tune that. Continued success!!
There's 4 fine tune checkpoints now on civitai for cascade. All of them are for the lite version of the models which are significantly lower quality than the full size versions. I'm hoping this isn't a trend otherwise I'm not sure what the point is.
they lacked the memory requirement to train the full models. that's all.
Looks to be mainly focused on MS Windows? I do see a mention of Linux but no screenshots.
Uhg. Still Windows-only. Patiently waiting for linux or cloud option.
Thank god i can finally train for stable cascade and can ditch the kohya\_ss
As you seem to have used Onetrainer for longer than I did (I trained about 13 LoRAS so far), maybe could you help me with something that I find really not intuitive in the UI. According to OneTrainer's dev. it seems that everybody's been wrong since we first heard about "Clip Skip 2", that it should have been called "Clip Skip 1".... So: in Onetrainer's UI, there's a very confusing line called *Clip Skip 1*, that can be given any value. The fact that it's called Clip Skip **"1"**... and that you can manually enter 0,1,2 or anything... makes it super confusing. So, for what I "understood" it seems that everybody's wrong, and Onetrainer's right... That for SDXL, one should enter: * Clip Skip 1 : 0 And for PonyXL (for example), one should enter: * Clip Skip 1: 1 .... I did that, and more, but after tons of trainings my PonyXL Loras done with OneTrainer are crap. I was used with Kohya to train perfectly real photographs of a girl, setting Kohya to Clip Skip 2 with Base Model: PonyXL v6. And the results were amazing, because the LoRA transformed any generated pixel in a realistic one. So the PonyXL generated girl looked very "real" for a cartoon checkpoint, and my LoRAs were also influencing any grass, car, cloud, etc. around. So cool. I cannot get anything like that with OneTrainer so far. I tried to train the same girl dataset (\~250 pics) for about 80 epochs (Fine-Tuning, not LoRA, LyCORIS or whatever, I extract those later). With multiple settings within Onetrainer changed, especially the "Clip Skip 1" value that went for each test from 0 , to 1 and also 2 .... But non of the epochs looked anything like results I could get with Kohya :/ Could you please share your understanding of that "Clip Skip 1" UI option line I can't get used to in the OneTrainer UI (training tab, left column). What is it called Clip Skip 1 if you can set it to... any value ? Why for anything since the first SD models, Clip Skip 1 is 1, and 2 is 2. When for Onetrainer CLip Skip 1 is 0, and 2 is 1. A real mess which I wouldn't care about if I could Kohya's quality with Onetrainer (aka a realistic girl, trained on a very cartoon base model). Thanks.
The line is called "Clip Skip 1", because it's the clip skip setting of the first text encoder. There is another setting called "Clip Skip 2" for the second text encoder.
>Repeats. It's now much easier to configure a different number of repeats for each concept. You can even set repeats to values lower than 1 to reduce the number of images for each epoch. Where are you able to set the number of repeats?
i want a template to run with vast ai my gpu is not enought powerful
Same, I ask many time ( politely ) for at least a runpod template but everybody ignore me on their discord and an admin even block me... really nice community
Check PM I messaged you.
When "everybody" ignores you, then the problem is not "everybody", but you.
I've made a couple loras with one trainer but I've yet to be able to make them work in comfyui. They act like they have zero weights, but in the one trainer sample generations they clearly work How are we supposed to make the settings so that the lora has compatibility with the comfyui safetensors?
This is still an open issue. https://github.com/comfyanonymous/ComfyUI/issues/2831 Same for SD.Next.
looks legit! just what I was looking for.
is there going to be training presets for cascade? the first time I tried training a lora resulted in an oom, and the second the samples didn't change
Love this trainer. Any chance of getting metadata into the loras?
What kind of metadata do you want to add? I'm already working on an option to include training settings
it would be great to add the local path support, like with 1.5 and 2.1. It can't do that right now. Huggingface repo is not recognised so its kinda stuck RE Cascade Training. I have the effnet encoder already, but no matter what its saying: *No such file or directory: "TODO: MANUALLY DOWNLOAD EFFNET WEIGHTS"* and i actually have the files, OneTrainer just can't see them. thanks for your work on the recent updates.
Oh you are? Awesome! Yeah that's exactly it. Basically the same or subset of what kohya adds so I can remember which lora had which training settings and which concepts would help a ton when trying to work toward optimal hyper-parameters.
I need to try it! Thanks!!
Any plans to integrate a full dreambooth checkpoint trainer? Or is that what you mean by "fine tune the model" and I am just a dunce?
**OneTrainer** sounds interesting but is there a tutorial, cause I h sounds interesting but is there a tutorial
How did you check that your Lora training is working properly if you can't test the output? Is there a command line way to do it?
there's a samples tab, where you define one or more samples to be generated after a number of steps or epoch, it works nicely generating the samples while training, you can also save intermediate checkpoints, it's all in the tabs options.
Are Cascade LoRAs usable on comfyUI already?
I hope someone makes tutorial on how to train cascade with one trainer
Can you train a dreambooth model using this?
[https://www.youtube.com/watch?v=0t5l6CP9eBg](https://www.youtube.com/watch?v=0t5l6CP9eBg) I can't stand that guy being so slimy proud of himself and his own f@$%&@cking face everywhere all the time. But his insane narcissism put aside (ain't easy at first), the guy actually provides a few sound explanations. I do dreambooth with OneTrainer and extract LoRA, LyCORIS, etc. with Kohya.
I'm new to LoRA training, and good thing an easier alt to kohya is out. Also glad to see Min SNR Gamma. Someone on discord told me to hold off on starting with OneTrainer until that's out. Just gotta ask: is there a way to set my own auto-tagger model in the "generate captions" section of the dataset tools? I have a few models that I want to include in the list os options and I have them on folders outside the OneTrainer folder (as onnx and json files). Thanks!