Considerations needed for multi subject LoKr fine tuning for flux

[michaelcyshield]

Hello, I have pictures of one person and was able to have success in making flux learn how to draw them, however upon expanding it to three people, the model lost some capability in drawing likeness that well. I increased the number of steps so that there would be the same number of steps/subject but that did not do the trick. Should I increase the size of the lora? or are there any other considerations like learning rate that I need to adjust, currently my lr = 3e-4 and I train for 30k steps, so 10k per subject since I trained for 10k steps for just the one subject and achieved good results.

config.json
lycoris_config.json

[michaelcyshield]

I have roughly the same number of images for each subject which is 35, all should be high quality

[michaelcyshield]

I was thinking of doing these changes what do you guys think?

"--learning_rate": "1e-4",
"--lr_scheduler": "cosine_with_restarts",
"--train_batch_size": 2,
"--max_train_steps": 40000,

"linear_dim": 5120,
"factor": 16,
"apply_preset": {
"module_algo_map": {
"FeedForward": {
"factor": 12
}
}
},
"multiplier": 1.2