Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fail to run on colab #852

Open
2 tasks done
pusemuckel888 opened this issue Jul 24, 2023 · 12 comments
Open
2 tasks done

Fail to run on colab #852

pusemuckel888 opened this issue Jul 24, 2023 · 12 comments
Labels
bug Something isn't working

Comments

@pusemuckel888
Copy link

pusemuckel888 commented Jul 24, 2023

Describe the bug

EDIT: Issue seems solved. Waitingfor confirmation. DO NOT USE MY WORKAROUND!
Well, they updated the repo. There is no need for my workaround anymore. Do NOT run my two lines anymore!
END EDIT

There are alot of tickets saying that so-vits-svc-fork is bugged or broken and cannot be used especially with the example Colab Notebook.
What happened was, that a package from the requirements for this repo was changed and you manually should install

!pip install pyworld==0.3.2
!pip install numpy==1.23.5

After that continue with the first step "Check GPU" and you should be able to clone the repo and use it for training. For the last step "Inference" you might needto run this command:

!pip install numpy==1.24

Check your logs for errors and remember this is only a workaround. There might be more elegant solutions and I hope @voicepaw will update the repo soon.

#843
#840
#838
#837

To Reproduce

!pip install pyworld==0.3.2
!pip install numpy==1.23.5

Additional context

No response

Version

4.1.1

Platform

Colab

Code of Conduct

  • I agree to follow this project's Code of Conduct.

No Duplicate

  • I have checked existing issues to avoid duplicates.
@pusemuckel888 pusemuckel888 added the bug Something isn't working label Jul 24, 2023
@Salemation
Copy link

pusemuckel888, thank you for trying to fix things in an elegant matter. and thanks again for clearing things up. Now I don't know if this issue is the same, but when I am trying to train a model that I already trained to 900 epochs. Even when I still have the G_0 files for both. it finish the process but do not start training again.
Colab

@pusemuckel888
Copy link
Author

pusemuckel888 commented Jul 25, 2023

pusemuckel888, thank you for trying to fix things in an elegant matter. and thanks again for clearing things up. Now I don't know if this issue is the same, but when I am trying to train a model that I already trained to 900 epochs. Even when I still have the G_0 files for both. it finish the process but do not start training again. Colab

Might need to restart the runtime inbetween.
OR
Try without Tensorboard: add a "#" before the lines to comment them out and exclude from execution, like so:

#@title Train
#%load_ext tensorboard
#%tensorboard --logdir drive/MyDrive/so-vits-svc-fork/logs/44k
!svc train --model-path drive/MyDrive/so-vits-svc-fork/logs/44k

Maybe tensorboard causes issues for you.
Check if Training works by looking at the log folder in drive: New G_ and D_ fIles should be created.
OR
If this does not help, in the top menu, click on "View" and then "Executed code history" to show the full log of what is happening.
OR
Delete your latest D_ and G_ files. Maybe those are corrupted, when Colab happened to disconnect just intime when creating these checkpoints.

@Salemation
Copy link

Thank you a million for updating. I went through the steps you mentioned and kept doing again and again, but it seems not work. I might be doing. I want to make it clear. I left training then it reached 900 epochs. Today I watn to continue training again. So I restarted the runtime and went again step by step as to train it from the start, the same steps I mean. Should I cancel a step or so? maybe I am doing something wrong. I attached a video ..

2023-07-25.16-24-34.mp4

@Salemation
Copy link

This is the log file
Capturegfgfgfgf

@pusemuckel888
Copy link
Author

pusemuckel888 commented Jul 25, 2023

This is the log file Capturegfgfgfgf

Looks ok so far to me. Try this:
Copy your files from the log/44k directory, to keep them safe.
Try deleting both your G_929.pth and D_929.pth. Then system will continue with 922 files. If that does not help, delete all the files in log/44k and try from 0, just to make sure the script works correctly.

@Salemation
Copy link

I believe the colab is broken or something. I went through all the steps you mentioned, I even decided to start a fresh new colab, choose different dataset, When I reach train, it's not working. don't know what happened to it.

@pusemuckel888
Copy link
Author

pusemuckel888 commented Jul 25, 2023

I believe the colab is broken or something. I went through all the steps you mentioned, I even decided to start a fresh new colab, choose different dataset, When I reach train, it's not working. don't know what happened to it.

I wanted to ask you, if you even had your graphics card activated, but I could tell that you have from your video :)
Well, they updated the repo. There is no need for my workaround anymore. Do NOT run my two lines anymore!

I have been trying for like 3 hours now with a fresh new model and ran through all the steps multiple times and just now it works again. As my workaround is redundant, there must have been changes made by some devs I guess. Maybe they fixed it just now.

Pls try running without my workaround and tell us if works for you as well now.
UPDATE: I tested a new colab notebook with a fresh model and on kaggle i could successfully continue train an existing model.

@Salemation
Copy link

I believe the colab is broken or something. I went through all the steps you mentioned, I even decided to start a fresh new colab, choose different dataset, When I reach train, it's not working. don't know what happened to it.

I wanted to ask you, if you even had your graphics card activated, but I could tell that you have from your video :) Well, they updated the repo. There is no need for my workaround anymore. Do NOT run my two lines anymore!

I have been trying for like 3 hours now with a fresh new model and ran through all the steps multiple times and just now it works again. As my workaround is redundant, there must have been changes made by some devs I guess. Maybe they fixed it just now.

Pls try running without my workaround and tell us if works for you as well now. UPDATE: I tested a new colab notebook with a fresh model and on kaggle i could successfully continue train an existing model.

Hi [pusemuckel888],

Sorry for the late response, had to work on whatever I reached at that point and I would like to share with you what I created in here. By the way, I just tried it and it's still not working. The same thing happening, I don't know why when I reach Train. that's it done.

Please see the link below for the video, Cillian Murphy voice cloned and changed the language to arabic, while deepfaking him on.

https://www.instagram.com/p/CvIi4mjNs0o/

@pusemuckel888
Copy link
Author

The issue is still there, again. Step "Train" stops executing without error.
I will try to debug "svc train" later.

@34j 34j changed the title WORKAROUND: so-vits-svc-fork is outdated WORKAROUND: fails to run on colab Jul 30, 2023
@34j 34j changed the title WORKAROUND: fails to run on colab WORKAROUND: fail to run on colab Jul 30, 2023
@34j 34j changed the title WORKAROUND: fail to run on colab Fail to run on colab Jul 30, 2023
@pusemuckel888
Copy link
Author

Well, training works for me again.
One little change I made was from this:

#@title Copy configs file
!cp configs/44k/config.json drive/MyDrive/so-vits-svc-fork

to this:

#@title Copy configs file
!cp /content/configs/44k/config.json /content/drive/MyDrive/so-vits-svc-fork

@Dyslexicon
Copy link

Can the colab itself be repaired and updated or forked rather than requiring the end user to modify scripts up front?
This colab worked perfectly a few months ago despite the initial learning curve.

There are now numerous roadblocks; the loading of dependencies stage is riddled with errors, the configs folder and .json file creation step doesnt create either, and the Training/Tensorflow step fails.

Thanks to anyone who is able to get it working again.

@Vadim910
Copy link

Hi. when I did the ‘Install dependencies’ box (more specifically the command ‘%pip install -U ipython’)(1st screenshot), I am told that I need to restart the session and when I do this the environment does not start ‘for unknown reason’ and shows this in the logs(2nd screenshot). what should i do?

Image

Image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

5 participants
@Salemation @Vadim910 @Dyslexicon @pusemuckel888 and others