Direct Windows support for Unsloth! #1849
Replies: 10 comments 13 replies
-
|
Btw @adityaghai07 @Captain-T2004 would you guys mind checking to see if I wrote the documentation correctly? It's Method #1: https://docs.unsloth.ai/get-started/installing-+-updating/windows-installation Thank you so much! |
Beta Was this translation helpful? Give feedback.
-
|
@shimmyshimmer Looks great! Just one thing , about the torch installation with cuda , torch install provides a pip install command with the compatible cuda version which I believe would make the process smoother. |
Beta Was this translation helpful? Give feedback.
-
|
I think it's good if also add python version and anaconda create command: |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for updating our Github steps @adityaghai07 @Captain-T2004 , I updated it again to make it cleaner. For some reason the image you guys uploaded got removed: https://github.com/unslothai/unsloth#windows-installation |
Beta Was this translation helpful? Give feedback.
-
|
@shimmyshimmer, I have made a PR to add the image to unsloth/images folder... |
Beta Was this translation helpful? Give feedback.
-
|
So, i have an issue: #1870 |
Beta Was this translation helpful? Give feedback.
-
|
Hey @areebuzair, The error is likely due to missing or incorrect install of cmake. In your visual studio code installed go to modify and then individual components and make sure all of these are selected and installed. And also make sure you are able to execute cmake --help in your powershell. |
Beta Was this translation helpful? Give feedback.
-
|
I switched to Python 3.12. Cmake does not work in PowerShell, so I ran my Python script from the MSVC developer command prompt, instead. |
Beta Was this translation helpful? Give feedback.
-
|
Not sure why, but have no version attribute.
|
Beta Was this translation helpful? Give feedback.
-
|
I managed to get Unsloth to work, but I ran into this Triton error. So I had to fix that. Also, I had to run my Python code from the MSVC Developer terminal so that I could get access to the CMake function. Also, to download the trained model, we need to build Llama.cpp manually. |
Beta Was this translation helpful? Give feedback.
-
|
unsloth env set up I get this issue Tried vllm installation with every time the issue below pops up |
Beta Was this translation helpful? Give feedback.
-
|
@leninloitongbam can you please confirm that you created a virtual environment first and then followed step by step as mentioned in the readme/documentation ? |
Beta Was this translation helpful? Give feedback.
-
|
yes, I created virtual env using "virtualenv -p py_path envname". tried old method with triton-windows (https://github.com/woct0rdho/triton-windows) installation every time the issue below pops up for window installation the issue is with vllm installation, and I think windows installation is happening incorrectly. |
Beta Was this translation helpful? Give feedback.
-
|
I encountered problems when I export the model on windows, This is the code. # Save to 8bit Q8_0
if True: model.save_pretrained_gguf("model", tokenizer)
# Remember to go to https://huggingface.co/settings/tokens for a token!
# And change hf to your username!
if False: model.push_to_hub_gguf("hf/model", tokenizer, token = "")
# Save to 16bit GGUF
if False: model.save_pretrained_gguf("model", tokenizer, quantization_method = "f16")
if False: model.push_to_hub_gguf("hf/model", tokenizer, quantization_method = "f16", token = "")
# Save to q4_k_m GGUF
if False: model.save_pretrained_gguf("model", tokenizer, quantization_method = "q4_k_m")
if False: model.push_to_hub_gguf("hf/model", tokenizer, quantization_method = "q4_k_m", token = "")
# Save to multiple GGUF options - much faster if you want multiple!
if False:
model.push_to_hub_gguf(
"hf/model", # Change hf to your username!
tokenizer,
quantization_method = ["q4_k_m", "q8_0", "q5_k_m"],
token = "", # Get a token at https://huggingface.co/settings/tokens
)And this is the problem |
Beta Was this translation helpful? Give feedback.
-
|
Hey @ispacewang, can you try deleting the llama.cpp folder in your working directory and running the command again. I believe what happened was the command was cloning llama.cpp but it was interrupted and now when ran again it expects llama.cpp to be setup correctly but it's not. |
Beta Was this translation helpful? Give feedback.
-
|
new error😥 RuntimeError Traceback (most recent call last)
File ~\miniconda3\Lib\site-packages\unsloth\save.py:1703, in unsloth_save_pretrained_gguf(self, save_directory, tokenizer, quantization_method, first_conversion, push_to_hub, token, private, is_main_process, state_dict, save_function, max_shard_size, safe_serialization, variant, save_peft_format, tags, temporary_location, maximum_memory_usage)
1702 try:
-> 1703 new_save_directory, old_username = unsloth_save_model(**arguments)
1704 makefile = None
File ~\miniconda3\Lib\site-packages\torch\utils\_contextlib.py:116, in context_decorator.<locals>.decorate_context(*args, **kwargs)
115 with ctx_factory():
--> 116 return func(*args, **kwargs)
File ~\miniconda3\Lib\site-packages\unsloth\save.py:587, in unsloth_save_model(model, tokenizer, save_directory, save_method, push_to_hub, token, is_main_process, state_dict, save_function, max_shard_size, safe_serialization, variant, save_peft_format, use_temp_dir, commit_message, private, create_pr, revision, commit_description, tags, temporary_location, maximum_memory_usage)
586 filename = os.path.join(temporary_location, f"{name}.pt")
--> 587 torch.save(W, filename, pickle_module = pickle, pickle_protocol = pickle.HIGHEST_PROTOCOL,)
588 # weights_only = True weirdly fails?
File ~\miniconda3\Lib\site-packages\torch\serialization.py:943, in save(obj, f, pickle_module, pickle_protocol, _use_new_zipfile_serialization, _disable_byteorder_record)
942 if _use_new_zipfile_serialization:
--> 943 with _open_zipfile_writer(f) as opened_zipfile:
944 _save(
945 obj,
946 opened_zipfile,
(...) 949 _disable_byteorder_record,
950 )
File ~\miniconda3\Lib\site-packages\torch\serialization.py:810, in _open_zipfile_writer(name_or_buffer)
809 container = _open_zipfile_writer_buffer
--> 810 return container(name_or_buffer)
File ~\miniconda3\Lib\site-packages\torch\serialization.py:781, in _open_zipfile_writer_file.__init__(self, name)
780 else:
--> 781 super().__init__(torch._C.PyTorchFileWriter(self.name, _compute_crc32))
RuntimeError: File _unsloth_temporary_saved_buffers\model.layers.2.mlp.gate_proj.weight.pt cannot be opened.
During handling of the above exception, another exception occurred:
RuntimeError Traceback (most recent call last)
Cell In[23], line 2
1 # Save to 8bit Q8_0
----> 2 if True: model.save_pretrained_gguf("model", tokenizer)
3 # Remember to go to https://huggingface.co/settings/tokens for a token!
4 # And change hf to your username!
5 if False: model.push_to_hub_gguf("hf/model", tokenizer, token = "")
File ~\miniconda3\Lib\site-packages\unsloth\save.py:1718, in unsloth_save_pretrained_gguf(self, save_directory, tokenizer, quantization_method, first_conversion, push_to_hub, token, private, is_main_process, state_dict, save_function, max_shard_size, safe_serialization, variant, save_peft_format, tags, temporary_location, maximum_memory_usage)
1716 python_install = install_python_non_blocking(["gguf", "protobuf"])
1717 git_clone.wait()
-> 1718 makefile = install_llama_cpp_make_non_blocking()
1719 new_save_directory, old_username = unsloth_save_model(**arguments)
1720 python_install.wait()
File ~\miniconda3\Lib\site-packages\unsloth\save.py:783, in install_llama_cpp_make_non_blocking()
781 check = os.system("cmake llama.cpp -B llama.cpp/build -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=OFF -DLLAMA_CURL=ON")
782 if check != 0:
--> 783 raise RuntimeError(f"*** Unsloth: Failed compiling llama.cpp using os.system(...) with error {check}. Please report this ASAP!")
784 pass
785 # f"cmake --build llama.cpp/build --config Release -j{psutil.cpu_count()*2} --clean-first --target {' '.join(LLAMA_CPP_TARGETS)}",
RuntimeError: *** Unsloth: Failed compiling llama.cpp using os.system(...) with error 1. Please report this ASAP! |
Beta Was this translation helpful? Give feedback.
-
|
I clicked the run button again, and it run into the same problem before. I used the llama.cpp/convert_lora_to_gguf.py script to generate a gguf file . I used ollama to run it but it didnt work. It says that it cant find the model, but ollama list shows the model name. |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
@ee22mtech14003 This is a torch compatibility issue. import torch
torch.cuda.is_available()If this returns false then your torch and cuda are not properly setup. |
Beta Was this translation helpful? Give feedback.
-
|
when i import unsloth，there is an error:
pytorch2.6+cu126，gpu card GTX1080ti，python3.12 |
Beta Was this translation helpful? Give feedback.
-
|
cuda_12.1, windows 11 pip install "unsloth[windows] @ git+https://github.com/unslothai/unsloth.git" after the pip command , i found its install torch2.6 with out cuda , so i install torch with this: pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121 the MSVC version is 19.43 after install when i import unsloth i got this error |
Beta Was this translation helpful? Give feedback.
-
|
@adityaghai07 @Captain-T2004 It looks like Triton can be pip installed on Windows! https://pypi.org/project/triton-windows/
|
Beta Was this translation helpful? Give feedback.
-
|
Great to know that! Me and @Captain-T2004 have made a PR for the same. |
Beta Was this translation helpful? Give feedback.
-
Thanks to @adityaghai07 and @Captain-T2004 in #1841 you are now able to install Unsloth directly via Windows.
TUTORIAL: https://docs.unsloth.ai/get-started/installing-+-updating/windows-installation
If anyone has tested it out - please let us know if it works well and if there are any issues! Thank you
Feature Overview
These changes allow user to install unsloth on windows directly without any dependency issues using a simple command:
Implementation Details
Reasoning
Testing Details
Beta Was this translation helpful? Give feedback.
All reactions