Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add uv as package manger for the generated project #5434

Open
wants to merge 22 commits into
base: master
Choose a base branch
from

Conversation

foarsitter
Copy link
Collaborator

@foarsitter foarsitter commented Oct 6, 2024

The project itself now uses uv and this is an attempt to bring uv to the generated project too.

  • Python commands are executed through uv run for local development. In Docker the .venv python interpreter is added to the $PATH so packages can be used system wide.
  • The uv.lock file is empty on start because is is updated when uv sync is executed.
  • For Heroku I added a pre-commit that compiles a requirements.txt
  • Added a trap to test_docker.sh to cleanup after testing so the next time will no fail due to an existing postgres volume.
  • In test_docker.sh I added a step to test building the docker production image

Before we can merge this we need one last round to update all the versions to the latest ones in the requirement.txt files.

@foarsitter foarsitter changed the title Uv generated project Add uv as package manger for the generated project Oct 7, 2024
@foarsitter
Copy link
Collaborator Author

Related to #3083 #1988

@foarsitter foarsitter force-pushed the uv-generated-project branch 2 times, most recently from bd785eb to 6bae8e7 Compare October 26, 2024 09:48
@foarsitter foarsitter marked this pull request as ready for review October 26, 2024 15:08
@foarsitter
Copy link
Collaborator Author

foarsitter commented Oct 26, 2024

Ready for testing 🎉

cookiecutter https://github.com/foarsitter/cookiecutter-django --checkout uv-generated-project

@foarsitter foarsitter force-pushed the uv-generated-project branch from 8f3ca43 to d12c2e7 Compare October 27, 2024 08:17
Copy link
Member

@browniebroke browniebroke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did a first pass and noted a few things. Haven't reviewed the Dockerfiles yet

@steveputman
Copy link
Contributor

Hi, there. I played around with using your fork to update some existing projects and eventually got them working but ended up reverting them to their production (pip) versions so that I could revisit when I had more time.

When working on an adapting an existing project, I got an error during the build, which was reproduced when I built a project from scratch using cookiecutter https://github.com/foarsitter/cookiecutter-django --checkout uv-generated-project:

[django python-build-stage 4/6] RUN --mount=type=cache,target=/root/.cache/uv --mount=type=bind,source=uv.lock,target=uv.lock --mount=type=bind,source=pyproject.toml,target=pyproject.toml uv sync --frozen --no-install-project --no-dev:
0.112 Using CPython 3.12.7 interpreter at: /usr/local/bin/python
0.112 Creating virtual environment at: .venv
0.113 error: Could not find root package my-awesome-project
'

I believe that it's caused by having the blank uv.lock. I thought since the local project had the host file rw that it would generate a usable uv.lock during the local build. Is there a way to get a usable version of uv.lock on the host filesystem without running uv sync on the host filesystem? Thanks for this--very cool!

@foarsitter
Copy link
Collaborator Author

@steveputman thanks looking into this!

During a build you cannot write to the host system.

Your interpreter is inside the container together with uv so you can run uv:

docker compose -f docker-compose.local.yml run django uv lock

Struggled a lot with this and it is not ideal. So hopefully someone has a better solution.

@steveputman
Copy link
Contributor

@foarsitter
The Docker documentation was sort of vague on that, but now I know.

Your solution is better than mine (I had uv installed on my host system so just ran uv lock in the project directory.

Thanks for the help! Will let you know if anything else comes up.

@rtpg
Copy link

rtpg commented Jan 3, 2025

I think you need to run uv sync on various start scripts (like celery), at least locally. Because you're mounting a folder to /app/.venv we actually don't have the installed virtualenv created in the base Dockerfile.

Of course we're also not sharing a virtualenv here, by my read/attempt to use this. It might be possible to just share a mount across all the services, but I'm having trouble getting that to work.

@pievalentin
Copy link

Great work! This project really need to have a better tool than the plain pip

@pievalentin
Copy link

pievalentin commented Feb 2, 2025

The dependabot would need updating too. I dont know if it supports pyproject declarations

@leandrodesouzadev
Copy link

The dependabot would need updating too. I dont know if it supports pyproject declarations

I believe it does, I've found some discussions on the project, here's the link to the issue.

@browniebroke
Copy link
Member

The dependabot would need updating too. I dont know if it supports pyproject declarations

This has been added now: dependabot/dependabot-core#10478 (comment)

@pievalentin
Copy link

a git pull away from a merge 😿

@foobacca
Copy link

foobacca commented Mar 10, 2025

Tried it. I wonder if the requirements currently in requirements/production.txt should be moved to an optional dependencies section in pyproject.toml, so something like:

[project.optional-dependencies]
production = [
  "gunicorn==23.0.0",
  # ...
]

Currently they appear in the main project dependencies, so they will be installed in the development virtualenv.

They can then be installed by running uv sync --extra=production - or explicitly excluded with uv sync --no-extra=production

uv docs about optional dependencies.

(And thanks for doing the work - I look forward to it being merged :) )

@foarsitter
Copy link
Collaborator Author

@foobacca I see development as a superset of production. Everything from production needs to be replicated in development but production doesn't need a test framework for example.

When the sentry_sdk is added as an extra dependency I cannot use capture_exception for example because the package is missing. Another example is testing the gunicorn command. I cannot do such thing when developing when I do not have the package.

@foobacca
Copy link

@foarsitter I see your point (though you can of course install the production dependencies if you want with uv sync --extra=production ).

I don't have a strong preference, but it does seem like a change from the master branch, where there is a separate requirements/production.txt file, suggesting that the maintainers liked to keep the production dependencies separate. So I wanted to make sure that change was explicit and intended.

I'll leave it to the maintainers. If they're happy with the change I have no objection. (And I'm quite capable of changing my project after running cookiecutter anyway 🙂 )

@foobacca
Copy link

One other idea. I would expect the [project] and [dependency-groups] section to be the first and second sections in pyproject.toml - putting the overall project definition and dependencies at the top where it's easy to find them.

I can obviously edit it myself after running cookiecutter, so not a big deal if others prefer it at the bottom, but thought I'd mention it.

@browniebroke
Copy link
Member

I don't have a strong preference, but it does seem like a change from the master branch, where there is a separate requirements/production.txt file, suggesting that the maintainers liked to keep the production dependencies separate. So I wanted to make sure that change was explicit and intended.

The production deps are installed locally on the master branch too, so not so much of change - see #4838 for more context

@pievalentin
Copy link

One other idea. I would expect the [project] and [dependency-groups] section to be the first and second sections in pyproject.toml - putting the overall project definition and dependencies at the top where it's easy to find them.

I can obviously edit it myself after running cookiecutter, so not a big deal if others prefer it at the bottom, but thought I'd mention it.

I agree. Dependencies should be at the top of the file

@foarsitter
Copy link
Collaborator Author

We use https://github.com/tox-dev/pyproject-fmt for dictating the order of appearance, so you don't need to worry about it :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
9 participants