Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

audit: check for Python-wide site-package usage #16663

Closed
wants to merge 1 commit into from

Conversation

iMichka
Copy link
Member

@iMichka iMichka commented Feb 15, 2024

See #16662

We would like to enforce vendoring for Python libraries, or the usage of a virtualenv in the formula's libexec directory, using a virtualenv.

  • Have you followed the guidelines in our Contributing document?
  • Have you checked to ensure there aren't other open Pull Requests for the same change?
  • Have you added an explanation of what your changes do and why you'd like us to include them?
  • Have you written new tests for your changes? Here's an example.
  • Have you successfully run brew style with your changes locally?
  • Have you successfully run brew typecheck with your changes locally?
  • Have you successfully run brew tests with your changes locally?

@iMichka
Copy link
Member Author

iMichka commented Feb 15, 2024

This needs an allowlist and further testing. It's just an early draft to see if this audit would make sense to enforce our vendoring policy.

Note: I am aware that the existing check_easy_install_pth check looks fishy nowadays, and will handle that in a separate pull request.

@Bo98
Copy link
Member

Bo98 commented Feb 15, 2024

Seems like it would catch a ton of things that happen to ship Python bindings which we have always allowed, unless we're planning on changing that?

Limiting this to pythonhosted.org URLs would however could make sense. I'd probably limit this to Homebrew/core only too as this doesn't make sense to restrict for third-party taps.

@MikeMcQuaid
Copy link
Member

Python bindings which we have always allowed, unless we're planning on changing that?

Don't think we should change that 👍🏻

Could maybe do something like always allow that if there's also something anything lib?

@iMichka
Copy link
Member Author

iMichka commented Feb 15, 2024

Limiting this to pythonhosted.org URLs would however could make sense. I'd probably limit this to Homebrew/core only too as this doesn't make sense to restrict for third-party taps.

Done

@iMichka
Copy link
Member Author

iMichka commented Feb 15, 2024

Python bindings which we have always allowed, unless we're planning on changing that?

Don't think we should change that 👍🏻

Could maybe do something like always allow that if there's also something anything lib?

I made an attempt to parse for .so files (will need to adapt for .dylib on Linux too).
This makes numpy pass, as it makes also python-cryptography pass (it contains a .so file). Basically this means we allow any .so/.dylib file we compiled from source.

The only remaining exception I am aware of is certifi, that is not correctly handled by my audit.
The check needs some testing on more formulae to be sure we cover all the cases (tests would be good too but might be hard to implement).

@iMichka
Copy link
Member Author

iMichka commented Feb 18, 2024

I opened another (unrelated) PR for the check_easy_install_pth check:
#16698

See Homebrew#16662

We would like to enforce vendoring for Python libraries,
or the usage of a virtualenv in the formula's libexec directory,
using a virtualenv.
@iMichka
Copy link
Member Author

iMichka commented Feb 19, 2024

Made some tests with the current implementation.

These are ok:

numpy -> Does not fail audit, due to Python bindings
scipy ->  Does not fail audit, due to Python bindings
python-matplotlib ->  Does not fail audit, due to Python bindings
python-cryptography ->  Does not fail audit, due to Python bindings
six -> Fails audit
pygments -> Fails audit
python-packaging -> Fails audit
python-pytz -> Fails audit
python-urllib3 -> Fails audit
python-tabulate -> Fails audit

These are not:

python-certifi -> Fails audit, but we want to make it pass
poetry ->  Does not fail audit, but should?
python-markupsafe ->  Does not fail audit (but should fail? Contains _speedups.cpython-312-darwin.so?)

@@ -143,6 +143,25 @@
EOS
end

def check_global_site_package_usage(formula)
return unless formula.tap.core_tap?
return unless formula.stable.url.start_with?("https://files.pythonhosted.org")

Check failure

Code scanning / CodeQL

Incomplete URL substring sanitization High

'
https://files.pythonhosted.org
' may be followed by an arbitrary host name.
@Bo98
Copy link
Member

Bo98 commented Feb 19, 2024

poetry -> Does not fail audit, but should?

I don't think so - it's a CLI tool that is installed under a virtualenv rather than directly to site-packages.

I do realise some dependents try to bypass that though and perhaps shouldn't.

Copy link

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

@github-actions github-actions bot added the stale No recent activity label Mar 12, 2024
@github-actions github-actions bot closed this Mar 19, 2024
@github-actions github-actions bot added the outdated PR was locked due to age label Apr 18, 2024
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Apr 18, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
outdated PR was locked due to age stale No recent activity
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants