[change] Move python dependencies to requirements.txt file #511

shwetd19 · 2025-01-30T13:01:27Z

Checklist

I have read the OpenWISP Contributing Guidelines.
I have manually tested the changes proposed in this pull request.
I have written new test cases for new code and/or updated existing tests for changes to existing code.
I have updated the documentation.

Reference to Existing Issue

Closes #507

Description of Changes

This PR here implements automated dependency management using Dependabot by extracting Python dependencies from pip.yml into a dedicated requirements.txt file.

Key Changes:

New Requirements File
- Created files/requirements.txt containing all Python dependencies
- Organized dependencies by category with clear comments
- Maintained version constraints where necessary
Updated Package Installation
- Modified tasks/pip.yml to use the new requirements file
- Implemented version constraints using pip's constraint feature
- Preserved compatibility with existing Ansible variables
- Maintained support for conditional package installation
Dependabot Configuration
- Added Python package ecosystem configuration
- Set weekly update schedule
- Configured PR labeling and commit message format

Testing Details:

Installation Testing

Tested with various configurations:
```
molecule test -s local
```
Integration Testing
- Validated Dependabot configuration
- Tested requirements file parsing
- Verified dependency resolution

Benefits:

Automated dependency updates via Dependabot
Improved dependency management
Maintained backward compatibility
Better organized package requirements

Screenshots

- Modified pip.yml to install dependencies from requirements.txt - Used role_path variable to reference requirements.txt in files directory - Implemented pip constraints to maintain version control from Ansible variables - Kept an extra_python_packages handling for custom additions - and preserved existing retry logic and environment settings

- Here grouped dependencies by their functionality (core, database, optional) - Added few comments to indicate optional packages and their purpose - While maintaining existing version constraints for critical packages - I've ensured compatibility with current OpenWISP installation process

- Added pip package ecosystem to dependabot configuration - While setting up weekly update schedule for dependency checks - Added all 'dependencies' and 'python' labels for better PR organization

nemesifier · 2025-01-31T19:14:01Z

.github/dependabot.yml

-      interval: "monthly" # Check for updates weekly
-    commit-message:
-      prefix: "[ci] "
+      prefix: "deps"


Suggested change

prefix: "deps"

prefix: "[deps]"

nemesifier · 2025-01-31T19:14:18Z

.github/dependabot.yml

    commit-message:
-      prefix: "[deps] "
-  - package-ecosystem: "github-actions" # Check for GitHub Actions updates


Why are you removing github-actions? Please don't.

I've made the fix for this in my recent commit

- Maintained consistent prefix formatting using square brackets: [deps] for pip and [ci] for GitHub Actions - Restored the GitHub Actions ecosystem configuration - Setup both update intervals to "weekly" as originally intended

nemesifier

@shwetd19 the build is failing. Please execute the tests locally before pushing.

shwetd19 · 2025-02-07T04:14:53Z

Hey @nemesifier ,

The main issue was that the requirements.txt file cannot be found during the installation process. The error message was consistent across different Ubuntu and Debian environments I've made the possible fix and did tested it properly this time

can you please review my PR

shwetd19 · 2025-02-10T09:42:48Z

Hey @pandafy @nemesifier can you please reivew this PR ?
pls let me know if any changes are required, Thanks!

pandafy

Let's only extract the Python dependencies which are pinned to a version. Take a note to respect the when condition of the existing tasks that installs Python modules. The role does not install all Python packages everytime.

pandafy · 2025-02-10T10:15:29Z

.github/dependabot.yml

I don't think we need any changes for this file.

fixed dependabot as it was previously in my latest commit

pandafy · 2025-02-11T09:59:44Z

files/requirements.txt

+psycopg2
+MySQL-python


These packages are installed only when specific database backends are used. Since, we are installing the latest version of these packages, adding them here is not fruitful.

modified requirements.txt file for removing Database adapters

pandafy · 2025-02-11T10:00:43Z

tasks/pip.yml

-    state: latest
+    state: present


Why did you change this? latest allows to upgrade the Python modules when using compatibility versioning. Same for other instances.

modified pip.yml for database adapters psycopg2 & MySQL-python and to fix the state as latest

shwetd19 · 2025-02-11T14:02:33Z

Hello @pandafy I've implemented the suggested changes in my recent commits, can you please check that once, Thanks!

pandafy · 2025-02-14T10:17:04Z

tasks/pip.yml

- name: Install raven (sentry client)
-  when: openwisp2_sentry.get('dsn')
-  pip:
-    name: raven
-    state: latest
-    virtualenv: "{{ virtualenv_path }}"
-    virtualenv_python: "{{ openwisp2_python }}"
-  retries: 5
-  delay: 10
+  retries: 3
+  delay: 5
  register: result
  until: result is success
  notify: Reload application


raven is only installed when openwisp2_sentry.get('dsn'), let's remove it from requirements.txt

pandafy · 2025-02-14T10:17:29Z

files/requirements.txt

+django-cors-headers~=4.4.0
+channels_redis~=4.2.0
+
+


Remove this extra blank line

pandafy · 2025-02-14T10:18:20Z

tasks/pip.yml

    state: latest
    virtualenv: "{{ virtualenv_path }}"
    virtualenv_python: "{{ openwisp2_python }}"
-  when: openwisp2_email_backend == "djcelery_email.backends.CeleryEmailBackend"


As I suggested in the previous review, don't add packages to requirements.txt which are installed conditionally.

we could install it anyway, it shouldn't be a big deal, what do you think?

pandafy · 2025-02-14T10:20:49Z

files/requirements.txt

+# Remove deprecated packages
+jsonfield2==*; python_version < "0.0"  # Ensures this is never installed


Will this remove the package if it was previously installed?

@nemesifier can we remove the task which ensures that this package is removed?

cc : @nemesifier

shwetd19 · 2025-02-22T11:17:54Z

Hey @pandafy @nemesifier

I've made the suggested changes as for ravan & now it usees the correct prefix format [deps]

devkapilbansal · 2025-02-23T19:08:54Z

tasks/pip.yml


- name: Install openwisp2 controller and its dependencies
+- name: Verify requirements.txt exists


Why do we need to verify that requirements.txt exists?

I've removed this part in my latest commit

devkapilbansal · 2025-02-23T19:11:24Z

tasks/pip.yml

- name: Install openwisp2 controller and its dependencies
+- name: Verify requirements.txt exists
+  stat:
+    path: "{{ role_path }}/files/requirements.txt"


Where is this var role_path defined?

devkapilbansal · 2025-02-23T19:14:01Z

tasks/pip.yml

  pip:
-    name:
-      - django-pipeline~=3.1.0
+    name: "{{ openwisp2_django_version }}"


Not sure but can we move Django also in requirements.txt file?
cc @nemesifier

Not for now, because that's configurable.

devkapilbansal · 2025-02-23T19:19:24Z

tasks/pip.yml

    state: latest
    virtualenv: "{{ virtualenv_path }}"
    virtualenv_python: "{{ openwisp2_python }}"
-  retries: 5
-  delay: 10
+  retries: 3


Is there a reason for changing retries and delays?

I think this is wrong.

I've fixed this part in my latest commit to default retries as 5 for all

nemesifier

Sorry to see this PR has stalled. Can we make the CI build pass @shwetd19? Tried responding to the questions, let me know if you have more doubts.

nemesifier · 2025-03-21T22:21:41Z

files/requirements.txt

+# Remove deprecated packages
+jsonfield2==*; python_version < "0.0"  # Ensures this is never installed


nemesifier · 2025-03-21T22:22:28Z

tasks/pip.yml

    state: latest
    virtualenv: "{{ virtualenv_path }}"
    virtualenv_python: "{{ openwisp2_python }}"
-  when: openwisp2_email_backend == "djcelery_email.backends.CeleryEmailBackend"


we could install it anyway, it shouldn't be a big deal, what do you think?

shwetd19 added 3 commits January 30, 2025 18:21

[change] Configured dependabot for Python dependencies

391ea20

- Added pip package ecosystem to dependabot configuration - While setting up weekly update schedule for dependency checks - Added all 'dependencies' and 'python' labels for better PR organization

nemesifier requested changes Jan 31, 2025

View reviewed changes

[change] Configured dependabot properly

fa90ca1

- Maintained consistent prefix formatting using square brackets: [deps] for pip and [ci] for GitHub Actions - Restored the GitHub Actions ecosystem configuration - Setup both update intervals to "weekly" as originally intended

shwetd19 requested a review from nemesifier February 4, 2025 05:41

nemesifier requested changes Feb 4, 2025

View reviewed changes

shwetd19 added 3 commits February 6, 2025 00:15

[change] Modified pip.yml to use requirements.txt file

07a2c84

[change] The role_path is looking for requirements.txt is fixed

7390bf0

[change] Modified requirements.txt for Python dependencies

ba3aea3

shwetd19 requested a review from nemesifier February 7, 2025 04:15

pandafy requested changes Feb 11, 2025

View reviewed changes

shwetd19 added 4 commits February 11, 2025 19:20

[change] fixed dependabot as it was previously

8df5413

[change] added a requirements.txt file for removing Database adapters

9c99aa1

[change] modified pip.yml to fix the state as latest

534eb1a

[change] modified pip.yml for database adapters psycopg2 & MySQL-python

3d8c5ee

shwetd19 requested a review from pandafy February 11, 2025 14:01

pandafy requested changes Feb 14, 2025

View reviewed changes

shwetd19 added 2 commits February 22, 2025 16:45

[change] Fixed requirements.txt file for ravan

35f55de

[change] Modified pip.yml to fix the raven installation

936e303

devkapilbansal reviewed Feb 23, 2025

View reviewed changes

shwetd19 requested review from devkapilbansal and pandafy March 3, 2025 07:58

Merge branch 'master' into 507-Use-dependabot-to-upgrade-dependencies

be96642

nemesifier requested changes Mar 21, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[change] Move python dependencies to requirements.txt file #511

[change] Move python dependencies to requirements.txt file #511

shwetd19 commented Jan 30, 2025

nemesifier Jan 31, 2025

nemesifier Jan 31, 2025

shwetd19 Feb 4, 2025

nemesifier left a comment

shwetd19 commented Feb 7, 2025

shwetd19 commented Feb 10, 2025

pandafy left a comment

pandafy Feb 10, 2025

shwetd19 Feb 11, 2025

pandafy Feb 11, 2025

shwetd19 Feb 11, 2025

pandafy Feb 11, 2025

shwetd19 Feb 11, 2025 •

edited

Loading

shwetd19 commented Feb 11, 2025

pandafy Feb 14, 2025

pandafy Feb 14, 2025

pandafy Feb 14, 2025

nemesifier Mar 21, 2025

pandafy Feb 14, 2025

shwetd19 Feb 22, 2025

nemesifier Mar 21, 2025

shwetd19 commented Feb 22, 2025

devkapilbansal Feb 23, 2025

shwetd19 Mar 3, 2025

devkapilbansal Feb 23, 2025

devkapilbansal Feb 23, 2025

nemesifier Feb 24, 2025

devkapilbansal Feb 23, 2025

nemesifier Feb 24, 2025

shwetd19 Mar 3, 2025

nemesifier left a comment

nemesifier Mar 21, 2025

nemesifier Mar 21, 2025

		# Remove deprecated packages
		jsonfield2==*; python_version < "0.0" # Ensures this is never installed


		- name: Install openwisp2 controller and its dependencies
		- name: Verify requirements.txt exists

		django-cors-headers~=4.4.0
		channels_redis~=4.2.0

[change] Move python dependencies to requirements.txt file #511

Are you sure you want to change the base?

[change] Move python dependencies to requirements.txt file #511

Conversation

shwetd19 commented Jan 30, 2025

Checklist

Reference to Existing Issue

Description of Changes

Key Changes:

Testing Details:

Tested with various configurations:

Benefits:

Screenshots

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nemesifier left a comment

Choose a reason for hiding this comment

shwetd19 commented Feb 7, 2025

shwetd19 commented Feb 10, 2025

pandafy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shwetd19 Feb 11, 2025 • edited Loading

Choose a reason for hiding this comment

shwetd19 commented Feb 11, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shwetd19 commented Feb 22, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nemesifier left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shwetd19 Feb 11, 2025 •

edited

Loading