Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"python" command to invoke modules directly does not work with "plugins" directory structure #2374

Open
1 task done
andy-maier opened this issue Jan 27, 2025 · 5 comments · May be fixed by #2427
Open
1 task done
Labels
new_contributor This PR is the first contribution by a new community member. techreview needs technical review

Comments

@andy-maier
Copy link
Contributor

andy-maier commented Jan 27, 2025

Summary

The Ansible documentation at https://docs.ansible.com/ansible/latest/dev_guide/developing_modules_general.html#verifying-your-module-code-locally shows this:


  • If for any reason (pdb, using print(), faster iteration, etc) you want to avoid going through Ansible, another way is to create an arguments file, a basic JSON config file that passes parameters to your module so that you can run it. Name the arguments file /tmp/args.json and add the following content:
    {
        "ANSIBLE_MODULE_ARGS": {
            "name": "hello",
            "new": true
        }
    }
    
  • Then the module can be tested locally and directly. This skips the packing steps and uses module_utils files directly:
    python library/my_test.py /tmp/args.json
    

This approach does not work when the collection uses the recommended plugins subdirectory structure and the module imports from the module_utils directory:

File plugins/module_utils/utils.py:

def func1():
    . . .

File plugins/modules/mod1.py:

from ..module_utils.utils import func1
. . .

The approach shown in the documentation fails for such a module with:

Traceback (most recent call last):
  File "..../plugins/modules/mod1.py", line NNN, in <module>
    from ..module_utils.common import func1 \
ImportError: attempted relative import with no known parent package

Such a module passes the Ansible sanity test, so it must be valid.

By the way, the documented approach of using ansible -m ... ("ad-hoc" command) also does not work with the plugins directory structure.

Issue Type

Bug Report

Component Name

documentation

Ansible Version

$ ansible --version
ansible [core 2.18.1]
  config file = None
  configured module search path = ['/Users/maiera/Projects/zhmcclient/repos/zhmc-ansible-modules/zhmc_ansible_modules']
  ansible python module location = /Users/maiera/virtualenvs/zhmcamtest312/lib/python3.12/site-packages/ansible
  ansible collection location = /Users/maiera/.ansible/collections:/usr/share/ansible/collections
  executable location = /Users/maiera/virtualenvs/zhmcamtest312/bin/ansible
  python version = 3.12.7 (main, Oct  1 2024, 02:05:46) [Clang 15.0.0 (clang-1500.3.9.4)] (/Users/maiera/virtualenvs/zhmcamtest312/bin/python)
  jinja version = 3.1.5
  libyaml = True

Configuration

# if using a version older than ansible-core 2.12 you should omit the '-t all'
$ ansible-config dump --only-changed -t all
CONFIG_FILE() = None
DEFAULT_MODULE_PATH(env: ANSIBLE_LIBRARY) = ['/Users/maiera/Projects/zhmcclient/repos/zhmc-ansible-modules/zhmc_ansible_modules']

GALAXY_SERVERS:

OS / Environment

macOS 14.2.7
Python 3.12.7

Steps to Reproduce

The issue can be demonstrated with the ibm.ibm_zhmc collection:

$ git clone https://github.com/zhmcclient/zhmc-ansible-modules.git
$ cd zhmc-ansible-modules
$ mkvirtualenv test312 -p python3.12
$ pip install -r requirements.txt
$ python plugins/modules/zhmc_cpc_list.py 
Traceback (most recent call last):
  File "..../plugins/modules/zhmc_cpc_list.py", line 223, in <module>
    from ..module_utils.common import log_init, open_session, close_session, \
ImportError: attempted relative import with no known parent package

Expected Results

The documentation should show an approach for locally invoking modules that works with the recommended plugins directory structure.

Actual Results

See above

Code of Conduct

  • I agree to follow the Ansible Code of Conduct
@ansibot ansibot added bug Something isn't working needs_triage Needs a first human triage before being processed. labels Jan 27, 2025
@ansibot
Copy link
Contributor

ansibot commented Jan 27, 2025

Files identified in the description:

None

If these files are incorrect, please update the component name section of the description or use the component bot command.

@sivel sivel transferred this issue from ansible/ansible Jan 28, 2025
@ansible-documentation-bot ansible-documentation-bot bot added the new_contributor This PR is the first contribution by a new community member. label Jan 28, 2025
@ansible-documentation-bot
Copy link
Contributor

Thanks for your Ansible docs contribution! We talk about Ansible documentation on Matrix at #docs:ansible.im if you ever want to join us and chat about the docs! We meet on Matrix every Tuesday. See the Ansible calendar for meeting details. We welcome additions to our weekly agenda items too. You can add the dawgs-meeting tag to a forum topic to bring it up at the next meeting.

@samccann samccann added the techreview needs technical review label Feb 25, 2025
@samccann
Copy link
Contributor

@sivel @felixfontein - we chatted abit about this issue in the DaWGs meeting and just wanted your nickel that this is a docs problem and not some underlying possible code issue?

@sivel
Copy link
Member

sivel commented Feb 25, 2025

Not a code issue.

We probably need more explicit info about the following if there is any hope to address it:

By the way, the documented approach of using ansible -m ... ("ad-hoc" command) also does not work with the plugins directory structure.

I'm assuming this is due to not using the --playbook arg with the ansible command with a collection in the current working directory, since there is no playbook for relative paths to operate off of, it must be specified with --playbook or an adjustment to the ANSIBLE_COLLECTIONS_PATH env var.

fwiw, I've always been against the recommendation of directly invoking modules with python. While it's certainly possible, core does a number of things to set up the environment, such as loading the python collection loader, which is unavailable just running python plugins/modules/foo.py. It can be done with collections, but would likely require modification to the PYTHONPATH env var, and running with python -m instead of a path to the module.

My 2c is instructions for python should only be documented for debugging modules that have gone through the ansiballz packing.

@sivel
Copy link
Member

sivel commented Feb 25, 2025

Example with the following cwd structure:

.
└── collections
    └── ansible_collections
        └── foo
            └── bar
                └── plugins
                    └── modules
                        └── baz.py
$ ansible localhost -m foo.bar.baz --playbook-dir=$PWD
localhost | SUCCESS => {
    "baz": "qux",
    "changed": false
}
$ PYTHONPATH=./collections/ python -m ansible_collections.foo.bar.plugins.modules.baz ./args.json

{"baz": "qux", "invocation": {"module_args": {}}}

I should note that the original instructions in that doc are more geared towards standalone modules, or contributing a module to core, as opposed to collections. The steps I provided above are collection specific.

Akasurde added a commit to Akasurde/ansible-documentation that referenced this issue Feb 25, 2025
* Updated developer guide for module development to work with
  collection and PYTHONPATH

Fixes: ansible#2374

Signed-off-by: Abhijeet Kasurde <[email protected]>
@Akasurde Akasurde linked a pull request Feb 25, 2025 that will close this issue
@samccann samccann removed bug Something isn't working needs_triage Needs a first human triage before being processed. labels Mar 4, 2025
Akasurde added a commit to Akasurde/ansible-documentation that referenced this issue Mar 11, 2025
* Updated developer guide for module development to work with
  collection and PYTHONPATH

Fixes: ansible#2374

Signed-off-by: Abhijeet Kasurde <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
new_contributor This PR is the first contribution by a new community member. techreview needs technical review
Projects
Status: 🆕 Triage
Development

Successfully merging a pull request may close this issue.

4 participants