"python" command to invoke modules directly does not work with "plugins" directory structure

[andy-maier]

Summary

The Ansible documentation at https://docs.ansible.com/ansible/latest/dev_guide/developing_modules_general.html#verifying-your-module-code-locally shows this:

---

• If for any reason (pdb, using print(), faster iteration, etc) you want to avoid going through Ansible, another way is to create an arguments file, a basic JSON config file that passes parameters to your module so that you can run it. Name the arguments file /tmp/args.json and add the following content:

  {
      "ANSIBLE_MODULE_ARGS": {
          "name": "hello",
          "new": true
      }
  }
  

• Then the module can be tested locally and directly. This skips the packing steps and uses module_utils files directly:

  python library/my_test.py /tmp/args.json
  

---

This approach does not work when the collection uses the recommended plugins subdirectory structure and the module imports from the module_utils directory:

File plugins/module_utils/utils.py:

def func1():
    . . .

File plugins/modules/mod1.py:

from ..module_utils.utils import func1
. . .

The approach shown in the documentation fails for such a module with:

Traceback (most recent call last):
  File "..../plugins/modules/mod1.py", line NNN, in <module>
    from ..module_utils.common import func1 \
ImportError: attempted relative import with no known parent package

Such a module passes the Ansible sanity test, so it must be valid.

By the way, the documented approach of using ansible -m ... ("ad-hoc" command) also does not work with the plugins directory structure.

Issue Type

Bug Report

Component Name

documentation

Ansible Version

$ ansible --version
ansible [core 2.18.1]
  config file = None
  configured module search path = ['/Users/maiera/Projects/zhmcclient/repos/zhmc-ansible-modules/zhmc_ansible_modules']
  ansible python module location = /Users/maiera/virtualenvs/zhmcamtest312/lib/python3.12/site-packages/ansible
  ansible collection location = /Users/maiera/.ansible/collections:/usr/share/ansible/collections
  executable location = /Users/maiera/virtualenvs/zhmcamtest312/bin/ansible
  python version = 3.12.7 (main, Oct  1 2024, 02:05:46) [Clang 15.0.0 (clang-1500.3.9.4)] (/Users/maiera/virtualenvs/zhmcamtest312/bin/python)
  jinja version = 3.1.5
  libyaml = True

Configuration

# if using a version older than ansible-core 2.12 you should omit the '-t all'
$ ansible-config dump --only-changed -t all
CONFIG_FILE() = None
DEFAULT_MODULE_PATH(env: ANSIBLE_LIBRARY) = ['/Users/maiera/Projects/zhmcclient/repos/zhmc-ansible-modules/zhmc_ansible_modules']

GALAXY_SERVERS:

OS / Environment

macOS 14.2.7
Python 3.12.7

Steps to Reproduce

The issue can be demonstrated with the ibm.ibm_zhmc collection:

$ git clone https://github.com/zhmcclient/zhmc-ansible-modules.git
$ cd zhmc-ansible-modules
$ mkvirtualenv test312 -p python3.12
$ pip install -r requirements.txt
$ python plugins/modules/zhmc_cpc_list.py 
Traceback (most recent call last):
  File "..../plugins/modules/zhmc_cpc_list.py", line 223, in <module>
    from ..module_utils.common import log_init, open_session, close_session, \
ImportError: attempted relative import with no known parent package

Expected Results

The documentation should show an approach for locally invoking modules that works with the recommended plugins directory structure.

Actual Results

See above

Code of Conduct

• I agree to follow the Ansible Code of Conduct

[ansibot]

Files identified in the description:

None

If these files are incorrect, please update the component name section of the description or use the component bot command.

[ansible-documentation-bot]

Thanks for your Ansible docs contribution! We talk about Ansible documentation on Matrix at #docs:ansible.im if you ever want to join us and chat about the docs! We meet on Matrix every Tuesday. See the Ansible calendar for meeting details. We welcome additions to our weekly agenda items too. You can add the dawgs-meeting tag to a forum topic to bring it up at the next meeting.

[samccann]

@sivel @felixfontein - we chatted abit about this issue in the DaWGs meeting and just wanted your nickel that this is a docs problem and not some underlying possible code issue?

[sivel]

Not a code issue.

We probably need more explicit info about the following if there is any hope to address it:

By the way, the documented approach of using ansible -m ... ("ad-hoc" command) also does not work with the plugins directory structure.

I'm assuming this is due to not using the --playbook arg with the ansible command with a collection in the current working directory, since there is no playbook for relative paths to operate off of, it must be specified with --playbook or an adjustment to the ANSIBLE_COLLECTIONS_PATH env var.

fwiw, I've always been against the recommendation of directly invoking modules with python. While it's certainly possible, core does a number of things to set up the environment, such as loading the python collection loader, which is unavailable just running python plugins/modules/foo.py. It can be done with collections, but would likely require modification to the PYTHONPATH env var, and running with python -m instead of a path to the module.

My 2c is instructions for python should only be documented for debugging modules that have gone through the ansiballz packing.

[sivel]

Example with the following cwd structure:

.
└── collections
    └── ansible_collections
        └── foo
            └── bar
                └── plugins
                    └── modules
                        └── baz.py

$ ansible localhost -m foo.bar.baz --playbook-dir=$PWD
localhost | SUCCESS => {
    "baz": "qux",
    "changed": false
}

$ PYTHONPATH=./collections/ python -m ansible_collections.foo.bar.plugins.modules.baz ./args.json

{"baz": "qux", "invocation": {"module_args": {}}}

I should note that the original instructions in that doc are more geared towards standalone modules, or contributing a module to core, as opposed to collections. The steps I provided above are collection specific.