Analysing IPython code - magics cause syntax error

[psychemedia]

I'm trying to run radon metrics over code in various Jupyter notebooks and if there are cells that include IPython block magic (notebook code source cell starts %%, for example %%capture) or includes some line magic (line starts with %, for example, %pip install pandas) then it breaks calls to radon.complexity.cc_visit and radon.metrics.h_visit with errors of the form:

SyntaxError: invalid syntax (<unknown>, line 3) 
Traceback (most recent call last):

  File "/usr/local/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3319, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File "<ipython-input-115-472082b951d2>", line 4, in <module>
    cc_visit(c)

  File "/usr/local/lib/python3.7/site-packages/radon/complexity.py", line 100, in cc_visit
    return cc_visit_ast(code2ast(code), **kwargs)

  File "/usr/local/lib/python3.7/site-packages/radon/visitors.py", line 32, in code2ast
    return ast.parse(source)

  File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 35, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)

  File "<unknown>", line 3
    %pip install pandas
    ^
SyntaxError: invalid syntax

Calls to radon.raw.analyze work fine.

Should there be a way to specify that % and %% prefixed directives should be ignored when the code is passed to the AST parser?

[rubik]

Sorry for the delay. This is not currently supported, so it would require a change in the parsing code. At the moment I cannot do it, but I will review any submitted PR. If implemented, it should be special behavior triggered by a command-line option, because it would require additional parsing.

[psychemedia]

I started wondering about this again, in the context of how the IPython interpreter works. It looks, for recent versions of IPython at least, as if code lines invoking magics etc may get converted to equivalent lines of Python code? eg using functions from Python.core.inputtransformer2. (It also seems that the code may have changed recently, which could make things complex if this packages needs to be able to parse various versions of IPython.)

This raises an interesting question about the code count measures: is a line of (IPython mediated) shell or magic a single line of code (as any other code) or a single line of something else (given you may need a mental context shift when considering it).

In a block cell magic, where the first line identifies the block magic, is that a line of code?

If a cell block magic sets up a code cell as a SQL block, taking lines of SQL code, are those each a single line of code, or something else?

More generally, is the notion of a sloc that it is single line of any code? eg how do sloc measures work in a polyglot code environment?

[rubik]

@psychemedia That's interesting. I would say they are lines of code. Maybe not Python code, but code nonetheless. Determining if they contain real Python code might be tricky though. How all that would affect Radon, though, is not so clear. It looks like a highly subjective matter.

[psychemedia]

They're more like macros. So python is called but via a command-line like function invocation.

IPython also lets you make shell calls prefixed with a !.