diff --git a/DOCS.md b/DOCS.md index ebacc5ea4..34cd5eac3 100644 --- a/DOCS.md +++ b/DOCS.md @@ -108,6 +108,8 @@ pip install coconut-develop ``` which will install the most recent working version from Coconut's [`develop` branch](https://github.com/evhub/coconut/tree/develop). Optional dependency installation is supported in the same manner as above. For more information on the current development build, check out the [development version of this documentation](http://coconut.readthedocs.io/en/develop/DOCS.html). Be warned: `coconut-develop` is likely to be unstable—if you find a bug, please report it by [creating a new issue](https://github.com/evhub/coconut/issues/new). +_Note: if you have an existing release version of `coconut` installed, you'll need to `pip uninstall coconut` before installing `coconut-develop`._ + ## Compilation ```{contents} @@ -142,7 +144,7 @@ dest destination directory for compiled files (defaults to ``` -h, --help show this help message and exit --and source [dest ...] - add an additional source/dest pair to compile + add an additional source/dest pair to compile (dest is optional) -v, -V, --version print Coconut and Python version information -t version, --target version specify target Python version (defaults to universal) @@ -726,6 +728,8 @@ The `..` operator has lower precedence than `::` but higher precedence than infi All function composition operators also have in-place versions (e.g. `..=`). +Since all forms of function composition always call the first function in the composition (`f` in `f ..> g` and `g` in `f <.. g`) with exactly the arguments passed into the composition, all forms of function composition will preserve all metadata attached to the first function in the composition, including the function's [signature](https://docs.python.org/3/library/inspect.html#inspect.signature) and any of that function's attributes. + ##### Example **Coconut:** @@ -747,7 +751,7 @@ Coconut uses a `$` sign right after an iterator before a slice to perform iterat Iterator slicing works just like sequence slicing, including support for negative indices and slices, and support for `slice` objects in the same way as can be done with normal slicing. Iterator slicing makes no guarantee, however, that the original iterator passed to it be preserved (to preserve the iterator, use Coconut's [`reiterable`](#reiterable) built-in). -Coconut's iterator slicing is very similar to Python's `itertools.islice`, but unlike `itertools.islice`, Coconut's iterator slicing supports negative indices, and will preferentially call an object's `__iter_getitem__` (always used if available) or `__getitem__` (only used if the object is a collections.abc.Sequence). Coconut's iterator slicing is also optimized to work well with all of Coconut's built-in objects, only computing the elements of each that are actually necessary to extract the desired slice. +Coconut's iterator slicing is very similar to Python's `itertools.islice`, but unlike `itertools.islice`, Coconut's iterator slicing supports negative indices, and will preferentially call an object's `__iter_getitem__` (always used if available) or `__getitem__` (only used if the object is a `collections.abc.Sequence`). Coconut's iterator slicing is also optimized to work well with all of Coconut's built-in objects, only computing the elements of each that are actually necessary to extract the desired slice. ##### Example @@ -874,6 +878,8 @@ Custom operators will often need to be surrounded by whitespace (or parentheses If a custom operator that is also a valid name is desired, you can use a backslash before the name to get back the name instead using Coconut's [keyword/variable disambiguation syntax](#handling-keywordvariable-name-overlap). +_Note: redefining existing Coconut operators using custom operator definition syntax is forbidden, including Coconut's built-in [Unicode operator alternatives](#unicode-alternatives)._ + ##### Examples **Coconut:** @@ -1030,6 +1036,8 @@ class CanAddAndSub(Protocol, Generic[T, U, V]): Coconut supports Unicode alternatives to many different operator symbols. The Unicode alternatives are relatively straightforward, and chosen to reflect the look and/or meaning of the original symbol. +_Note: these are only the default, built-in unicode operators. Coconut supports [custom operator definition](#custom-operators) to define your own._ + ##### Full List ``` @@ -2057,6 +2065,41 @@ print(p1(5)) quad = 5 * x**2 + 3 * x + 1 ``` +### Keyword Argument Name Elision + +When passing in long variable names as keyword arguments of the same name, Coconut supports the syntax +``` +f(...=long_variable_name) +``` +as a shorthand for +``` +f(long_variable_name=long_variable_name) +``` + +Such syntax is also supported in [partial application](#partial-application) and [anonymous `namedtuple`s](#anonymous-namedtuples). + +##### Example + +**Coconut:** +```coconut +really_long_variable_name_1 = get_1() +really_long_variable_name_2 = get_2() +main_func( + ...=really_long_variable_name_1, + ...=really_long_variable_name_2, +) +``` + +**Python:** +```coconut_python +really_long_variable_name_1 = get_1() +really_long_variable_name_2 = get_2() +main_func( + really_long_variable_name_1=really_long_variable_name_1, + really_long_variable_name_2=really_long_variable_name_2, +) +``` + ### Anonymous Namedtuples Coconut supports anonymous [`namedtuple`](https://docs.python.org/3/library/collections.html#collections.namedtuple) literals, such that `(a=1, b=2)` can be used just as `(1, 2)`, but with added names. Anonymous `namedtuple`s are always pickleable. @@ -2067,6 +2110,8 @@ The syntax for anonymous namedtuple literals is: ``` where, if `` is given for any field, [`typing.NamedTuple`](https://docs.python.org/3/library/typing.html#typing.NamedTuple) is used instead of `collections.namedtuple`. +Anonymous `namedtuple`s also support [keyword argument name elision](#keyword-argument-name-elision). + ##### `_namedtuple_of` On Python versions `>=3.6`, `_namedtuple_of` is provided as a built-in that can mimic the behavior of anonymous namedtuple literals such that `_namedtuple_of(a=1, b=2)` is equivalent to `(a=1, b=2)`. Since `_namedtuple_of` is only available on Python 3.6 and above, however, it is generally recommended to use anonymous namedtuple literals instead, as they work on any Python version. @@ -3013,6 +3058,7 @@ The new methods provided by `multiset` on top of `collections.Counter` are: - multiset.**isdisjoint**(_other_): Return True if two multisets have a null intersection. - multiset.**\_\_xor\_\_**(_other_): Return the symmetric difference of two multisets as a new multiset. Specifically: `a ^ b = (a - b) | (b - a)` - multiset.**count**(_item_): Return the number of times an element occurs in a multiset. Equivalent to `multiset[item]`, but additionally verifies the count is non-negative. +- multiset.**\_\_fmap\_\_**(_func_): Apply a function to the contents of the multiset, preserving counts; magic method for [`fmap`](#fmap). Coconut also ensures that `multiset` supports [rich comparisons and `Counter.total()`](https://docs.python.org/3/library/collections.html#collections.Counter) on all Python versions. @@ -3158,9 +3204,9 @@ _Can't be done without a series of method definitions for each data type. See th #### `fmap` -**fmap**(_func_, _obj_, *, _starmap\_over\_mappings_=`False`) +**fmap**(_func_, _obj_) -In Haskell, `fmap(func, obj)` takes a data type `obj` and returns a new data type with `func` mapped over the contents. Coconut's `fmap` function does the exact same thing in Coconut. +In Haskell, `fmap(func, obj)` takes a data type `obj` and returns a new data type with `func` mapped over the contents. Coconut's `fmap` function does the exact same thing for Coconut's [data types](#data). `fmap` can also be used on built-ins such as `str`, `list`, `set`, and `dict` as a variant of `map` that returns back an object of the same type. The behavior of `fmap` for a given object can be overridden by defining an `__fmap__(self, func)` magic method that will be called whenever `fmap` is invoked on that object. Note that `__fmap__` implementations should always satisfy the [Functor Laws](https://wiki.haskell.org/Functor). @@ -3178,6 +3224,8 @@ async def fmap_over_async_iters(func, async_iter): ``` such that `fmap` can effectively be used as an async map. +_DEPRECATED: `fmap(func, obj, fallback_to_init=True)` will fall back to `obj.__class__(map(func, obj))` if no `fmap` implementation is available rather than raise `TypeError`._ + ##### Example **Coconut:** diff --git a/__coconut__/__init__.pyi b/__coconut__/__init__.pyi index 4a42bb999..75c660612 100644 --- a/__coconut__/__init__.pyi +++ b/__coconut__/__init__.pyi @@ -169,23 +169,29 @@ enumerate = enumerate _coconut_py_str = py_str _coconut_super = super +_coconut_enumerate = enumerate +_coconut_filter = filter +_coconut_range = range +_coconut_reversed = reversed +_coconut_zip = zip zip_longest = _coconut.zip_longest memoize = _lru_cache - - reduce = _coconut.functools.reduce takewhile = _coconut.itertools.takewhile dropwhile = _coconut.itertools.dropwhile -tee = _coconut_tee = _coconut.itertools.tee -starmap = _coconut_starmap = _coconut.itertools.starmap +tee = _coconut.itertools.tee +starmap = _coconut.itertools.starmap cartesian_product = _coconut.itertools.product -multiset = _coconut_multiset = _coconut.collections.Counter - +multiset = _coconut.collections.Counter _coconut_tee = tee _coconut_starmap = starmap +_coconut_cartesian_product = cartesian_product +_coconut_multiset = multiset + + parallel_map = concurrent_map = _coconut_map = map @@ -200,6 +206,7 @@ def scan( iterable: _t.Iterable[_U], initial: _T = ..., ) -> _t.Iterable[_T]: ... +_coconut_scan = scan class MatchError(Exception): @@ -968,6 +975,7 @@ class cycle(_t.Iterable[_T]): def __fmap__(self, func: _t.Callable[[_T], _U]) -> _t.Iterable[_U]: ... def __copy__(self) -> cycle[_T]: ... def __len__(self) -> int: ... +_coconut_cycle = cycle class groupsof(_t.Generic[_T]): @@ -981,6 +989,7 @@ class groupsof(_t.Generic[_T]): def __copy__(self) -> groupsof[_T]: ... def __len__(self) -> int: ... def __fmap__(self, func: _t.Callable[[_t.Tuple[_T, ...]], _U]) -> _t.Iterable[_U]: ... +_coconut_groupsof = groupsof class windowsof(_t.Generic[_T]): @@ -996,6 +1005,7 @@ class windowsof(_t.Generic[_T]): def __copy__(self) -> windowsof[_T]: ... def __len__(self) -> int: ... def __fmap__(self, func: _t.Callable[[_t.Tuple[_T, ...]], _U]) -> _t.Iterable[_U]: ... +_coconut_windowsof = windowsof class flatten(_t.Iterable[_T]): @@ -1228,6 +1238,7 @@ def lift(func: _t.Callable[[_T, _U], _W]) -> _coconut_lifted_2[_T, _U, _W]: ... def lift(func: _t.Callable[[_T, _U, _V], _W]) -> _coconut_lifted_3[_T, _U, _V, _W]: ... @_t.overload def lift(func: _t.Callable[..., _W]) -> _t.Callable[..., _t.Callable[..., _W]]: ... +_coconut_lift = lift def all_equal(iterable: _Iterable) -> bool: ... diff --git a/_coconut/__init__.pyi b/_coconut/__init__.pyi index e60765ee8..ed242669c 100644 --- a/_coconut/__init__.pyi +++ b/_coconut/__init__.pyi @@ -135,6 +135,7 @@ pandas_numpy_modules: _t.Any = ... jax_numpy_modules: _t.Any = ... tee_type: _t.Any = ... reiterables: _t.Any = ... +fmappables: _t.Any = ... Ellipsis = Ellipsis NotImplemented = NotImplemented diff --git a/coconut/command/cli.py b/coconut/command/cli.py index 5e9c930a1..73af5fde9 100644 --- a/coconut/command/cli.py +++ b/coconut/command/cli.py @@ -77,7 +77,7 @@ type=str, nargs="+", action="append", - help="add an additional source/dest pair to compile", + help="add an additional source/dest pair to compile (dest is optional)", ) arguments.add_argument( diff --git a/coconut/command/command.py b/coconut/command/command.py index ebeeace41..3bcd5fd7d 100644 --- a/coconut/command/command.py +++ b/coconut/command/command.py @@ -23,6 +23,7 @@ import os import time import shutil +import random from contextlib import contextmanager from subprocess import CalledProcessError @@ -68,6 +69,7 @@ error_color_code, jupyter_console_commands, default_jobs, + create_package_retries, ) from coconut.util import ( univ_open, @@ -96,6 +98,8 @@ can_parse, invert_mypy_arg, run_with_stack_size, + memoized_isdir, + memoized_isfile, ) from coconut.compiler.util import ( should_indent, @@ -293,16 +297,17 @@ def execute_args(self, args, interact=True, original_args=None): raise CoconutException("cannot compile with --no-write when using --mypy") # process all source, dest pairs - src_dest_package_triples = [ - self.process_source_dest(src, dst, args) - for src, dst in ( - [(args.source, args.dest)] - + (getattr(args, "and") or []) - ) - ] + src_dest_package_triples = [] + for and_args in [(args.source, args.dest)] + (getattr(args, "and") or []): + if len(and_args) == 1: + src, = and_args + dest = None + else: + src, dest = and_args + src_dest_package_triples.append(self.process_source_dest(src, dest, args)) # disable jobs if we know we're only compiling one file - if len(src_dest_package_triples) <= 1 and not any(package for _, _, package in src_dest_package_triples): + if len(src_dest_package_triples) <= 1 and not any(memoized_isdir(source) for source, dest, package in src_dest_package_triples): self.disable_jobs() # do compilation @@ -363,12 +368,12 @@ def process_source_dest(self, source, dest, args): processed_source = fixpath(source) # validate args - if (args.run or args.interact) and os.path.isdir(processed_source): + if (args.run or args.interact) and memoized_isdir(processed_source): if args.run: raise CoconutException("source path %r must point to file not directory when --run is enabled" % (source,)) if args.interact: raise CoconutException("source path %r must point to file not directory when --run (implied by --interact) is enabled" % (source,)) - if args.watch and os.path.isfile(processed_source): + if args.watch and memoized_isfile(processed_source): raise CoconutException("source path %r must point to directory not file when --watch is enabled" % (source,)) # determine dest @@ -389,9 +394,9 @@ def process_source_dest(self, source, dest, args): package = False else: # auto-decide package - if os.path.isfile(source): + if memoized_isfile(processed_source): package = False - elif os.path.isdir(source): + elif memoized_isdir(processed_source): package = True else: raise CoconutException("could not find source path", source) @@ -442,17 +447,17 @@ def compile_path(self, path, write=True, package=True, **kwargs): """Compile a path and returns paths to compiled files.""" if not isinstance(write, bool): write = fixpath(write) - if os.path.isfile(path): + if memoized_isfile(path): destpath = self.compile_file(path, write, package, **kwargs) return [destpath] if destpath is not None else [] - elif os.path.isdir(path): + elif memoized_isdir(path): return self.compile_folder(path, write, package, **kwargs) else: raise CoconutException("could not find source path", path) def compile_folder(self, directory, write=True, package=True, **kwargs): """Compile a directory and returns paths to compiled files.""" - if not isinstance(write, bool) and os.path.isfile(write): + if not isinstance(write, bool) and memoized_isfile(write): raise CoconutException("destination path cannot point to a file when compiling a directory") filepaths = [] for dirpath, dirnames, filenames in os.walk(directory): @@ -581,11 +586,21 @@ def get_package_level(self, codepath): return package_level return 0 - def create_package(self, dirpath): + def create_package(self, dirpath, retries_left=create_package_retries): """Set up a package directory.""" filepath = os.path.join(dirpath, "__coconut__.py") - with univ_open(filepath, "w") as opened: - writefile(opened, self.comp.getheader("__coconut__")) + try: + with univ_open(filepath, "w") as opened: + writefile(opened, self.comp.getheader("__coconut__")) + except OSError: + logger.log_exc() + if retries_left <= 0: + logger.warn("Failed to write header file at", filepath) + else: + # sleep a random amount of time from 0 to 0.1 seconds to + # stagger calls across processes + time.sleep(random.random() / 10) + self.create_package(dirpath, retries_left - 1) def submit_comp_job(self, path, callback, method, *args, **kwargs): """Submits a job on self.comp to be run in parallel.""" @@ -660,7 +675,7 @@ def running_jobs(self, exit_on_error=True): def has_hash_of(self, destpath, code, package_level): """Determine if a file has the hash of the code.""" - if destpath is not None and os.path.isfile(destpath): + if destpath is not None and memoized_isfile(destpath): with univ_open(destpath, "r") as opened: compiled = readfile(opened) hashash = gethash(compiled) @@ -969,7 +984,10 @@ def start_jupyter(self, args): # pass the kernel to the console or otherwise just launch Jupyter now that we know our kernel is available if args[0] in jupyter_console_commands: - args += ["--kernel", kernel] + if any(a.startswith("--kernel") for a in args): + logger.warn("unable to specify Coconut kernel in 'jupyter " + args[0] + "' command as --kernel was already specified in the given arguments") + else: + args += ["--kernel", kernel] run_args = jupyter + args if newly_installed_kernels: @@ -989,7 +1007,7 @@ def watch(self, src_dest_package_triples, run=False, force=False): def recompile(path, src, dest, package): path = fixpath(path) - if os.path.isfile(path) and os.path.splitext(path)[1] in code_exts: + if memoized_isfile(path) and os.path.splitext(path)[1] in code_exts: with self.handling_exceptions(): if dest is True or dest is None: writedir = dest @@ -1043,7 +1061,7 @@ def site_uninstall(self): python_lib = self.get_python_lib() pth_file = os.path.join(python_lib, os.path.basename(coconut_pth_file)) - if os.path.isfile(pth_file): + if memoized_isfile(pth_file): os.remove(pth_file) logger.show_sig("Removed %s from %s" % (os.path.basename(coconut_pth_file), python_lib)) else: diff --git a/coconut/command/util.py b/coconut/command/util.py index 8403def86..85fdaa404 100644 --- a/coconut/command/util.py +++ b/coconut/command/util.py @@ -46,6 +46,7 @@ pickleable_obj, get_encoding, get_clock_time, + memoize, ) from coconut.constants import ( WINDOWS, @@ -132,6 +133,10 @@ # ----------------------------------------------------------------------------------------------------------------------- +memoized_isdir = memoize(64)(os.path.isdir) +memoized_isfile = memoize(64)(os.path.isfile) + + def writefile(openedfile, newcontents): """Set the contents of a file.""" openedfile.seek(0) diff --git a/coconut/compiler/compiler.py b/coconut/compiler/compiler.py index 6f0ff640c..9a7ba1bd6 100644 --- a/coconut/compiler/compiler.py +++ b/coconut/compiler/compiler.py @@ -1536,6 +1536,7 @@ def ln_comment(self, ln): else: lni = ln - 1 + # line number must be at start of comment for extract_line_num_from_comment if self.line_numbers and self.keep_lines: if self.minify: comment = str(ln) + " " + self.kept_lines[lni] @@ -2333,6 +2334,8 @@ def split_function_call(self, tokens, loc): star_args.append(argstr) elif arg[0] == "**": dubstar_args.append(argstr) + elif arg[0] == "...": + kwd_args.append(arg[1] + "=" + arg[1]) else: kwd_args.append(argstr) else: @@ -3043,6 +3046,8 @@ def anon_namedtuple_handle(self, tokens): types[i] = typedef else: raise CoconutInternalException("invalid anonymous named item", tok) + if name == "...": + name = item names.append(name) items.append(item) diff --git a/coconut/compiler/grammar.py b/coconut/compiler/grammar.py index 0c830210e..5099a2de5 100644 --- a/coconut/compiler/grammar.py +++ b/coconut/compiler/grammar.py @@ -32,7 +32,6 @@ from functools import partial from coconut._pyparsing import ( - CaselessLiteral, Forward, Group, Literal, @@ -115,6 +114,7 @@ boundary, compile_regex, always_match, + caseless_literal, ) @@ -798,17 +798,17 @@ class Grammar(object): octint = combine(Word("01234567") + ZeroOrMore(underscore.suppress() + Word("01234567"))) hexint = combine(Word(hexnums) + ZeroOrMore(underscore.suppress() + Word(hexnums))) - imag_j = CaselessLiteral("j") | fixto(CaselessLiteral("i"), "j") + imag_j = caseless_literal("j") | fixto(caseless_literal("i", suppress=True), "j") basenum = combine( integer + dot + Optional(integer) | Optional(integer) + dot + integer, ) | integer - sci_e = combine(CaselessLiteral("e") + Optional(plus | neg_minus)) + sci_e = combine(caseless_literal("e") + Optional(plus | neg_minus)) numitem = ~(Literal("0") + Word(nums + "_", exact=1)) + combine(basenum + Optional(sci_e + integer)) imag_num = combine(numitem + imag_j) - bin_num = combine(CaselessLiteral("0b") + Optional(underscore.suppress()) + binint) - oct_num = combine(CaselessLiteral("0o") + Optional(underscore.suppress()) + octint) - hex_num = combine(CaselessLiteral("0x") + Optional(underscore.suppress()) + hexint) + bin_num = combine(caseless_literal("0b") + Optional(underscore.suppress()) + binint) + oct_num = combine(caseless_literal("0o") + Optional(underscore.suppress()) + octint) + hex_num = combine(caseless_literal("0x") + Optional(underscore.suppress()) + hexint) number = ( bin_num | oct_num @@ -848,10 +848,10 @@ class Grammar(object): u_string = Forward() f_string = Forward() - bit_b = CaselessLiteral("b") - raw_r = CaselessLiteral("r") - unicode_u = CaselessLiteral("u").suppress() - format_f = CaselessLiteral("f").suppress() + bit_b = caseless_literal("b") + raw_r = caseless_literal("r") + unicode_u = caseless_literal("u", suppress=True) + format_f = caseless_literal("f", suppress=True) string = combine(Optional(raw_r) + string_item) # Python 2 only supports br"..." not rb"..." @@ -1133,6 +1133,7 @@ class Grammar(object): dubstar + test | star + test | unsafe_name + default + | ellipsis_tokens + equals.suppress() + refname | namedexpr_test ) function_call_tokens = lparen.suppress() + ( @@ -1178,11 +1179,11 @@ class Grammar(object): subscriptgrouplist = itemlist(subscriptgroup, comma) anon_namedtuple = Forward() + maybe_typedef = Optional(colon.suppress() + typedef_test) anon_namedtuple_ref = tokenlist( Group( - unsafe_name - + Optional(colon.suppress() + typedef_test) - + equals.suppress() + test, + unsafe_name + maybe_typedef + equals.suppress() + test + | ellipsis_tokens + maybe_typedef + equals.suppress() + refname, ), comma, ) @@ -1235,9 +1236,9 @@ class Grammar(object): set_literal = Forward() set_letter_literal = Forward() - set_s = fixto(CaselessLiteral("s"), "s") - set_f = fixto(CaselessLiteral("f"), "f") - set_m = fixto(CaselessLiteral("m"), "m") + set_s = caseless_literal("s") + set_f = caseless_literal("f") + set_m = caseless_literal("m") set_letter = set_s | set_f | set_m setmaker = Group( (new_namedexpr_test + FollowedBy(rbrace))("test") @@ -1288,8 +1289,8 @@ class Grammar(object): Group(condense(dollar + lbrack) + subscriptgroup + rbrack.suppress()) # $[ | Group(condense(dollar + lbrack + rbrack)) # $[] | Group(condense(lbrack + rbrack)) # [] - | Group(dot + ~unsafe_name + ~lbrack + ~dot) # . | Group(questionmark) # ? + | Group(dot + ~unsafe_name + ~lbrack + ~dot) # . ) + ~questionmark partial_trailer = ( Group(fixto(dollar, "$(") + function_call) # $( diff --git a/coconut/compiler/header.py b/coconut/compiler/header.py index da436a7fa..1ba8b4188 100644 --- a/coconut/compiler/header.py +++ b/coconut/compiler/header.py @@ -579,6 +579,8 @@ def NamedTuple(name, fields): except ImportError: class YouNeedToInstallTypingExtensions{object}: __slots__ = () + def __init__(self): + raise _coconut.TypeError('Protocols cannot be instantiated') Protocol = YouNeedToInstallTypingExtensions typing.Protocol = Protocol '''.format(**format_dict), diff --git a/coconut/compiler/templates/header.py_template b/coconut/compiler/templates/header.py_template index 347eb1178..5fa1e9760 100644 --- a/coconut/compiler/templates/header.py_template +++ b/coconut/compiler/templates/header.py_template @@ -35,6 +35,7 @@ class _coconut{object}:{COMMENT.EVERYTHING_HERE_MUST_BE_COPIED_TO_STUB_FILE} jax_numpy_modules = {jax_numpy_modules} tee_type = type(itertools.tee((), 1)[0]) reiterables = abc.Sequence, abc.Mapping, abc.Set + fmappables = list, tuple, dict, set, frozenset abc.Sequence.register(collections.deque) Ellipsis, NotImplemented, NotImplementedError, Exception, AttributeError, ImportError, IndexError, KeyError, NameError, TypeError, ValueError, StopIteration, RuntimeError, all, any, bool, bytes, callable, classmethod, complex, dict, enumerate, filter, float, frozenset, getattr, hasattr, hash, id, int, isinstance, issubclass, iter, len, list, locals, globals, map, min, max, next, object, property, range, reversed, set, setattr, slice, str, sum, super, tuple, type, vars, zip, repr, print{comma_bytearray} = Ellipsis, NotImplemented, NotImplementedError, Exception, AttributeError, ImportError, IndexError, KeyError, NameError, TypeError, ValueError, StopIteration, RuntimeError, all, any, bool, bytes, callable, classmethod, complex, dict, enumerate, filter, float, frozenset, getattr, hasattr, hash, id, int, isinstance, issubclass, iter, len, list, locals, globals, map, min, max, next, object, property, range, reversed, set, setattr, slice, str, sum, {lstatic}super{rstatic}, tuple, type, vars, zip, {lstatic}repr{rstatic}, {lstatic}print{rstatic}{comma_bytearray} def _coconut_handle_cls_kwargs(**kwargs): @@ -183,7 +184,7 @@ def tee(iterable, n=2): class _coconut_has_iter(_coconut_baseclass): __slots__ = ("lock", "iter") def __new__(cls, iterable): - self = _coconut.object.__new__(cls) + self = _coconut.super(_coconut_has_iter, cls).__new__(cls) self.lock = _coconut.threading.Lock() self.iter = iterable return self @@ -200,7 +201,7 @@ class reiterable(_coconut_has_iter): def __new__(cls, iterable): if _coconut.isinstance(iterable, _coconut.reiterables): return iterable - return _coconut_has_iter.__new__(cls, iterable) + return _coconut.super({_coconut_}reiterable, cls).__new__(cls, iterable) def get_new_iter(self): """Tee the underlying iterator.""" with self.lock: @@ -330,21 +331,28 @@ def _coconut_iter_getitem(iterable, index): return () iterable = _coconut.itertools.islice(iterable, 0, n) return _coconut.tuple(iterable)[i::step] -class _coconut_base_compose(_coconut_baseclass): - __slots__ = ("func", "func_infos") +class _coconut_base_compose(_coconut_baseclass):{COMMENT.no_slots_to_allow_update_wrapper}{COMMENT.must_use_coconut_attrs_to_avoid_interacting_with_update_wrapper} def __init__(self, func, *func_infos): - self.func = func - self.func_infos = [] + try: + _coconut.functools.update_wrapper(self, func) + except _coconut.AttributeError: + pass + if _coconut.isinstance(func, _coconut_base_compose): + self._coconut_func = func._coconut_func + func_infos = func._coconut_func_infos + func_infos + else: + self._coconut_func = func + self._coconut_func_infos = [] for f, stars, none_aware in func_infos: if _coconut.isinstance(f, _coconut_base_compose): - self.func_infos.append((f.func, stars, none_aware)) - self.func_infos += f.func_infos + self._coconut_func_infos.append((f._coconut_func, stars, none_aware)) + self._coconut_func_infos += f._coconut_func_infos else: - self.func_infos.append((f, stars, none_aware)) - self.func_infos = _coconut.tuple(self.func_infos) + self._coconut_func_infos.append((f, stars, none_aware)) + self._coconut_func_infos = _coconut.tuple(self._coconut_func_infos) def __call__(self, *args, **kwargs): - arg = self.func(*args, **kwargs) - for f, stars, none_aware in self.func_infos: + arg = self._coconut_func(*args, **kwargs) + for f, stars, none_aware in self._coconut_func_infos: if none_aware and arg is None: return arg if stars == 0: @@ -357,9 +365,9 @@ class _coconut_base_compose(_coconut_baseclass): raise _coconut.RuntimeError("invalid internal stars value " + _coconut.repr(stars) + " in " + _coconut.repr(self) + " {report_this_text}") return arg def __repr__(self): - return _coconut.repr(self.func) + " " + " ".join(".." + "?"*none_aware + "*"*stars + "> " + _coconut.repr(f) for f, stars, none_aware in self.func_infos) + return _coconut.repr(self._coconut_func) + " " + " ".join(".." + "?"*none_aware + "*"*stars + "> " + _coconut.repr(f) for f, stars, none_aware in self._coconut_func_infos) def __reduce__(self): - return (self.__class__, (self.func,) + self.func_infos) + return (self.__class__, (self._coconut_func,) + self._coconut_func_infos) def __get__(self, obj, objtype=None): if obj is None: return self @@ -500,7 +508,7 @@ class scan(_coconut_has_iter): optionally starting from initial.""" __slots__ = ("func", "initial") def __new__(cls, function, iterable, initial=_coconut_sentinel): - self = _coconut_has_iter.__new__(cls, iterable) + self = _coconut.super({_coconut_}scan, cls).__new__(cls, iterable) self.func = function self.initial = initial return self @@ -531,8 +539,7 @@ class reversed(_coconut_has_iter): if _coconut.isinstance(iterable, _coconut.range): return iterable[::-1] if _coconut.getattr(iterable, "__reversed__", None) is None or _coconut.isinstance(iterable, (_coconut.list, _coconut.tuple)): - self = _coconut_has_iter.__new__(cls, iterable) - return self + return _coconut.super({_coconut_}reversed, cls).__new__(cls, iterable) return _coconut.reversed(iterable) def __repr__(self): return "reversed(%s)" % (_coconut.repr(self.iter),) @@ -573,7 +580,7 @@ class flatten(_coconut_has_iter):{COMMENT.cant_implement_len_else_list_calls_bec raise _coconut.ValueError("flatten: levels cannot be negative") if levels == 0: return iterable - self = _coconut_has_iter.__new__(cls, iterable) + self = _coconut.super({_coconut_}flatten, cls).__new__(cls, iterable) self.levels = levels self._made_reit = False return self @@ -672,7 +679,7 @@ Additionally supports Cartesian products of numpy arrays.""" for i, a in _coconut.enumerate(numpy.ix_(*iterables)): arr[..., i] = a return arr.reshape(-1, _coconut.len(iterables)) - self = _coconut.object.__new__(cls) + self = _coconut.super({_coconut_}cartesian_product, cls).__new__(cls) self.iters = iterables self.repeat = repeat return self @@ -774,7 +781,7 @@ class _coconut_base_parallel_concurrent_map(map): def get_pool_stack(cls): return cls.threadlocal_ns.__dict__.setdefault("pool_stack", [None]) def __new__(cls, function, *iterables, **kwargs): - self = {_coconut_}map.__new__(cls, function, *iterables) + self = _coconut.super(_coconut_base_parallel_concurrent_map, cls).__new__(cls, function, *iterables) self.result = None self.chunksize = kwargs.pop("chunksize", 1) self.strict = kwargs.pop("strict", False) @@ -869,7 +876,7 @@ class zip_longest(zip): __slots__ = ("fillvalue",) __doc__ = getattr(_coconut.zip_longest, "__doc__", "Version of zip that fills in missing values with fillvalue.") def __new__(cls, *iterables, **kwargs): - self = {_coconut_}zip.__new__(cls, *iterables, strict=False) + self = _coconut.super({_coconut_}zip_longest, cls).__new__(cls, *iterables, strict=False) self.fillvalue = kwargs.pop("fillvalue", None) if kwargs: raise _coconut.TypeError(cls.__name__ + "() got unexpected keyword arguments " + _coconut.repr(kwargs)) @@ -1080,7 +1087,7 @@ class cycle(_coconut_has_iter): before stopping.""" __slots__ = ("times",) def __new__(cls, iterable, times=None): - self = _coconut_has_iter.__new__(cls, iterable) + self = _coconut.super({_coconut_}cycle, cls).__new__(cls, iterable) if times is None: self.times = None else: @@ -1135,7 +1142,7 @@ class windowsof(_coconut_has_iter): If that is not the desired behavior, fillvalue can be passed and will be used in place of missing values.""" __slots__ = ("size", "fillvalue", "step") def __new__(cls, size, iterable, fillvalue=_coconut_sentinel, step=1): - self = _coconut_has_iter.__new__(cls, iterable) + self = _coconut.super({_coconut_}windowsof, cls).__new__(cls, iterable) self.size = _coconut.operator.index(size) if self.size < 1: raise _coconut.ValueError("windowsof: size must be >= 1; not %r" % (self.size,)) @@ -1177,7 +1184,7 @@ class groupsof(_coconut_has_iter): """ __slots__ = ("group_size", "fillvalue") def __new__(cls, n, iterable, fillvalue=_coconut_sentinel): - self = _coconut_has_iter.__new__(cls, iterable) + self = _coconut.super({_coconut_}groupsof, cls).__new__(cls, iterable) self.group_size = _coconut.operator.index(n) if self.group_size < 1: raise _coconut.ValueError("group size must be >= 1; not %r" % (self.group_size,)) @@ -1453,18 +1460,27 @@ class multiset(_coconut.collections.Counter{comma_object}): if result < 0: raise _coconut.ValueError("multiset has negative count for " + _coconut.repr(item)) return result + def __fmap__(self, func): + return self.__class__(_coconut.dict((func(obj), num) for obj, num in self.items())) {def_total_and_comparisons}{assign_multiset_views}_coconut.abc.MutableSet.register(multiset) -def _coconut_base_makedata(data_type, args): +def _coconut_base_makedata(data_type, args, from_fmap=False, fallback_to_init=False): if _coconut.hasattr(data_type, "_make") and _coconut.issubclass(data_type, _coconut.tuple): return data_type._make(args) if _coconut.issubclass(data_type, (_coconut.range, _coconut.abc.Iterator)): return args if _coconut.issubclass(data_type, _coconut.str): return "".join(args) - return data_type(args) -def makedata(data_type, *args): + if fallback_to_init or _coconut.issubclass(data_type, _coconut.fmappables): + return data_type(args) + if from_fmap: + raise _coconut.TypeError("no known __fmap__ implementation for " + _coconut.repr(data_type) + " (pass fallback_to_init=True to fall back on __init__ and __iter__)") + raise _coconut.TypeError("no known makedata implementation for " + _coconut.repr(data_type) + " (pass fallback_to_init=True to fall back on __init__)") +def makedata(data_type, *args, **kwargs): """Construct an object of the given data_type containing the given arguments.""" - return _coconut_base_makedata(data_type, args) + fallback_to_init = kwargs.pop("fallback_to_init", False) + if kwargs: + raise _coconut.TypeError("makedata() got unexpected keyword arguments " + _coconut.repr(kwargs)) + return _coconut_base_makedata(data_type, args, fallback_to_init=fallback_to_init) {def_datamaker} {class_amap} def fmap(func, obj, **kwargs): @@ -1474,6 +1490,7 @@ def fmap(func, obj, **kwargs): Override by defining obj.__fmap__(func). """ starmap_over_mappings = kwargs.pop("starmap_over_mappings", False) + fallback_to_init = kwargs.pop("fallback_to_init", False) if kwargs: raise _coconut.TypeError("fmap() got unexpected keyword arguments " + _coconut.repr(kwargs)) obj_fmap = _coconut.getattr(obj, "__fmap__", None) @@ -1505,9 +1522,9 @@ def fmap(func, obj, **kwargs): if aiter is not _coconut.NotImplemented: return _coconut_amap(func, aiter) if starmap_over_mappings: - return _coconut_base_makedata(obj.__class__, {_coconut_}starmap(func, obj.items()) if _coconut.isinstance(obj, _coconut.abc.Mapping) else {_coconut_}map(func, obj)) + return _coconut_base_makedata(obj.__class__, {_coconut_}starmap(func, obj.items()) if _coconut.isinstance(obj, _coconut.abc.Mapping) else {_coconut_}map(func, obj), from_fmap=True, fallback_to_init=fallback_to_init) else: - return _coconut_base_makedata(obj.__class__, {_coconut_}map(func, obj.items() if _coconut.isinstance(obj, _coconut.abc.Mapping) else obj)) + return _coconut_base_makedata(obj.__class__, {_coconut_}map(func, obj.items() if _coconut.isinstance(obj, _coconut.abc.Mapping) else obj), from_fmap=True, fallback_to_init=fallback_to_init) def _coconut_memoize_helper(maxsize=None, typed=False): return maxsize, typed def memoize(*args, **kwargs): @@ -1744,7 +1761,7 @@ class lift(_coconut_baseclass): """ __slots__ = ("func",) def __new__(cls, func, *func_args, **func_kwargs): - self = _coconut.object.__new__(cls) + self = _coconut.super({_coconut_}lift, cls).__new__(cls) self.func = func if func_args or func_kwargs: self = self(*func_args, **func_kwargs) @@ -1868,48 +1885,134 @@ def _coconut_call_or_coefficient(func, *args): func = func * x{COMMENT.no_times_equals_to_avoid_modification} return func class _coconut_SupportsAdd(_coconut.typing.Protocol): + """Coconut (+) Protocol. Equivalent to: + + class SupportsAdd[T, U, V](Protocol): + def __add__(self: T, other: U) -> V: + raise NotImplementedError(...) + """ def __add__(self, other): raise NotImplementedError("Protocol methods cannot be called at runtime ((+) in a typing context is a Protocol)") class _coconut_SupportsMinus(_coconut.typing.Protocol): + """Coconut (-) Protocol. Equivalent to: + + class SupportsMinus[T, U, V](Protocol): + def __sub__(self: T, other: U) -> V: + raise NotImplementedError + def __neg__(self: T) -> V: + raise NotImplementedError + """ def __sub__(self, other): raise NotImplementedError("Protocol methods cannot be called at runtime ((-) in a typing context is a Protocol)") def __neg__(self): raise NotImplementedError("Protocol methods cannot be called at runtime ((-) in a typing context is a Protocol)") class _coconut_SupportsMul(_coconut.typing.Protocol): + """Coconut (*) Protocol. Equivalent to: + + class SupportsMul[T, U, V](Protocol): + def __mul__(self: T, other: U) -> V: + raise NotImplementedError(...) + """ def __mul__(self, other): raise NotImplementedError("Protocol methods cannot be called at runtime ((*) in a typing context is a Protocol)") class _coconut_SupportsPow(_coconut.typing.Protocol): + """Coconut (**) Protocol. Equivalent to: + + class SupportsPow[T, U, V](Protocol): + def __pow__(self: T, other: U) -> V: + raise NotImplementedError(...) + """ def __pow__(self, other): raise NotImplementedError("Protocol methods cannot be called at runtime ((**) in a typing context is a Protocol)") class _coconut_SupportsTruediv(_coconut.typing.Protocol): + """Coconut (/) Protocol. Equivalent to: + + class SupportsTruediv[T, U, V](Protocol): + def __truediv__(self: T, other: U) -> V: + raise NotImplementedError(...) + """ def __truediv__(self, other): raise NotImplementedError("Protocol methods cannot be called at runtime ((/) in a typing context is a Protocol)") class _coconut_SupportsFloordiv(_coconut.typing.Protocol): + """Coconut (//) Protocol. Equivalent to: + + class SupportsFloordiv[T, U, V](Protocol): + def __floordiv__(self: T, other: U) -> V: + raise NotImplementedError(...) + """ def __floordiv__(self, other): raise NotImplementedError("Protocol methods cannot be called at runtime ((//) in a typing context is a Protocol)") class _coconut_SupportsMod(_coconut.typing.Protocol): + """Coconut (%) Protocol. Equivalent to: + + class SupportsMod[T, U, V](Protocol): + def __mod__(self: T, other: U) -> V: + raise NotImplementedError(...) + """ def __mod__(self, other): raise NotImplementedError("Protocol methods cannot be called at runtime ((%) in a typing context is a Protocol)") class _coconut_SupportsAnd(_coconut.typing.Protocol): + """Coconut (&) Protocol. Equivalent to: + + class SupportsAnd[T, U, V](Protocol): + def __and__(self: T, other: U) -> V: + raise NotImplementedError(...) + """ def __and__(self, other): raise NotImplementedError("Protocol methods cannot be called at runtime ((&) in a typing context is a Protocol)") class _coconut_SupportsXor(_coconut.typing.Protocol): + """Coconut (^) Protocol. Equivalent to: + + class SupportsXor[T, U, V](Protocol): + def __xor__(self: T, other: U) -> V: + raise NotImplementedError(...) + """ def __xor__(self, other): raise NotImplementedError("Protocol methods cannot be called at runtime ((^) in a typing context is a Protocol)") class _coconut_SupportsOr(_coconut.typing.Protocol): + """Coconut (|) Protocol. Equivalent to: + + class SupportsOr[T, U, V](Protocol): + def __or__(self: T, other: U) -> V: + raise NotImplementedError(...) + """ def __or__(self, other): raise NotImplementedError("Protocol methods cannot be called at runtime ((|) in a typing context is a Protocol)") class _coconut_SupportsLshift(_coconut.typing.Protocol): + """Coconut (<<) Protocol. Equivalent to: + + class SupportsLshift[T, U, V](Protocol): + def __lshift__(self: T, other: U) -> V: + raise NotImplementedError(...) + """ def __lshift__(self, other): raise NotImplementedError("Protocol methods cannot be called at runtime ((<<) in a typing context is a Protocol)") class _coconut_SupportsRshift(_coconut.typing.Protocol): + """Coconut (>>) Protocol. Equivalent to: + + class SupportsRshift[T, U, V](Protocol): + def __rshift__(self: T, other: U) -> V: + raise NotImplementedError(...) + """ def __rshift__(self, other): raise NotImplementedError("Protocol methods cannot be called at runtime ((>>) in a typing context is a Protocol)") class _coconut_SupportsMatmul(_coconut.typing.Protocol): + """Coconut (@) Protocol. Equivalent to: + + class SupportsMatmul[T, U, V](Protocol): + def __matmul__(self: T, other: U) -> V: + raise NotImplementedError(...) + """ def __matmul__(self, other): raise NotImplementedError("Protocol methods cannot be called at runtime ((@) in a typing context is a Protocol)") class _coconut_SupportsInv(_coconut.typing.Protocol): + """Coconut (~) Protocol. Equivalent to: + + class SupportsInv[T, V](Protocol): + def __invert__(self: T) -> V: + raise NotImplementedError(...) + """ def __invert__(self): raise NotImplementedError("Protocol methods cannot be called at runtime ((~) in a typing context is a Protocol)") _coconut_self_match_types = {self_match_types} -_coconut_Expected, _coconut_MatchError, _coconut_count, _coconut_enumerate, _coconut_flatten, _coconut_filter, _coconut_ident, _coconut_map, _coconut_multiset, _coconut_range, _coconut_reiterable, _coconut_reversed, _coconut_starmap, _coconut_tee, _coconut_zip, TYPE_CHECKING, reduce, takewhile, dropwhile = Expected, MatchError, count, enumerate, flatten, filter, ident, map, multiset, range, reiterable, reversed, starmap, tee, zip, False, _coconut.functools.reduce, _coconut.itertools.takewhile, _coconut.itertools.dropwhile +_coconut_Expected, _coconut_MatchError, _coconut_cartesian_product, _coconut_count, _coconut_cycle, _coconut_enumerate, _coconut_flatten, _coconut_filter, _coconut_groupsof, _coconut_ident, _coconut_lift, _coconut_map, _coconut_multiset, _coconut_range, _coconut_reiterable, _coconut_reversed, _coconut_scan, _coconut_starmap, _coconut_tee, _coconut_windowsof, _coconut_zip, _coconut_zip_longest, TYPE_CHECKING, reduce, takewhile, dropwhile = Expected, MatchError, cartesian_product, count, cycle, enumerate, flatten, filter, groupsof, ident, lift, map, multiset, range, reiterable, reversed, scan, starmap, tee, windowsof, zip, zip_longest, False, _coconut.functools.reduce, _coconut.itertools.takewhile, _coconut.itertools.dropwhile{COMMENT.anything_added_here_should_be_copied_to_stub_file} diff --git a/coconut/compiler/util.py b/coconut/compiler/util.py index 126136543..e6de4537f 100644 --- a/coconut/compiler/util.py +++ b/coconut/compiler/util.py @@ -53,6 +53,7 @@ Regex, Empty, Literal, + CaselessLiteral, Group, ParserElement, _trim_arity, @@ -886,6 +887,14 @@ def any_len_perm_at_least_one(*elems, **kwargs): return any_len_perm_with_one_of_each_group(*groups_and_elems) +def caseless_literal(literalstr, suppress=False): + """Version of CaselessLiteral that always parses to the given literalstr.""" + if suppress: + return CaselessLiteral(literalstr).suppress() + else: + return fixto(CaselessLiteral(literalstr), literalstr) + + # ----------------------------------------------------------------------------------------------------------------------- # UTILITIES: # ----------------------------------------------------------------------------------------------------------------------- @@ -1040,6 +1049,21 @@ def split_comment(line, move_indents=False): return line[:i] + indent, line[i:] +def extract_line_num_from_comment(line, default=None): + """Extract the line number from a line with a line number comment, else return default.""" + _, all_comments = split_comment(line) + for comment in all_comments.split("#"): + words = comment.strip().split(None, 1) + if words: + first_word = words[0].strip(":") + try: + return int(first_word) + except ValueError: + pass + logger.log("failed to extract line num comment from", line) + return default + + def rem_comment(line): """Remove a comment from a line.""" base, comment = split_comment(line) diff --git a/coconut/constants.py b/coconut/constants.py index e42f8a8cb..fc8815357 100644 --- a/coconut/constants.py +++ b/coconut/constants.py @@ -80,6 +80,7 @@ def get_bool_env_var(env_var, default=False): ((PY2 and not PY26) or PY35) and not (PYPY and WINDOWS) and (PY37 or not PYPY) + and sys.version_info[:2] != (3, 7) ) MYPY = ( PY37 @@ -647,6 +648,8 @@ def get_bool_env_var(env_var, default=False): jupyter_console_commands = ("console", "qtconsole") +create_package_retries = 1 + # ----------------------------------------------------------------------------------------------------------------------- # HIGHLIGHTER CONSTANTS: # ----------------------------------------------------------------------------------------------------------------------- @@ -830,7 +833,8 @@ def get_bool_env_var(env_var, default=False): ), "kernel": ( ("ipython", "py2"), - ("ipython", "py3;py<38"), + ("ipython", "py3;py<37"), + ("ipython", "py==37"), ("ipython", "py38"), ("ipykernel", "py2"), ("ipykernel", "py3;py<38"), @@ -906,17 +910,17 @@ def get_bool_env_var(env_var, default=False): "argparse": (1, 4), "pexpect": (4,), ("trollius", "py2;cpy"): (2, 2), - "requests": (2, 29), + "requests": (2, 31), ("numpy", "py34"): (1,), ("numpy", "py2;cpy"): (1,), ("dataclasses", "py==36"): (0, 8), ("aenum", "py<34"): (3,), "pydata-sphinx-theme": (0, 13), "myst-parser": (1,), - "mypy[python2]": (1, 2), - ("jupyter-console", "py37"): (6,), + "mypy[python2]": (1, 3), + ("jupyter-console", "py37"): (6, 6), ("typing", "py<35"): (3, 10), - ("typing_extensions", "py37"): (4, 5), + ("typing_extensions", "py37"): (4, 6), ("ipython", "py38"): (8,), ("ipykernel", "py38"): (6,), ("jedi", "py39"): (0, 18), @@ -926,13 +930,15 @@ def get_bool_env_var(env_var, default=False): # don't upgrade until myst-parser supports the new version "sphinx": (6,), - # don't upgrade this; it breaks on Python 3.6 + # don't upgrade this; it breaks on Python 3.7 + ("ipython", "py==37"): (7, 34), + # don't upgrade these; it breaks on Python 3.6 ("pandas", "py36"): (1,), ("jupyter-client", "py36"): (7, 1, 2), ("typing_extensions", "py==36"): (4, 1), # don't upgrade these; they break on Python 3.5 ("ipykernel", "py3;py<38"): (5, 5), - ("ipython", "py3;py<38"): (7, 9), + ("ipython", "py3;py<37"): (7, 9), ("jupyter-console", "py>=35;py<37"): (6, 1), ("jupyter-client", "py==35"): (6, 1, 12), ("jupytext", "py3"): (1, 8), @@ -965,12 +971,13 @@ def get_bool_env_var(env_var, default=False): # should match the reqs with comments above pinned_reqs = ( "sphinx", + ("ipython", "py==37"), ("pandas", "py36"), ("jupyter-client", "py36"), ("typing_extensions", "py==36"), ("jupyter-client", "py<35"), ("ipykernel", "py3;py<38"), - ("ipython", "py3;py<38"), + ("ipython", "py3;py<37"), ("jupyter-console", "py>=35;py<37"), ("jupyter-client", "py==35"), ("jupytext", "py3"), @@ -1003,7 +1010,7 @@ def get_bool_env_var(env_var, default=False): ("prompt_toolkit", "mark2"): _, ("jedi", "py<39"): _, ("pywinpty", "py2;windows"): _, - ("ipython", "py3;py<38"): _, + ("ipython", "py3;py<37"): _, } classifiers = ( @@ -1139,7 +1146,7 @@ def get_bool_env_var(env_var, default=False): # INTEGRATION CONSTANTS: # ----------------------------------------------------------------------------------------------------------------------- -# must be replicated in DOCS +# must be replicated in DOCS; must include --line-numbers for xonsh line number extraction coconut_kernel_kwargs = dict(target="sys", line_numbers=True, keep_lines=True, no_wrap=True) icoconut_dir = os.path.join(base_dir, "icoconut") diff --git a/coconut/exceptions.py b/coconut/exceptions.py index c49429cf0..33e0c40b4 100644 --- a/coconut/exceptions.py +++ b/coconut/exceptions.py @@ -97,7 +97,7 @@ def __init__(self, message, source=None, point=None, ln=None, extra=None, endpoi @property def kwargs(self): """Get the arguments as keyword arguments.""" - return dict(zip(self.args, self.argnames)) + return dict(zip(self.argnames, self.args)) def message(self, message, source, point, ln, extra=None, endpoint=None, filename=None): """Creates a SyntaxError-like message.""" @@ -185,16 +185,18 @@ def syntax_err(self): """Creates a SyntaxError.""" kwargs = self.kwargs if self.point_to_endpoint and "endpoint" in kwargs: - point = kwargs.pop("endpoint") + point = kwargs["endpoint"] else: - point = kwargs.pop("point") - kwargs["point"] = kwargs["endpoint"] = None - ln = kwargs.pop("ln") - filename = kwargs.pop("filename", None) + point = kwargs.get("point") + ln = kwargs.get("ln") + filename = kwargs.get("filename") + kwargs["point"] = kwargs["endpoint"] = kwargs["ln"] = kwargs["filename"] = None err = SyntaxError(self.message(**kwargs)) - err.offset = point - err.lineno = ln + if point is not None: + err.offset = point + if ln is not None: + err.lineno = ln if filename is not None: err.filename = filename return err diff --git a/coconut/icoconut/root.py b/coconut/icoconut/root.py index 45fc6f1ad..326a2dd62 100644 --- a/coconut/icoconut/root.py +++ b/coconut/icoconut/root.py @@ -21,6 +21,7 @@ import os import sys +import logging try: import asyncio @@ -44,11 +45,8 @@ conda_build_env_var, coconut_kernel_kwargs, ) -from coconut.terminal import ( - logger, - internal_assert, -) -from coconut.util import override +from coconut.terminal import logger +from coconut.util import override, memoize_with_exceptions from coconut.compiler import Compiler from coconut.compiler.util import should_indent from coconut.command.util import Runner @@ -94,25 +92,10 @@ RUNNER = Runner(COMPILER) -parse_block_memo = {} - +@memoize_with_exceptions(128) def memoized_parse_block(code): - """Memoized version of parse_block.""" - internal_assert(lambda: code not in parse_block_memo.values(), "attempted recompilation of", code) - success, result = parse_block_memo.get(code, (None, None)) - if success is None: - try: - parsed = COMPILER.parse_block(code, keep_state=True) - except Exception as err: - success, result = False, err - else: - success, result = True, parsed - parse_block_memo[code] = (success, result) - if success: - return result - else: - raise result + return COMPILER.parse_block(code, keep_state=True) def syntaxerr_memoized_parse_block(code): @@ -283,6 +266,11 @@ class CoconutKernel(IPythonKernel, object): }, ] + def __init__(self, *args, **kwargs): + super(CoconutKernel, self).__init__(*args, **kwargs) + if self.log is None: + self.log = logging.getLogger(__name__) + @override def do_complete(self, code, cursor_pos): # first try with Jedi completions diff --git a/coconut/integrations.py b/coconut/integrations.py index bbed00a40..7636e1e7e 100644 --- a/coconut/integrations.py +++ b/coconut/integrations.py @@ -25,6 +25,7 @@ coconut_kernel_kwargs, disabled_xonsh_modes, ) +from coconut.util import memoize_with_exceptions # ----------------------------------------------------------------------------------------------------------------------- # IPYTHON: @@ -94,6 +95,15 @@ class CoconutXontribLoader(object): runner = None timing_info = [] + @memoize_with_exceptions(128) + def _base_memoized_parse_xonsh(self, code): + return self.compiler.parse_xonsh(code, keep_state=True) + + def memoized_parse_xonsh(self, code): + """Memoized self.compiler.parse_xonsh.""" + # .strip() outside the memoization + return self._base_memoized_parse_xonsh(code.strip()) + def new_parse(self, parser, code, mode="exec", *args, **kwargs): """Coconut-aware version of xonsh's _parse.""" if self.loaded and mode not in disabled_xonsh_modes: @@ -106,7 +116,7 @@ def new_parse(self, parser, code, mode="exec", *args, **kwargs): parse_start_time = get_clock_time() quiet, logger.quiet = logger.quiet, True try: - code = self.compiler.parse_xonsh(code, keep_state=True) + code = self.memoized_parse_xonsh(code) except CoconutException as err: err_str = format_error(err).splitlines()[0] code += " #" + err_str @@ -115,17 +125,49 @@ def new_parse(self, parser, code, mode="exec", *args, **kwargs): self.timing_info.append(("parse", get_clock_time() - parse_start_time)) return parser.__class__.parse(parser, code, mode=mode, *args, **kwargs) - def new_try_subproc_toks(self, ctxtransformer, *args, **kwargs): + def new_try_subproc_toks(self, ctxtransformer, node, *args, **kwargs): """Version of try_subproc_toks that handles the fact that Coconut code may have different columns than Python code.""" mode = ctxtransformer.mode if self.loaded: ctxtransformer.mode = "eval" try: - return ctxtransformer.__class__.try_subproc_toks(ctxtransformer, *args, **kwargs) + return ctxtransformer.__class__.try_subproc_toks(ctxtransformer, node, *args, **kwargs) finally: ctxtransformer.mode = mode + def new_ctxvisit(self, ctxtransformer, node, inp, *args, **kwargs): + """Version of ctxvisit that ensures looking up original lines in inp + using Coconut line numbers will work properly.""" + if self.loaded: + from xonsh.tools import get_logical_line + + # hide imports to avoid circular dependencies + from coconut.terminal import logger + from coconut.compiler.util import extract_line_num_from_comment + + compiled = self.memoized_parse_xonsh(inp) + + original_lines = tuple(inp.splitlines()) + used_lines = set() + new_inp_lines = [] + last_ln = 1 + for compiled_line in compiled.splitlines(): + ln = extract_line_num_from_comment(compiled_line, default=last_ln + 1) + try: + line, _, _ = get_logical_line(original_lines, ln - 1) + except IndexError: + logger.log_exc() + line = original_lines[-1] + if line in used_lines: + line = "" + else: + used_lines.add(line) + new_inp_lines.append(line) + last_ln = ln + inp = "\n".join(new_inp_lines) + "\n" + return ctxtransformer.__class__.ctxvisit(ctxtransformer, node, inp, *args, **kwargs) + def __call__(self, xsh, **kwargs): # hide imports to avoid circular dependencies from coconut.util import get_clock_time @@ -147,11 +189,12 @@ def __call__(self, xsh, **kwargs): main_parser.parse = MethodType(self.new_parse, main_parser) ctxtransformer = xsh.execer.ctxtransformer + ctxtransformer.try_subproc_toks = MethodType(self.new_try_subproc_toks, ctxtransformer) + ctxtransformer.ctxvisit = MethodType(self.new_ctxvisit, ctxtransformer) + ctx_parser = ctxtransformer.parser ctx_parser.parse = MethodType(self.new_parse, ctx_parser) - ctxtransformer.try_subproc_toks = MethodType(self.new_try_subproc_toks, ctxtransformer) - self.timing_info.append(("load", get_clock_time() - start_time)) self.loaded = True diff --git a/coconut/root.py b/coconut/root.py index 48e7e69b1..712f277b0 100644 --- a/coconut/root.py +++ b/coconut/root.py @@ -23,7 +23,7 @@ # VERSION: # ----------------------------------------------------------------------------------------------------------------------- -VERSION = "3.0.0" +VERSION = "3.0.1" VERSION_NAME = None # False for release, int >= 1 for develop DEVELOP = False diff --git a/coconut/tests/main_test.py b/coconut/tests/main_test.py index d73a33d0b..444228b19 100644 --- a/coconut/tests/main_test.py +++ b/coconut/tests/main_test.py @@ -848,14 +848,6 @@ def test_simple_minify(self): @add_test_func_names class TestExternal(unittest.TestCase): - # more appveyor timeout prevention - if not (WINDOWS and PY2): - def test_pyprover(self): - with using_path(pyprover): - comp_pyprover() - if PY38: - run_pyprover() - if not PYPY or PY2: def test_prelude(self): with using_path(prelude): @@ -869,11 +861,19 @@ def test_bbopt(self): if not PYPY and PY38 and not PY310: install_bbopt() - def test_pyston(self): - with using_path(pyston): - comp_pyston(["--no-tco"]) - if PYPY and PY2: - run_pyston() + # more appveyor timeout prevention + if not WINDOWS: + def test_pyprover(self): + with using_path(pyprover): + comp_pyprover() + if PY38: + run_pyprover() + + def test_pyston(self): + with using_path(pyston): + comp_pyston(["--no-tco"]) + if PYPY and PY2: + run_pyston() # ----------------------------------------------------------------------------------------------------------------------- diff --git a/coconut/tests/src/cocotest/agnostic/main.coco b/coconut/tests/src/cocotest/agnostic/main.coco index f9a5a067d..2e5402122 100644 --- a/coconut/tests/src/cocotest/agnostic/main.coco +++ b/coconut/tests/src/cocotest/agnostic/main.coco @@ -56,6 +56,7 @@ def run_main(outer_MatchError, test_easter_eggs=False) -> bool: non_py26_test, non_py32_test, py3_spec_test, + py33_spec_test, py36_spec_test, py37_spec_test, py38_spec_test, @@ -66,6 +67,8 @@ def run_main(outer_MatchError, test_easter_eggs=False) -> bool: assert non_py32_test() is True if sys.version_info >= (3,): assert py3_spec_test() is True + if sys.version_info >= (3, 3): + assert py33_spec_test() is True if sys.version_info >= (3, 6): assert py36_spec_test(tco=using_tco) is True if sys.version_info >= (3, 7): diff --git a/coconut/tests/src/cocotest/agnostic/primary.coco b/coconut/tests/src/cocotest/agnostic/primary.coco index 8f61821a0..453106920 100644 --- a/coconut/tests/src/cocotest/agnostic/primary.coco +++ b/coconut/tests/src/cocotest/agnostic/primary.coco @@ -1589,4 +1589,16 @@ def primary_test() -> bool: assert ["abc" ;; "def"] == [['abc'], ['def']] assert {"a":0, "b":1}$[0] == "a" assert (|0, NotImplemented, 2|)$[1] is NotImplemented + assert m{1, 1, 2} |> fmap$(.+1) == m{2, 2, 3} + assert (+) ..> ((*) ..> (/)) == (+) ..> (*) ..> (/) == ((+) ..> (*)) ..> (/) + def f(x, y=1) = x, y # type: ignore + f.is_f = True # type: ignore + assert (f ..*> (+)).is_f # type: ignore + really_long_var = 10 + assert (...=really_long_var) == (10,) + assert (...=really_long_var, abc="abc") == (10, "abc") + assert (abc="abc", ...=really_long_var) == ("abc", 10) + assert (...=really_long_var).really_long_var == 10 + n = [0] + assert n[0] == 0 return True diff --git a/coconut/tests/src/cocotest/agnostic/specific.coco b/coconut/tests/src/cocotest/agnostic/specific.coco index 128f82dcd..9c936dddd 100644 --- a/coconut/tests/src/cocotest/agnostic/specific.coco +++ b/coconut/tests/src/cocotest/agnostic/specific.coco @@ -44,6 +44,19 @@ def py3_spec_test() -> bool: return True +def py33_spec_test() -> bool: + """Tests for any py33+ version.""" + from inspect import signature + def f(x, y=1) = x, y + def g(a, b=2) = a, b + assert signature(f ..*> g) == signature(f) == signature(f ..> g) + assert signature(f <*.. g) == signature(g) == signature(f <.. g) + assert signature(f$(0) ..> g) == signature(f$(0)) + assert signature(f ..*> (+)) == signature(f) + assert signature((f ..*> g) ..*> g) == signature(f) + return True + + def py36_spec_test(tco: bool) -> bool: """Tests for any py36+ version.""" from dataclasses import dataclass diff --git a/coconut/tests/src/cocotest/agnostic/suite.coco b/coconut/tests/src/cocotest/agnostic/suite.coco index 46c2fdd5f..b542db14e 100644 --- a/coconut/tests/src/cocotest/agnostic/suite.coco +++ b/coconut/tests/src/cocotest/agnostic/suite.coco @@ -1045,6 +1045,11 @@ forward 2""") == 900 assert get_glob() == 0 assert wrong_get_set_glob(20) == 10 assert take_xy(xy("a", "b")) == ("a", "b") + assert InitAndIter(range(3)) |> fmap$((.+1), fallback_to_init=True) == InitAndIter(range(1, 4)) + assert_raises(-> InitAndIter(range(3)) |> fmap$(.+1), TypeError) + really_long_var = 10 + assert ret_args_kwargs(...=really_long_var) == ((), {"really_long_var": 10}) == ret_args_kwargs$(...=really_long_var)() + assert ret_args_kwargs(123, ...=really_long_var, abc="abc") == ((123,), {"really_long_var": 10, "abc": "abc"}) == ret_args_kwargs$(123, ...=really_long_var, abc="abc")() # must come at end assert fibs_calls[0] == 1 diff --git a/coconut/tests/src/cocotest/agnostic/util.coco b/coconut/tests/src/cocotest/agnostic/util.coco index 86aa712a8..59b3ec93c 100644 --- a/coconut/tests/src/cocotest/agnostic/util.coco +++ b/coconut/tests/src/cocotest/agnostic/util.coco @@ -530,6 +530,13 @@ def summer(): summer.acc += summer.args.pop() return summer() +class InitAndIter: + def __init__(self, it): + self.it = tuple(it) + def __iter__(self) = iter(self.it) + def __eq__(self, other) = + self.__class__ == other.__class__ and self.it == other.it + # Data Blocks: try: diff --git a/coconut/tests/src/cocotest/non_strict/non_strict_test.coco b/coconut/tests/src/cocotest/non_strict/non_strict_test.coco index 099e0dad2..33bea2e47 100644 --- a/coconut/tests/src/cocotest/non_strict/non_strict_test.coco +++ b/coconut/tests/src/cocotest/non_strict/non_strict_test.coco @@ -45,7 +45,7 @@ def non_strict_test() -> bool: assert False match A.CONST in 11: # type: ignore assert False - assert A.CONST == 10 + assert A.CONST == 10 == A.("CONST") match {"a": 1, "b": 2}: # type: ignore case {"a": a}: pass diff --git a/coconut/tests/src/extras.coco b/coconut/tests/src/extras.coco index a94313b5a..6411ff8a2 100644 --- a/coconut/tests/src/extras.coco +++ b/coconut/tests/src/extras.coco @@ -29,34 +29,41 @@ if IPY: if PY35: import asyncio from coconut.icoconut import CoconutKernel # type: ignore + from jupyter_client.session import Session else: CoconutKernel = None # type: ignore + Session = object # type: ignore -def assert_raises(c, exc, not_exc=None, err_has=None): - """Test whether callable c raises an exception of type exc.""" - if not_exc is None and exc is CoconutSyntaxError: - not_exc = CoconutParseError +def assert_raises(c, Exc, not_Exc=None, err_has=None): + """Test whether callable c raises an exception of type Exc.""" + if not_Exc is None and Exc is CoconutSyntaxError: + not_Exc = CoconutParseError # we don't check err_has without the computation graph since errors can be quite different if not USE_COMPUTATION_GRAPH: err_has = None try: c() - except exc as err: - if not_exc is not None: - assert not isinstance(err, not_exc), f"{err} instance of {not_exc}" + except Exc as err: + if not_Exc is not None: + assert not isinstance(err, not_Exc), f"{err} instance of {not_Exc}" if err_has is not None: if isinstance(err_has, tuple): assert any(has in str(err) for has in err_has), f"{str(err)!r} does not contain any of {err_has!r}" else: assert err_has in str(err), f"{err_has!r} not in {str(err)!r}" - if exc `isinstance` CoconutSyntaxError: - assert "SyntaxError" in str(exc.syntax_err()) + if err `isinstance` CoconutSyntaxError: + syntax_err = err.syntax_err() + assert syntax_err `isinstance` SyntaxError + syntax_err_str = str(syntax_err) + assert syntax_err_str.splitlines()$[0] in str(err), (syntax_err_str, str(err)) + assert "unprintable" not in syntax_err_str, syntax_err_str + assert " bool: assert_raises((def -> import \(_coconut)), ImportError, err_has="should never be done at runtime") # NOQA assert_raises((def -> import \_coconut), ImportError, err_has="should never be done at runtime") # NOQA @@ -359,30 +375,50 @@ def test_kernel() -> bool: asyncio.set_event_loop(loop) else: loop = None # type: ignore + k = CoconutKernel() + fake_session = FakeSession() + k.shell.displayhook.session = fake_session + exec_result = k.do_execute("derp = pow$(?, 2)", False, True, {"two": "(+)(1, 1)"}, True) |> unwrap_future$(loop) assert exec_result["status"] == "ok" assert exec_result["user_expressions"]["two"]["data"]["text/plain"] == "2" - assert k.do_execute("operator ++", False, True, {}, True) |> unwrap_future$(loop) - assert k.do_execute("(++) = 1", False, True, {}, True) |> unwrap_future$(loop) + + assert k.do_execute("operator ++", False, True, {}, True) |> unwrap_future$(loop) |> .["status"] == "ok" + assert k.do_execute("(++) = 1", False, True, {}, True) |> unwrap_future$(loop) |> .["status"] == "ok" + + fail_result = k.do_execute("f([] {})", False, True, {}, True) |> unwrap_future$(loop) + captured_msg_type, captured_msg_content = fake_session.captured_messages[-1] + assert fail_result["status"] == "error" == captured_msg_type, fail_result + assert fail_result["ename"] == "SyntaxError" == captured_msg_content["ename"], fail_result + assert fail_result["traceback"] == captured_msg_content["traceback"], fail_result + assert len(fail_result["traceback"]) == 1, fail_result + assert "parsing failed" in fail_result["traceback"][0], fail_result + assert fail_result["evalue"] == captured_msg_content["evalue"], fail_result + assert "parsing failed" in fail_result["evalue"], fail_result + assert k.do_is_complete("if abc:")["status"] == "incomplete" assert k.do_is_complete("f(")["status"] == "incomplete" assert k.do_is_complete("abc")["status"] == "complete" + inspect_result = k.do_inspect("derp", 4, 0) assert inspect_result["status"] == "ok" assert inspect_result["found"] assert inspect_result["data"]["text/plain"] + complete_result = k.do_complete("der", 1) assert complete_result["status"] == "ok" assert "derp" in complete_result["matches"] assert complete_result["cursor_start"] == 0 assert complete_result["cursor_end"] == 1 + keyword_complete_result = k.do_complete("ma", 1) assert keyword_complete_result["status"] == "ok" assert "match" in keyword_complete_result["matches"] assert "map" in keyword_complete_result["matches"] assert keyword_complete_result["cursor_start"] == 0 assert keyword_complete_result["cursor_end"] == 1 + return True diff --git a/coconut/util.py b/coconut/util.py index 216d0e4e3..98489f5b4 100644 --- a/coconut/util.py +++ b/coconut/util.py @@ -205,12 +205,35 @@ def noop_ctx(): def memoize(maxsize=None, *args, **kwargs): """Decorator that memoizes a function, preventing it from being recomputed if it is called multiple times with the same arguments.""" + assert maxsize is None or isinstance(maxsize, int), maxsize if lru_cache is None: return lambda func: func else: return lru_cache(maxsize, *args, **kwargs) +def memoize_with_exceptions(*memo_args, **memo_kwargs): + """Decorator that works like memoize but also memoizes exceptions.""" + def memoizer(func): + @memoize(*memo_args, **memo_kwargs) + def memoized_safe_func(*args, **kwargs): + res = exc = None + try: + res = func(*args, **kwargs) + except Exception as exc: + return res, exc + else: + return res, exc + + def memoized_func(*args, **kwargs): + res, exc = memoized_safe_func(*args, **kwargs) + if exc is not None: + raise exc + return res + return memoized_func + return memoizer + + class keydefaultdict(defaultdict, object): """Version of defaultdict that calls the factory with the key.""" diff --git a/setup.cfg b/setup.cfg index 2e9053c06..7fa9076ba 100644 --- a/setup.cfg +++ b/setup.cfg @@ -2,4 +2,5 @@ universal = 1 [metadata] -license_file = LICENSE.txt +license_files = + LICENSE.txt