move python to src to avoid import confusion

Author: bollwyvl

.github/workflows/test.yml

@@ -69,8 +69,10 @@ jobs:
       - name: Upload test config
         uses: actions/upload-artifact@v3
         with:
-          name: pyproject-toml
-          path: pyproject.toml
+          name: test-assets
+          path: |
+            pyproject.toml
+            examples
           if-no-files-found: error
 
   test:
@@ -145,7 +147,7 @@ jobs:
 
       - uses: actions/download-artifact@v3
         with:
-          name: pyproject-toml
+          name: test-assets
 
       - name: Install test deps
         run: |-
@@ -157,6 +159,8 @@ jobs:
           jupyter labextension list 2>&1 | grep -ie "@jupyterlite/pyodide-kernel.*OK"
 
       - name: Run the tests
+        env:
+          LITE_PYODIDE_KERNEL_DEMO: ./examples
         run: pytest
 
       - name: Upload reports

.gitignore

@@ -5,7 +5,7 @@ node_modules/
 *.egg-info/
 .ipynb_checkpoints
 *.tsbuildinfo
-jupyterlite_pyodide_kernel/labextension
+src/jupyterlite_pyodide_kernel/labextension
 
 # Created by https://www.gitignore.io/api/python
 # Edit at https://www.gitignore.io/?templates=python
@@ -118,4 +118,4 @@ packages/pyodide-kernel/pypi/*.json
 _pypi.ts
 .jupyterlite.doit.db
 _output
-jupyterlite_pyodide_kernel/tests/fixtures/.pyodide*/
+src/jupyterlite_pyodide_kernel/tests/fixtures/.pyodide*/

.prettierignore

@@ -2,7 +2,7 @@ node_modules/
 **/node_modules
 **/lib
 **/package.json
-jupyterlite_pyodide_kernel/labextension/
+src/jupyterlite_pyodide_kernel/labextension/
 **/.pyodide/
 .pytest_cache/
 docs/

.readthedocs.yml

@@ -7,23 +7,15 @@ build:
     nodejs: '18'
   apt_packages:
     - libarchive-dev
-
   jobs:
     pre_build:
+      - python -m pip install -r requirements-dev.txt
       - jlpm --frozen-lockfile
-      - jlpm build
+      - jlpm build:prod
       - jlpm dist
       # pre-build the lite site to isolate build errors
       - jlpm docs:lite
 
-python:
-  install:
-    - method: pip
-      path: .
-      extra_requirements:
-        - dev
-        - docs
-
 sphinx:
   builder: html
   configuration: docs/conf.py

examples/intro.ipynb

@@ -81,7 +81,9 @@
    "source": [
     "## What _can't_ it do?\n",
     "\n",
-    "While a very broad subset of the CPython standard library, and much of the open source, scientific Python ecosystem are available, there are some limitation."
+    "While a very broad subset of the CPython standard library, and much of the open source, scientific Python ecosystem are available, there are some limitation.\n",
+    "\n",
+    "The best way to find out is to **try it out**! However, some known implementation-level limitations are listed below."
    ]
   },
   {
@@ -91,14 +93,28 @@
    "source": [
     "### Differences from CPython\n",
     "\n",
-    "A number of aspects of the underlying Python implementation that don't work the same way as their upstream counterparts, if at all.\n",
+    "A number of aspects of the underlying Python implementation that don't work the same way as their upstream counterparts, if at all. Uses of these features will need to be updated, or put behind some check.\n",
     "\n",
     "- no threads\n",
     "- no subprocesses\n",
-    "- no custom event loops\n",
-    "- some blocking features are made asynchronous, such as `await input` \n",
+    "- no custom event asynchronous loops\n",
+    "- package `data_files` are not installed\n",
+    "- some blocking features are made asynchronous\n",
+    "  - `input` must be `await`ed\n",
+    "  - the `urllib` family of HTTP features don't work\n",
+    "      - though [pyodide-http](https://github.com/koenvo/pyodide-http) can be explicitly installed, imported and used"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "961bb18f-954b-4624-bf85-774d451addec",
+   "metadata": {},
+   "source": [
+    "### Differences from `IPython` and `ipykernel`\n",
+    "\n",
+    "`jupyterlite-pyodide-kernel` ships a _shim_ of `ipykernel`, and changes a number of features in `ipython`\n",
     "\n",
-    "Libraries and notebooks that use these features will need to be updated."
+    "- many [magics](https://ipython.readthedocs.io/en/stable/interactive/magics.html) don't work"
    ]
   },
   {
@@ -107,7 +123,7 @@
    "metadata": {},
    "source": [
     "### Differences from Pyodide\n",
-    "Inside a [WebWorker](https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Using_web_workers), there are a number of features that don't work (yet).\n",
+    "Because `jupyterlite-pyodide-kernel` runs inside a [WebWorker](https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Using_web_workers), a number of features don't work (yet).\n",
     "\n",
     "- no access to the browser's DOM\n",
     "    - in particular, the Pyodide [matplotlib](https://github.com/pyodide/matplotlib-pyodide) backend will not work"
@@ -122,15 +138,15 @@
     "\n",
     "This site follows the pattern for an [offline site](https://jupyterlite.readthedocs.io/en/0.1.0-beta/howto/configure/advanced/offline.html) based on a minimal, but fully self-contained, distribution of Pyodide. \n",
     "\n",
-    "> ℹ️ For a more full-featured site with full Pyodide distributions, hosted from a CDN, see: \n",
-    ">\n",
-    "> - the JupyterLite [documentation](https://jupyterlite.readthedocs.io) on [ReadTheDocs](https://jupyterlite.readthedocs.io/en/0.1.0-beta/reference/demo.html)\n",
-    "> - the JupyterLite [demo](https://jupyterlite.github.io/demo) on [GitHub Pages](https://jupyterlite.readthedocs.io/en/0.1.0-beta/quickstart/deploy.html)\n",
-    "\n",
     "This means:\n",
     "- no additional resources are loaded from any other sites on the internet\n",
     "  - serving from a content delivery network (CDN) is often faster, but _could_ disappear or change in the future \n",
-    "- only the Python 3.10 standard library is available by default"
+    "- only the Python 3.10 standard library is available by default\n",
+    "\n",
+    "> ℹ️ For a more full-featured site with full Pyodide distributions, hosted from a CDN, see: \n",
+    ">\n",
+    "> - the JupyterLite [documentation](https://jupyterlite.readthedocs.io) on [ReadTheDocs](https://jupyterlite.readthedocs.io/en/0.1.0-beta/reference/demo.html)\n",
+    "> - the JupyterLite [demo](https://jupyterlite.github.io/demo) on [GitHub Pages](https://jupyterlite.readthedocs.io/en/0.1.0-beta/quickstart/deploy.html)"
    ]
   },
   {
@@ -184,7 +200,7 @@
    "source": [
     "## How do I use more client extension packages?\n",
     "\n",
-    "Some python packages, like widgets, will require client extensions, and need to be available when the site loads for a site visitor.\n",
+    "Some python packages, like all Jupyter widgets, will require client extensions, and need to be available when the site loads for a site visitor.\n",
     "\n",
     "Any number of JupyterLab [extensions]() can be [added at **build time**](https://jupyterlite.readthedocs.io/en/latest/howto/configure/simple_extensions.html), but not all will be compatible:\n",
     "\n",

examples/jupyter_lite_config.json

@@ -2,10 +2,8 @@
   "LiteBuildConfig": {
     "contents": ["."],
     "output_dir": "../build/docs-app",
-    "ignore_sys_prefix": ["federated_extensions"],
     "federated_extensions": [
-      "https://files.pythonhosted.org/packages/df/82/4576cbc07ebace8c7734fe08b2c2f9123b7ebecd29e932a3b839b6bee2cb/jupyterlab_widgets-3.0.5-py3-none-any.whl",
-      "../jupyterlite_pyodide_kernel/labextension"
+      "https://files.pythonhosted.org/packages/df/82/4576cbc07ebace8c7734fe08b2c2f9123b7ebecd29e932a3b839b6bee2cb/jupyterlab_widgets-3.0.5-py3-none-any.whl"
     ],
     "cache_dir": "../build/.lite-cache"
   },

nx.json

@@ -19,17 +19,15 @@
     "build": {
       "dependsOn": ["build:py", "^build"],
       "outputs": [
-        "{projectRoot}/lib",
-        "{projectRoot}/jupyterlite_pyodide_kernel/labextension/package.json",
-        "{projectRoot}/jupyterlite_pyodide_kernel/labextension/static/pypi/all.json"
+        "{projectRoot}/src/jupyterlite_pyodide_kernel/labextension/package.json",
+        "{projectRoot}/src/jupyterlite_pyodide_kernel/labextension/static/pypi/all.json"
       ]
     },
     "build:prod": {
       "dependsOn": ["build:py", "^build:prod"],
       "outputs": [
-        "{projectRoot}/lib",
-        "{projectRoot}/jupyterlite_pyodide_kernel/labextension/package.json",
-        "{projectRoot}/jupyterlite_pyodide_kernel/labextension/static/pypi/all.json"
+        "{projectRoot}/src/jupyterlite_pyodide_kernel/labextension/package.json",
+        "{projectRoot}/src/jupyterlite_pyodide_kernel/labextension/static/pypi/all.json"
       ]
     }
   }

package.json

@@ -22,10 +22,12 @@
     "dist:npm": "lerna run --stream dist",
     "dist:pypi": "pyproject-build .",
     "dist": "jlpm dist:pypi && jlpm dist:npm",
-    "docs:lite": "cd examples && jupyter lite build",
+    "docs:lite": "jlpm docs:lite:build && jlpm docs:lite:archive && jlpm docs:lite:check",
+    "docs:lite:build": "cd examples && jupyter lite build",
+    "docs:lite:archive": "cd examples && jupyter lite archive",
     "docs:lite:check": "cd examples && jupyter lite check",
     "docs:sphinx": "sphinx-build -W -b html docs build/docs",
-    "docs": "jlpm docs:lite && jlpm docs:sphinx && jlpm docs:lite:check",
+    "docs": "jlpm docs:lite && jlpm docs:sphinx",
     "eslint:check": "eslint . --cache --ext .ts,.tsx",
     "eslint": "jlpm eslint:check --fix",
     "lint": "jlpm lint:js && jlpm lint:py",
@@ -35,8 +37,8 @@
     "lint:py": "jlpm lint:py:black && jlpm lint:py:ruff --fix-only",
     "lint:py:pip": "python -m pip check",
     "lint:py:check": "jlpm lint:py:pip && jlpm lint:py:black --check && jlpm lint:py:ruff",
-    "lint:py:black": "black scripts jupyterlite_pyodide_kernel packages/pyodide-kernel/py",
-    "lint:py:ruff": "ruff scripts jupyterlite_pyodide_kernel packages/pyodide-kernel/py",
+    "lint:py:black": "black scripts src packages/pyodide-kernel/py",
+    "lint:py:ruff": "ruff scripts src packages/pyodide-kernel/py",
     "prettier:base": "prettier \"**/*{.ts,.tsx,.js,.jsx,.css,.json,.md,.yml,.yaml}\"",
     "prettier:check": "jlpm prettier:base --check",
     "prettier": "jlpm prettier:base --write --list-different",

packages/pyodide-kernel-extension/package.json

@@ -31,15 +31,16 @@
     "schema/*.json"
   ],
   "scripts": {
-    "build": "jlpm build:lib && jlpm build:labextension:dev",
-    "build:prod": "jlpm build:lib && jlpm build:labextension",
+    "build": "jlpm build:lib && jlpm build:labextension:dev && jlpm dev",
+    "build:prod": "jlpm build:lib && jlpm build:labextension && jlpm dev",
     "build:labextension": "jupyter labextension build .",
     "build:labextension:dev": "jupyter labextension build --development True .",
     "build:lib": "tsc -b",
+    "dev": "cd ../../ && jupyter labextension develop --overwrite .",
     "dist": "cd ../../dist && npm pack ../packages/pyodide-kernel-extension",
     "clean": "jlpm clean:lib",
     "clean:lib": "rimraf lib tsconfig.tsbuildinfo",
-    "clean:labextension": "rimraf ../../jupyterlite_pyodide_kernel/labextension",
+    "clean:labextension": "rimraf ../../src/jupyterlite_pyodide_kernel/labextension",
     "clean:all": "jlpm clean:lib && jlpm clean:labextension",
     "docs": "typedoc src",
     "watch": "run-p watch:src watch:labextension",
@@ -63,13 +64,17 @@
   },
   "jupyterlab": {
     "extension": true,
-    "outputDir": "../../jupyterlite_pyodide_kernel/labextension",
+    "outputDir": "../../src/jupyterlite_pyodide_kernel/labextension",
     "webpackConfig": "webpack.config.js",
     "sharedPackages": {
       "@jupyterlite/kernel": {
         "bundled": false,
         "singleton": true
       },
+      "@jupyterlite/pyodide-kernel": {
+        "bundled": true,
+        "singleton": true
+      },
       "@jupyterlite/server": {
         "bundled": false,
         "singleton": true

pyproject.toml

@@ -67,8 +67,8 @@ lint = [
 ]
 
 test = [
-    "pytest-console-scripts",
     "pytest-cov",
+    "pytest-console-scripts",
     "pytest-html",
     "pytest-xdist",
     "pytest",
@@ -87,20 +87,17 @@ docs = [
     "libarchive-c"
 ]
 
-[tool.hatch.metadata]
-allow-direct-references = true
-
 [tool.hatch.version]
-path = "jupyterlite_pyodide_kernel/_version.py"
+path = "src/jupyterlite_pyodide_kernel/_version.py"
 
 [tool.hatch.build.targets.sdist]
-artifacts = ["jupyterlite_pyodide_kernel/labextension"]
+artifacts = ["src/jupyterlite_pyodide_kernel/labextension"]
 exclude = [
     ".github",
-    "/jupyterlite_pyodide_kernel/tests/fixtures/.pyodide"
+    "/src/jupyterlite_pyodide_kernel/tests/fixtures/.pyodide"
 ]
 include = [
-    "/jupyterlite_pyodide_kernel",
+    "/src/jupyterlite_pyodide_kernel",
     "/package.json",
     "/install.json",
     "/ts*.json",
@@ -109,11 +106,14 @@ include = [
 
 [tool.hatch.build.targets.wheel]
 include = [
-    "/jupyterlite_pyodide_kernel",
+    "/src/jupyterlite_pyodide_kernel",
 ]
 
+[tool.hatch.build.targets.wheel.sources]
+"src" = ""
+
 [tool.hatch.build.targets.wheel.shared-data]
-"jupyterlite_pyodide_kernel/labextension" = "share/jupyter/labextensions/@jupyterlite/pyodide-kernel-extension"
+"src/jupyterlite_pyodide_kernel/labextension" = "share/jupyter/labextensions/@jupyterlite/pyodide-kernel-extension"
 "install.json" = "share/jupyter/labextensions/@jupyterlite/pyodide-kernel-extension/install.json"
 
 [tool.pytest.ini_options]
@@ -126,19 +126,31 @@ addopts = [
     "--script-launch-mode=subprocess",
     # parallel
     "-n=auto",
-    # cov
+    # coverage scope
     "--cov=jupyterlite_pyodide_kernel",
-    "--cov-report=term-missing:skip-covered",
-    "--cov-report=html:build/reports/htmlcov",
+    # coverage options
     "--cov-branch",
-    "--cov-fail-under=83",
-    "--no-cov-on-fail",
+    "--cov-context=test",
+    # coverage reporting
+    "--cov-report=html:build/reports/htmlcov",
+    "--cov-report=term-missing:skip-covered",
+    # coverage threshold (local will be slightly be higher)
+    "--cov-fail-under=92",
     # html
     "--html=build/reports/pytest.html",
     "--self-contained-html",
 ]
-testpaths = [
-    "tests/",
+
+[tool.coverage.run]
+data_file = "build/reports/.coverage"
+
+[tool.coverage.html]
+show_contexts = true
+
+[tool.coverage.paths]
+jupyterlite_pyodide_kernel = [
+    "src/jupyterlite_pyodide_kernel",
+    "*/src/jupyterlite_pyodide_kernel"
 ]
 
 [tool.jupyter-releaser.options]

jupyterlite_pyodide_kernel/__init__.py src/jupyterlite_pyodide_kernel/__init__.py

@@ -0,0 +1,30 @@
+"""Use the demo site for a more extensive test."""
+import pytest
+import shutil
+import os
+from pathlib import Path
+
+from .conftest import HERE
+
+IN_TREE_EXAMPLES = HERE / "../../../examples"
+EXAMPLES = Path(os.environ.get("LITE_PYODIDE_KERNEL_DEMO", IN_TREE_EXAMPLES))
+
+if not EXAMPLES.exists():  # pragma: no cover
+    pytest.skip(
+        "not in a source checkout, skipping example test", allow_module_level=True
+    )
+
+
+def test_examples(script_runner, tmp_path):
+    """verity the demo site builds (if it available)"""
+    examples = tmp_path / EXAMPLES.name
+    shutil.copytree(EXAMPLES, examples)
+
+    build = script_runner.run("jupyter", "lite", "build", cwd=str(examples))
+    assert build.success
+
+    build = script_runner.run("jupyter", "lite", "archive", cwd=str(examples))
+    assert build.success
+
+    build = script_runner.run("jupyter", "lite", "check", cwd=str(examples))
+    assert build.success

jupyterlite_pyodide_kernel/_version.py src/jupyterlite_pyodide_kernel/_version.py

@@ -7,7 +7,7 @@
 from jupyterlite_pyodide_kernel.constants import PYODIDE_VERSION
 from .conftest import HERE
 
-PACKAGES = HERE / "../../packages"
+PACKAGES = HERE / "../../../packages"
 KERNEL_PKG = PACKAGES / "pyodide-kernel"
 KERNEL_PKG_JSON = KERNEL_PKG / "package.json"