publishing to PyPI #50
Conversation
psuszyns
force-pushed
the
publish_to_pypi
branch
2 times, most recently
from
f2445d5 to
4579d8c
Compare
README.rst
Outdated
| >>> from pysequila import SequilaSession | ||
| >>> curr_dir = os.getcwd() | ||
| >>> ss = SequilaSession \ | ||
| .builder \ | ||
| .config("spark.jars.packages", "org.biodatageeks:sequila_2.12:1.1.0") \ |
There was a problem hiding this comment.
There are some errors when I try to use a newer version, this one is the newest that works for me as of now.
There was a problem hiding this comment.
Then pls create an issue - so that we keep track of it.
| requires = ["setuptools", "typeguard==2.9.1", "pyspark==3.2.2", "findspark", "pandas"] | ||
| requires = [ | ||
| "typeguard==2.9.1", | ||
| "pyspark==3.2.2", |
There was a problem hiding this comment.
Version is not in sync with SeQuiLa (latest release relies on Spark 3.4.x)- just wondering if we could add some constraints like >=3.4 <3.5 etc instead of hardcoding it here - semanting versioning is respected in the Spark community so any breaking changes shouldn't be introduced if only minor version changes.
There was a problem hiding this comment.
We can add such constraints in Poetry - I wanted to switch to Poetry in step 2, after verifying that everything works, with publishing, with setup.py build system.
I tried to update to Spark 3.4.4 or 3.4.3 and then use https://mvnrepository.com/artifact/org.biodatageeks/sequila_2.12/1.3.6 but this doesn't work. Before tracking down what is the reason for those failures I wanted to first publish a working version with pyspark==3.2.2 and sequila _2.12:1.1.0
There was a problem hiding this comment.
Again the same as previously - pls report that as an issue that upgrade to 3.4.x is not possible now
| steps: | ||
| - uses: actions/checkout@v4 | ||
| - name: Set up Python | ||
| uses: actions/setup-python@v4 |
There was a problem hiding this comment.
- I can' see any automatic tagging and semantic versioning steps?
- It would be also nice to have a PR-workflow that run some basic checks like black, isort, and unit tests
- pre-commit hook setup
- It's not setup to run on our self-hosted runners
- Would consider to use poetry instead (we try to use it in other projects as well) for the sake of lock file, cleaner way of managing dependencies and unification of development.
There was a problem hiding this comment.
I like to divide work to small deliverable parts. I wanted to first setup a basic publishing scheme, make sure the setup on the PyPI side is correct, and then add modifications one by one.
Regarding your individual points:
- This configuration will trigger a publish to PyPI when someone creates a new 'Release': https://github.com/biodatageeks/pysequila/releases/new - which also creates a tag in the repo. Another options would be to setup this in a way so that:
- it automatically publish when someone pushes a new tag with name starting with "v" on the master branch
- it automatically creates a new tag with 'next' version and automatically publish
From those options I think the last one is the worst, since we want control over which version number are we bumping. I think the first one (manually creating a release in github) is the best one. Releasing a new version should be a concious decision, proceeded by manual testing.
- Sure, in a next pull request.
- Yes, it would be nice to have some checks - but what exactly would we want? Also, that's definitely a separate task.
- fixed this
- I have a ready Poetry setup locally and wanted to push it after this PR
There was a problem hiding this comment.
Ok so please create follow-up issues that we won't forget about these points.
psuszyns
force-pushed
the
publish_to_pypi
branch
from
4579d8c to
6d20e5d
Compare
No description provided.