v0.7.3

Small patch release for the gbasf2 process adding tests and better error checks for subprocess to make future debugging of problems like e.g. #138 easier

Added

• Check output of gb2_proxy_init for errors by @meliache in #142
  • if gb2_proxy_init fails due to a wrong certificate password, re-run the command until the user enters a correct password
  • raises a CalledProcessError when there is any other error string in the stdout of gb2_proxy_init. Since that script doesn't exit with errorcodes in case of errors, otherwise errors could go unnoticed and resulted errors in later commands, such as when using gb2_proxy_info. Tracking down which command originally failed might be some work, so this should make debugging much easiert.
• Don't subpress CalledProcessError in get_proxy_info
• Add unit tests for setup_dirac_proxy and get_proxy_info by mocking possible outputs of gb2_proxy_info and gb2_proxy_init.

Full Changelog: v0.7.2...v0.7.3

v0.7.2

Added

• Test ignore_additional_command_line_args option by @meliache in #128

Fixed

• Fix gb2_proxy_init error due to wrong HOME from gbasf2 setup script by @meliache in #141

Changed

• Remove unused Gbasf2Process helper method to capture failed files from stdout by @meliache in #136

Documentation

• Add Moritz to contributors list in documentation by @meliache in #133
• Add Artur Gottmann to contributors by @meliache in #137

Full Changelog: v0.7.1...v0.7.2

v0.7.1

Added

• Added inherits_without decorator to enable inheritance of everything except chosen parameters from task by @sognetic in #106
• tasks can return all input or all outputfiles with a single call by @anselm-baur in #111
• Allow for gbasf2 projects with multiple output sub<xy> directories by @meliache in #122

Fixed

• Bugfix: Don't use deprecated exception messsage property by @meliache in #120
• Fix gbasf2 batch example for new basf2 releases: import ROOT by @meliache in #123
• Ignore flake8 error for unused ROOT import by @meliache in #124

Changed

• Give instances of MasterTask in doc examples less sensitive names by @meliache in #118
• Replace parsing of gb2 command output with DIRAC API calls by @philiptgrace in #121
• Use latest gbasf2 release on cvmfs as default install directory by @meliache in #126
• Improve dirac proxy validity time handling by @philiptgrace in #127
• For gbasf2 download with gb2_ds_get, use new --failed_lfns option to get file with LFNs for which download failed instead of parsing stdout.
  by @ArturAkh in #132

New Contributors

• @sognetic made their first contribution in #106

Full Changelog: v0.6.7...v0.7.1

v0.6.7

Added

• Set progress bar in central scheduler for tasks executed as a gbasf2 project
  showing what percentage of jobs in the project is done and display the total numbers in the status.

Changed

• Fix gbasf2 download retry issues for new gbasf2 releases
• Fix in gbasf2 batch for memory error caused by changes in the ROOT from basf2 externals v10 (affects latest light releases).

v0.6.6

Added

• Improved gbasf2 download from grid. In particular, when re-trying a download, only re-download files which have previously failed. Store failed files in a failed_files.txt. The downside is that this relies on command output parsing which might break between releases. If errors occur, this can be worked around by removing the failed_files.txt, triggering a full re-download.

v0.6.5

#97 Gbasf2 Bugfix: Fix download for failed files

Keeping the releases up to date is nice, but in the future I should try to avoid multiple patch releases in a day as happened today. Maybe by waiting before publishing a tag as a release or using pre-releases.

v0.6.4

#95 Fix to show correct version number for b2luigi.__version__

Minor patch release but I decided to release this early so that users can use the version number to validate that they are using the latest release.

v0.6.3: Merge pull request #91 from ArturAkh/main

This release features small quality-of-life improvements and fixes for the gbasf2 batch, so I decide to make it a minor release.

Since we're still major release 0, instead of SemVer I think I will be creating minor releases for significant changes to luigi themselves and where all users should read the release notes and patch release for small patches that come out shortly after a release or when I do small non-api-breaking changes to individual batches only, which only affect users of that batch and don't really change b2luigi itself.

Gbasf2 Batch

Features

• #75 allow defining grid input LFNs via text files with gbasf2_input_dslist setting, analogous to gbasf2 --input_dslist
• #91: Several improvements of gbasf2 handling (thanks to @ArturAkh)
  • possibility to add input datafiles with gbasf2_input_datafiles option, which will be downloaded from SE's in addition. This is useful in case the sandbox files exceed 10 MB.
  • improved rescheduling: instead of performing it for each single failed job separately, perform it at once. Keeping track of n_retries is still maintained in the implementation of this pull request.
  • improved downloading of datasets: in case of failed downloads only the ones which are failed, are downloaded, based on a collection of LFNs from created from gb2_ds_get stdout.
• more unit tests for more stability in the future and getting a handle on growing complexity.

Fixes

• #91: Several improvements of gbasf2 handling
  • fix of RuntimeError ---> RuntimeWorking conversion: first argument of warnings.warn should be a string. Otherwise, getting a uncatched TypeError, followed by a PipeError of luigi.
  • added an improved handling of the JobStatus for Done jobs, since in some (rare) cases, JobStatus is set to Done, while ApplicationStatus is not Done (in particular, has an Upload error for output file).

Meta

• #92 Fix CI shield on github

v0.6.2

• #88 Bugfix gbasf2 dataset download where failed download raises runtime error instead of intended warning, thanks to @philiptgrace for finding and fixing this.

v0.6.1

Upps, release v0.6.0 was mistakenly missing two PR's, #79 and #81, since I added the tag at the HEAD of the last branch that I merged and that branch didn't contain those PR's yet. into that release (#79), but that branch wasn't rebased to the head of main and didn't contain the LSF bugfix PR #81 and the gbasf2 feature PR #77 for supporting global tags. So this patch release includes those PRs and also it includes a fix to our PyPi publishing workflow (#82).

Bugfixes:

• #81: Bugfix in LSF batch code for getting settings
• #82: Fix missing depency in github workflow for automatic publishing to PyPi

Features:

• #77: the global tags have been also added to the sub-set of the basf2-state that is pickled and send to the grid. Remember, the gbasf2 batch wrapper just pickle the basf2 path and sends this to grid, so everything that is saved in the basf2 state is not transferred. In the previous release we already added pickling the basf2 variable aliases separately, now the global tags have also been added. If you have ideas how to handle this more generally, feel free to contribute via issue #35