helper to establish singularity container images out of stored docker images

[yarikoptic]

ATM e.g. in https://github.com/ReproNim/containers we created singularity images via establishing more or less dummy Singularity file to build singularity images on singularity hub. But with singularity hub having established various measures to prevent abuse, I will be switching to just create singularity images locally from docker images (e.g. via singularity pull docker://...). BUT if we add docker containers, we already have all needed data (layers etc) available under annex! I think it might be really nice to have a helper which we could datalad run to produce (and possibly re-produce) singularity containers from dockers we have in the repo (not from online hub which might change its content etc)

[yarikoptic]

may be @vsoch can readily point on how to do that ;)

[vsoch]

I would:

1. Add a Dockerfile to the repos that you want to build into one or more containers
2. Use two CI recipes, one to build on PR (test) and another to deploy on merge to master
3. Push to a Quay.io organization registry (maybe grab datalad / repronim orgs?) via a robot token.

It's exactly the same as what we worked on for tributors, you could copy paste the recipes fairly verbatim.

[yarikoptic]

the idea is to build singularity (not docker) images from docker layers we already have locally under git-annex control.

[vsoch]

Then just use the same logic here https://github.com/singularityhub/docker2singularity. If you don't want to use the Docker daemon to retrieve and dump layers, then instead of running docker export just put the layers there instead :)

[yarikoptic]

Right! we could probably just do what we do already -- recompose the docker image from stored layers and manifest -- and then just run docker2singularity! So conceptually a "two liner" ;)

[yarikoptic]

Probably we should introduce datalad containers-convert [--source NAME] NAME command. Unfortunately we do not record in .datalad/config what type of the container each is - all fields are expanded and the only ways are either through inspecting where image points to (directories for docker ATM and singularity image file for singularity) and/or analysis of cmdexec. On one hand - following "minimal metadata" approach - it is optimal, but it is somewhat suboptimal in this particular case looking forward if we start to support other types of containers.