Fails to use system-wide packages due to missing permissions

[ximion]

Hi!
I played around with the new system-wide dub packages (#838) feature recently, and unfortunately it doesn't seem to work right yet.
Dub tries to write into paths like /usr/include/d/test-1.0/test/.dub/, which of course fails because this global path is read-only.
Ideally, dub would use a different temporary location for storing the build artifacts. Ideally, this would be in the .dub directory of the currently-built package, so we avoid writing to $HOME, or some other configurable location.

In other news, I am working on integrating the Meson build system with dub, so Meson can publish packages usable by dub (easy, just install them as packages into a system path and have Meson generate a dub.json) and use packages that exclusively use dub as well.
For that, I have dub commands run as build targets by ninja. I use commands like this:
dub build test -b plain -v -c test-static --root build-dir
I would expect dub to build the testpackage that is installed in /usr/include/d/test-1.0/test/ and output its build artifacts to the build-dir directory, but that doesn't seem to be what happens here.
Instead I get:

Performing "plain" build using /usr/bin/ldc2 for x86_64.
Target '/usr/include/d/test-1.0/test/.dub/build/test-plain-linux.posix-x86_64-ldc2_2078-DAB625B69498E2F13B2B69D41EDFFDE0/libTest.a' doesn't exist, need rebuild.
/usr/include/d/test-1.0/test/.dub: Permission denied

It would be pretty cool to pass dub a global package path in /usr/include/d/<pkg>, have it build the package and publish the result in a custom directory as well. That would avoid the permission error, and also make it easy to integrate dub as part of other build systems, by having it take care of building selected dependencies.
Cheers,
Matthias

[grillo-delmal]

Any update on this issue? I tried to do something similar on the latest dub commit and got the same result.

[s-ludwig]

#1650 was supposed to solve the main issue, but unfortunately has become outdated and I need to find the time to adjust it properly.

[grillo-delmal]

I've been toying around with the --cache=* and --tempbuild flags to see how viable is to package dub libraries and software using the latest commit of dub.

• When building an app with the --tempbuild flag, the local dependencies and the built project end up in tmp... but for some reason, this doesn't happen with libs installed in a system path,
• I noticed that if I try to package just the source of a static library without any built binaries, the dependency build process fails because dub tries to store the result in the source folder and fails because it's a read-only path. Might be because the targetPath/targetName variables are set on the sdl file, but I need to properly test this before getting any propper conclusions,,.
  • The problem is solved by storing the resulting binary in the library package, but I don't think that this would be the correct way to package D libraries since you can have the case where 2 different apps or libraries depend on different flags on the same packaged library.
  • Unless you build package each library binary with all the permutations of configurations/version flags needed this wouldn't end up being viable, and even if you could, dub is not prepared to detect and distinguish binaries of the same package built with different options.

I want to build proper reproducible test cases for this things, but I will take my time doing that, for now I'll leave this here.

[Geod24]

Note that with the latest dub, you should not need to do anything to have it work. Dub will now build in ~/.dub/cache/some_unique_path/ by default.

[grillo-delmal]

I built a test suite to show what I'm talking about here: https://github.com/grillo-delmal/dub-system-wide-lib-tests. (didn't use neither --cache nor --tempbuild options on them considering Geod24's comment)

I ran the tests and got the following results: grillo-delmal/dub-system-wide-lib-tests#1

• Tests 0 and 2 work fine since those just use local libraries
• Tests 1, 3 and 4, fail because when dub tries to build the system libraries, even if they are bult inside the cache, dub tries to install the resulting binaries in the read only system folder.

Is this expected? I am understanding correctly how this should work?

[Geod24]

I'm sorry my most recent comment was misleading. I mean with dub master. With the latest dub (v1.30.0), this issue is not yet fixed. You can build dub by cloning this repository and running dub build, it will then be in bin/dub.

I had a quick look over the tests, but I think I need to sit down and have a proper look at it. I am hoping to do this next week, unless you can run them with dub master before that.

EDIT: Thanks for the detailed tests!

[grillo-delmal]

I'm sorry my most recent comment was misleading. I mean with dub master. With the latest dub (v1.30.0), this issue is not yet fixed. You can build dub by cloning this repository and running dub build, it will then be in bin/dub.

I know, my tests are running on the latest commit (1.30.0^53.git1330c9d7).

I could build dub inside the container so it's easier to understand, but I already had that version packaged on a COPR repository, which was faster for me to use.

[grillo-delmal]

I updated the container to download and build dub from the repo :), the results are the same

[grillo-delmal]

Today I learned how to run my podman based test suite on the github workflow thingy :D

I ran the tests on dub's latest commit (v1.31.0-beta.1-6-gdcbfcf24) and it still fails

[Geod24]

Great! I started to look into the issue yesterday. And with a clear head, your explanation makes it clear where the problem is. We probably shouldn't copy things for packages, just for the project. I was trying to refactor things so it's easier to express.

[grillo-delmal]

Did some experiments during this week on the latest version, and apps depending on system libs living in a read-only space compile if you pass the --temp-build parameter. The main problem is that you have to rescue the resulting files from the temporary folder, which is kinda a roundabout way to deal with that...

I still think that #2713 would be the ideal solution... but it's been stuck since last year...