Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Workspace install with meta-project and monolithic build #17675

Open
wants to merge 16 commits into
base: develop2
Choose a base branch
from

Conversation

memsharded
Copy link
Member

@memsharded memsharded commented Jan 31, 2025

Changelog: Feature: Workspace new workspace install command for monolithic super-projects containing multiple editables.
Changelog: Feature: New conan new workspace template contains CMake-based monolithic super-project that works with conan workspace install.
Docs: https://github.com/conan-io/docs/pull/XXXX

Very early stage proof of concept based on FetchContent

Close #17665
Close #17723

@Artalus
Copy link

Artalus commented Feb 13, 2025

Quick question: is the conanfilews.py already set in stone?
We are ending up with a whole bunch of Conan files to describe the setup - projects' own conanfile.py, python_requires.py to deduplicate code between them, conanws.py to describe the workspace programmatically, conanws.yml with declarative data for it, and now this thing... Can its functionality not be offloaded to python_requires or conanws.py?

@Artalus
Copy link

Artalus commented Feb 13, 2025

Can we somehow filter (in a user-configurable way) what part of the workspace gets installed?

I checked out this PR's branch, and the conan new workspace example works alright - but only in its bare minimum setup. In our monorepo (and I believe in many others) there are OS-specific projects that f.x. work only on Windows. If I specify such project in conanws.yml and it happens to require a Windows-only dependency (like msys2 or 7z), conan workspace install fails.
I tried to raise ConanInvalidConfiguration() in recipe's def validate or def configure, remembering how CCI's c3i tends to ignore those instead of failing a CI run, but no luck.

One more point in favor of such filtering - a user might be not interested in the whole set of deps for all monorepo, f.x. if their team is only focused on a small subset of projects in there. On my previous job we had essentially a few conanfile.txts, one per OS, and devs would often complain that conan install would prepare a few gigs of packages when they only need to work on 1 project out of 50.

So maybe do something like conan workspace install app1, similar to conan workspace build app1?

@memsharded
Copy link
Member Author

Quick question: is the conanfilews.py already set in stone?

No, it is not, it is the very first iteration of an "incubating" feature. Things are very far from being set in stone.
The work at the moment is very focused on the functionality, not the UX. It is very important that the proposed workflow and tooling works, things like the ergonomics of files definitions, etc, is not even considered in this kind of first implementation of the concepts.

I like the idea of putting it inside the conanws.py file, I'll experiment with it.

@memsharded
Copy link
Member Author

I checked out this PR's branch, and the conan new workspace example works alright - but only in its bare minimum setup. In our monorepo (and I believe in many others) there are OS-specific projects that f.x. work only on Windows. If I specify such project in conanws.yml and it happens to require a Windows-only dependency (like msys2 or 7z), conan workspace install fails.
I tried to raise ConanInvalidConfiguration() in recipe's def validate or def configure, remembering how CCI's c3i tends to ignore those instead of failing a CI run, but no luck.

If the workspace will contain different folders as editables for different platforms, this sounds like a job for the conanws.py editables() method, that can implement the filtering you want based on the current OS, isn't it?

One more point in favor of such filtering - a user might be not interested in the whole set of deps for all monorepo, f.x. if their team is only focused on a small subset of projects in there. On my previous job we had essentially a few conanfile.txts, one per OS, and devs would often complain that conan install would prepare a few gigs of packages when they only need to work on 1 project out of 50.

So maybe do something like conan workspace install app1, similar to conan workspace build app1?

The individual package flow still works, it is always possible to do conan install app1 --build=editable or conan build app1 --build=editable. But that wouldn't be connected to the monolithic build, or did I misunderstood something.

@Artalus
Copy link

Artalus commented Feb 14, 2025

this sounds like a job for the conanws.py editables() method, that can implement the filtering you want based on the current OS, isn't it?

If this is exactly the intended use case for having conanws.py over a conanws.yml - awesome, will explore it further! I only started experimenting with WS recently, so not sure what goes where 🙂

The individual package flow still works, it is always possible to do conan install app1 --build=editable or conan build app1 --build=editable. But that wouldn't be connected to the monolithic build, or did I misunderstood something.

I am currently considering this feature from the perspective of #17549 .
conan workspace install w/ monolithic build fixes my issue (when liba has a private 3rdparty dependency). But reverting to plain conan install would gives me include propagation errors again...

@memsharded
Copy link
Member Author

If this is exactly the intended use case for having conanws.py over a conanws.yml - awesome, will explore it further! I only started experimenting with WS recently, so not sure what goes where

yes, exactly, the conanws.yml can have a "static" set of editables, but for dynamic definition, the editables() method of conanws.py would be the way to go. You can get some inspiration from tests:

class MyWorkspace(Workspace):

I am currently considering this feature from the perspective of #17549 .
conan workspace install w/ monolithic build fixes my issue (when liba has a private 3rdparty dependency). But reverting to plain conan install would gives me include propagation errors again...

That would be better managed in the other issue. But the plain conan install with editable is intended to work with independent subprojects, it is not valid to have your own meta-project that bypasses somehow the normal find_package() flow. If you are using independent sub-projects and still failing, please provide details in that ticket and we will follow up there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[question] Conan 2 workspace, and integration with IDEs Using FetchContent for Workspace meta-project
6 participants