-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add B2Deps generator. #13592
Open
grafikrobot
wants to merge
10
commits into
conan-io:release/2.0
Choose a base branch
from
bfgroup:b2
base: release/2.0
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Add B2Deps generator. #13592
Changes from all commits
Commits
Show all changes
10 commits
Select commit
Hold shift + click to select a range
4ca09b6
Add B2Deps generator for integration with b2 build system.
grafikrobot e3f0536
Tweak doc strings for accuracy.
grafikrobot 02be636
Fix typo.
grafikrobot 97efbe6
Generate per-dependency variation jam files.
grafikrobot a3ed4f5
Add additional link relevant properties.
grafikrobot b518e01
Move variation functions to utils.py.
grafikrobot f3fd283
Fix for py36 compat.
grafikrobot 6806451
Fix typo.
grafikrobot 6fd4d19
Add dep info in generated files.
grafikrobot e8a6f78
Fix to use consumer settings for gened files.
grafikrobot File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
from conan.tools.b2.b2deps import B2Deps |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,290 @@ | ||
from conan.internal import check_duplicated_generator | ||
from conan.tools.b2.util import * | ||
from conans.errors import ConanException | ||
from conans.util.files import save, chdir | ||
from conans.paths import get_conan_user_home | ||
from hashlib import md5 | ||
|
||
|
||
class B2Deps(object): | ||
""" | ||
B2Deps generates files that are automatically loaded by the B2 build system. | ||
The files define localized subprojects and targets for dependencies. | ||
""" | ||
|
||
def __init__(self, conanfile): | ||
self._conanfile = conanfile | ||
self._conanhome = get_conan_user_home() | ||
|
||
def generate(self): | ||
""" | ||
This method will save the generated files to the conanfile.source_folder | ||
""" | ||
source_folder = self._conanfile.source_folder | ||
self._conanfile.output.highlight(f"Writing B2Deps to {source_folder}") | ||
with chdir(source_folder): | ||
check_duplicated_generator(self, self._conanfile) | ||
generator_files = self.content | ||
for generator_file, content in generator_files.items(): | ||
self._conanfile.output.info(f"Saved B2Deps file {generator_file}") | ||
save(generator_file, content) | ||
|
||
@property | ||
def content(self): | ||
""" | ||
Generates two content files: conanbuildinfo.jam and | ||
conanbuildinfo-ID.jam. The former defines common package definition | ||
function and include the latter conanbuildinfo-ID.jam files. The | ||
conanbuildinfo-ID.jam files define sub-projects and targets for each | ||
settings variation. The generated files have stable names and content. | ||
Hence they can be added to source control. | ||
""" | ||
self._content = {} | ||
self._content_conanbuildinfo_jam() | ||
self._content_conanbuildinfo_variation_jam() | ||
for ck in self._content.keys(): | ||
self._content[ck] = self._conanbuildinfo_header_text+"\n"+self._content[ck] | ||
return self._content | ||
|
||
def _content_conanbuildinfo_jam(self): | ||
# Generate the common conanbuildinfo.jam which does four things: | ||
# | ||
# Defines common utility functions to make the rest of the code short | ||
# and includes the conanbuildinfo-*.jam sub-files. | ||
cbi = [self._conanbuildinfo_common_text] | ||
# The combined text. | ||
self._content['conanbuildinfo.jam'] = "\n".join(cbi) | ||
|
||
def _content_conanbuildinfo_variation_jam(self): | ||
# Generate the current build variation conanbuildinfo-/variation/.jam. | ||
for require, dependency in self._conanfile.dependencies.items(): | ||
# Only generate defs for direct dependencies. | ||
if not require.direct: | ||
continue | ||
# The base name of the dependency. | ||
dep_name = dependency.ref.name | ||
# B2 equivalent of the dependency name. We keep all names lower case. | ||
dep_name_b2 = dep_name.lower() | ||
# The dependency cpp_info. We need to consider that there's a | ||
# "_depname" component. Such components are a kludge to appease | ||
# cmake generators and will eventually go away. This special | ||
# component holds the real root definitions of the dependency. | ||
dep_cpp_info = dependency.cpp_info | ||
if '_'+dep_name in dep_cpp_info.components: | ||
dep_cpp_info = dep_cpp_info.components['_'+dep_name] | ||
# The settings and options the dependency requires, i.e. finds relevant. | ||
variant_settings = self._conanfile.settings | ||
variant_options = dependency.options | ||
# The variant specific file to add this dependency to. | ||
dep_variant_jam = B2Deps._conanbuildinfo_variation_jam( | ||
dep_name_b2, variant_settings, variant_options) | ||
if not dep_variant_jam in self._content: | ||
self._content[dep_variant_jam] = "" | ||
# Declare/define the local project for the dependency. | ||
cbiv = [ | ||
'#|', | ||
f'{dependency.pref}', | ||
'[settings]', | ||
variant_settings.dumps(), | ||
'[options]', | ||
variant_options.dumps(), | ||
'|#', | ||
f'pkg-project {dep_name_b2} ;'] | ||
# Declare any system libs that we refer to (in usage requirements). | ||
system_libs = set(dependency.cpp_info.system_libs) | ||
for name, component in dependency.cpp_info.get_sorted_components().items(): | ||
system_libs |= set(component.system_libs) | ||
cbiv += self._content_conanbuildinfo_variation_declare_syslibs( | ||
dep_name_b2, system_libs, settings=variant_settings, options=variant_options) | ||
# Declare any package libs for usage requirements. The first one is | ||
# the main/global dependency. | ||
cbiv += self._content_conanbuildinfo_variation_declare_libs( | ||
dep_name_b2, dep_cpp_info, settings=variant_settings, options=variant_options) | ||
# Followed by any components of the dependency. But skipping the | ||
# special _depname component. As that is already declare as the | ||
# main/global lib. | ||
for name, component in dependency.cpp_info.get_sorted_components().items(): | ||
if name.lower() == '_'+dep_name_b2: | ||
continue | ||
cbiv += self._content_conanbuildinfo_variation_declare_libs( | ||
dep_name_b2, component, settings=variant_settings, options=variant_options) | ||
# Declare the main target of the dependency. This is an alias that | ||
# refers to all the previous targets and adds all the defines, | ||
# flags, etc for consumers. | ||
cbiv += self._content_conanbuildinfo_variation_declare_target( | ||
dep_name_b2, dep_name_b2, | ||
dep_cpp_info, | ||
settings=variant_settings, options=variant_options) | ||
# Similarly declare the component targets, if any. | ||
for name, component in dependency.cpp_info.get_sorted_components().items(): | ||
# Again, always skipping the kludge component as it's already | ||
# defined. | ||
if "_"+dep_name_b2 == name.lower(): | ||
continue | ||
cbiv += self._content_conanbuildinfo_variation_declare_target( | ||
dep_name_b2, name.lower(), component, settings=variant_settings, options=variant_options) | ||
# Add the combined text. | ||
self._content[dep_variant_jam] += "\n".join(cbiv)+"\n" | ||
|
||
def _content_conanbuildinfo_variation_declare_libs(self, name, cpp_info, settings=None, options=None): | ||
name = name.lower() | ||
cbi_libs = [] | ||
variation = ' '.join(b2_features(b2_variation(settings, options))) | ||
for lib in cpp_info.libs: | ||
search = ' '.join( | ||
[f'<search>"{b2_path(d.replace(self._conanhome, "$(CONAN_HOME)"))}"' for d in cpp_info.libdirs+cpp_info.bindirs]) | ||
# The lib targets are prefixed with "lib." to distinguish them | ||
# from dependency main targets as it's often the case that the | ||
# dependency has the same name as the library consumers link to. | ||
cbi_libs += [ | ||
f'pkg-lib {name}//lib.{lib} : : <name>{lib}', | ||
f' {variation}', | ||
f' {search} ;'] | ||
return cbi_libs | ||
|
||
def _content_conanbuildinfo_variation_declare_syslibs(self, name, systemlibs, settings=None, options=None): | ||
name = name.lower() | ||
cbi_libs = [] | ||
variation = ' '.join(b2_features(b2_variation(settings, options))) | ||
for lib in systemlibs: | ||
# Although system libs won't collide in the names. We still prefix | ||
# the target names with "lib." for consistency and easier reference | ||
# in the main targets. | ||
cbi_libs += [ | ||
f'pkg-lib {name}//lib.{lib} : : <name>{lib}', | ||
f' {variation} ;'] | ||
return cbi_libs | ||
|
||
def _content_conanbuildinfo_variation_declare_target(self, name, target, cpp_info, settings=None, options=None): | ||
cbi_target = [] | ||
# Target, no sources. The empty target is to catch incompatible build | ||
# requirements matches by falling back to an unbuildable result. | ||
cbi_target += [ | ||
f'pkg-alias {name}//{target} : : <build>no ;', | ||
f'pkg-alias {name}//{target} : :'] | ||
# Requirements: | ||
cbi_target += [ | ||
f' {" ".join(b2_features(b2_variation(settings, options)))}'] | ||
cbi_target += [ | ||
f' <source>lib.{l}' for l in cpp_info.libs+cpp_info.system_libs] | ||
# No default-build: | ||
cbi_target += [" : :"] | ||
# Usage-requirements: | ||
cbi_target += [ | ||
f' <include>"{b2_path(d.replace(self._conanhome, "$(CONAN_HOME)"))}"' for d in cpp_info.includedirs] | ||
grafikrobot marked this conversation as resolved.
Show resolved
Hide resolved
|
||
cbi_target += [f' <define>"{d}"' for d in cpp_info.defines] | ||
cbi_target += [f' <cflags>"{f}"' for f in cpp_info.cflags] | ||
cbi_target += [f' <cxxflags>"{f}"' for f in cpp_info.cxxflags] | ||
cbi_target += [ | ||
f' <main-target-type>SHARED_LIB:<linkflags>"{f}"' for f in cpp_info.sharedlinkflags] | ||
cbi_target += [f' <main-target-type>EXE:<linkflags>"{f}"' for f in cpp_info.exelinkflags] | ||
cbi_target += [" ;"] | ||
return cbi_target | ||
|
||
@staticmethod | ||
def _conanbuildinfo_variation_jam(name, settings, options=None): | ||
return 'conanbuildinfo-{}-{}.jam'.format( | ||
name, b2_variation_key(settings, options)) | ||
|
||
_conanbuildinfo_header_text = """\ | ||
#| | ||
B2 definitions for Conan packages. This is a generated file. | ||
Edit the corresponding conanfile.txt/py instead. | ||
|# | ||
""" | ||
|
||
_conanbuildinfo_common_text = """\ | ||
import path ; | ||
import project ; | ||
import modules ; | ||
import feature ; | ||
import os ; | ||
|
||
rule pkg-project ( id ) | ||
{ | ||
local id-mod = [ project.find $(id:L) : . ] ; | ||
if ! $(id-mod) | ||
{ | ||
local parent-prj = [ project.current ] ; | ||
local parent-mod = [ $(parent-prj).project-module ] ; | ||
local id-location = [ path.join | ||
[ project.attribute $(parent-mod) location ] | ||
$(id:L) ] ; | ||
id-mod = [ project.load $(id-location) : synthesize ] ; | ||
project.push-current [ project.current ] ; | ||
project.initialize $(id-mod) : $(id-location) ; | ||
project.pop-current ; | ||
project.inherit-attributes $(id-mod) : $(parent-mod) ; | ||
local attributes = [ project.attributes $(id-mod) ] ; | ||
$(attributes).set parent-module : $(parent-mod) : exact ; | ||
if [ project.is-jamroot-module $(parent-mod) ] | ||
{ | ||
use-project /$(id:L) : $(id:L) ; | ||
} | ||
} | ||
return $(id-mod) ; | ||
} | ||
|
||
rule pkg-target ( target : sources * : requirements * : default-build * : usage-requirements * ) | ||
{ | ||
target = [ MATCH "(.*)//(.*)" : $(target) ] ; | ||
local id-mod = [ pkg-project $(target[1]) ] ; | ||
project.push-current [ project.target $(id-mod) ] ; | ||
local bt = [ BACKTRACE 1 ] ; | ||
local rulename = [ MATCH "pkg-(.*)" : $(bt[4]) ] ; | ||
modules.call-in $(id-mod) : | ||
$(rulename) $(target[2]) : $(sources) : $(requirements) : $(default-build) | ||
: $(usage-requirements) ; | ||
project.pop-current ; | ||
} | ||
|
||
IMPORT $(__name__) : pkg-target : $(__name__) : pkg-alias ; | ||
IMPORT $(__name__) : pkg-target : $(__name__) : pkg-lib ; | ||
|
||
rule conan-home ( ) | ||
{ | ||
local conan_home = [ os.environ CONAN_HOME ] ; | ||
if ! $(conan_home) | ||
{ | ||
local conanrc = [ path.glob-in-parents [ path.join [ path.pwd ] "_" ] : ".conanrc" ] ; | ||
if $(conanrc) | ||
{ | ||
local conanrc_file = [ FILE_OPEN [ path.native $(conanrc) ] : t ] ; | ||
conan_home = [ MATCH "^conan_home=(.*) | ||
" ^conan_home=(.*) : $(conanrc_file) ] ; | ||
conan_home = [ path.make $(conan_home[1]) ] ; | ||
if [ MATCH ^(~/) : $(conan_home) ] | ||
{ | ||
local home = [ os.home-directories ] ; | ||
conan_home = [ path.join [ path.make $(home[1]) ] [ MATCH "^~/(.*)" : $(conan_home) ] ] ; | ||
} | ||
else if ! [ path.is-rooted $(conan_home) ] | ||
{ | ||
conan_home = [ path.root $(conan_home) $(conanrc:D) ] ; | ||
} | ||
} | ||
} | ||
if ! $(conan_home) | ||
{ | ||
local home = [ os.home-directories ] ; | ||
conan_home = [ path.join [ path.make $(home[1]) ] .conan2 ] ; | ||
} | ||
return $(conan_home) ; | ||
} | ||
|
||
path-constant CONAN_HOME : [ conan-home ] ; | ||
|
||
if ! ( relwithdebinfo in [ feature.values variant ] ) | ||
{ | ||
variant relwithdebinfo : : <optimization>speed <debug-symbols>on <inlining>full <runtime-debugging>off ; | ||
} | ||
if ! ( minsizerel in [ feature.values variant ] ) | ||
{ | ||
variant minsizerel : : <optimization>space <debug-symbols>off <inlining>full <runtime-debugging>off ; | ||
} | ||
|
||
for local __cbi__ in [ GLOB $(__file__:D) : conanbuildinfo-*.jam ] | ||
{ | ||
include $(__cbi__) ; | ||
} | ||
""" |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Access to the
get_conan_user_home()
sounds like a not valid solution. A generator shouldn't need the conan home location, that should be an abstraction for the generators.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's used to figure out host independent paths instead of the absolute paths that normally end up in generator files. That part of the path in the generated files is replaced with a Jam variable,
$(CONAN_HOME)
, which is dynamically obtained when the consumer does a build. I admit what I'm doing is novel. But seems perfectly doable/reasonable within the context of the generator and install IMO.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am still not sure about this. Generators also must work the same when a package is in
editable
mode for example, so the package is not in the cache at all, not in the CONAN_HOME. The same should happen for "deployed" artifacts, like https://blog.conan.io/2023/05/23/Conan-agnostic-deploy-dependencies.html, where the generated files will be used without Conan, and should be relocatable.This is another concern that should be a bit more orthogonal. I think it is better to have first the "simple" approach, in which paths are absolute paths to the cache, and work from there, based on the needs, than adding this that might have other unexpected rough edges
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good points on editable and deploy. I'll have to think about those, and do some testing.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thing is where b2 deps puts the generated files it's expected for them to be stable and likely included in source control. As it's how the b2 conan integration works. Without that the automatic declaration of package targets just doesn't happen. Or, like the previous version of this, it means users needing to rerun the b2deps for each environment.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
First.. There are multiple references to
--deploy=full_deploy
in that blog post which is incorrect. They should be--deployer=full_deploy
(it confused me for a few minutes when I copy pasted commands and they errored.).Second.. Both editable and full_deploy work just fine as is with this b2deps generator. The reason being that the use of conanhome only happens for packages that are installed to the home cache. For anything else the generated conanbuildinfo-*.jam files contain absolute paths to the packages (source path in editable or output/full_deploy). The reason being that it's a simple path replacement:
One thing that I could do is to make the full_deploy paths be relative to the conanbuildinfo*.jam generated files instead of absolute paths. That way they are fully relocatable. But as you said, it's better to start "simple" :-) And that's something I can work out later since I suspect full_deploy is a less common feature.
TLDR; It works as is.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am afraid this is not the design of any Conan generators. They are not expected to be stable, but they are located in most cases inside the "build" folder (the
generators
folder insidebuild
folder) and intended to be.gitignored
and cleaned regularly. The reason for this is that these files can and will change for every single configuration change (architecture, shared, build_type), and not absolutely every binary variant can be captured in a file path of a generator.I still think that this must be simplified and follow the rest of the generators design, allowing absolute paths in the generated files and managing them as non stable and not in version control.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I guess there are a couple of different things in that..
Each of those is separable from the others.
(1) I'm perfectly fine not calling them stable. After all it's just a label. They might or might not be stable.
(2) I can certainly do that. And then they are almost certainly not going to be stable.
(3) Sure, I don't have to say they can go in version control. Users might still put them there regardless though.
(4) I can't change this. If they are generated some place other than as a sibling to the conanfile.txt/py they will not. Specifically the B2 package manager integration will not load them. The declaration of the sub-targets will not work. Users would be forced to instruct/define/setup some manual way to define that "build/generator" folder to b2 instead of the automatic integration. Which goes very much against the b2 goals of removing complexity.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I guess I could abandon the idea of the B2Deps generator entirely and create something to read the conan packages directly, and entirely in b2. Which happens to be the way a pending b2 PR to support vcpkg integration works ATM.