Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add B2Deps generator. #13592

Open
wants to merge 10 commits into
base: release/2.0
Choose a base branch
from
1 change: 1 addition & 0 deletions conan/tools/b2/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
from conan.tools.b2.b2deps import B2Deps
290 changes: 290 additions & 0 deletions conan/tools/b2/b2deps.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,290 @@
from conan.internal import check_duplicated_generator
from conan.tools.b2.util import *
from conans.errors import ConanException
from conans.util.files import save, chdir
from conans.paths import get_conan_user_home
from hashlib import md5


class B2Deps(object):
"""
B2Deps generates files that are automatically loaded by the B2 build system.
The files define localized subprojects and targets for dependencies.
"""

def __init__(self, conanfile):
self._conanfile = conanfile
self._conanhome = get_conan_user_home()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Access to the get_conan_user_home() sounds like a not valid solution. A generator shouldn't need the conan home location, that should be an abstraction for the generators.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's used to figure out host independent paths instead of the absolute paths that normally end up in generator files. That part of the path in the generated files is replaced with a Jam variable, $(CONAN_HOME), which is dynamically obtained when the consumer does a build. I admit what I'm doing is novel. But seems perfectly doable/reasonable within the context of the generator and install IMO.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am still not sure about this. Generators also must work the same when a package is in editable mode for example, so the package is not in the cache at all, not in the CONAN_HOME. The same should happen for "deployed" artifacts, like https://blog.conan.io/2023/05/23/Conan-agnostic-deploy-dependencies.html, where the generated files will be used without Conan, and should be relocatable.

It's used to figure out host independent paths instead of the absolute paths that normally end up in generator files.

This is another concern that should be a bit more orthogonal. I think it is better to have first the "simple" approach, in which paths are absolute paths to the cache, and work from there, based on the needs, than adding this that might have other unexpected rough edges

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good points on editable and deploy. I'll have to think about those, and do some testing.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's used to figure out host independent paths instead of the absolute paths that normally end up in generator files.

This is another concern that should be a bit more orthogonal. I think it is better to have first the "simple" approach, in which paths are absolute paths to the cache, and work from there, based on the needs, than adding this that might have other unexpected rough edges

Thing is where b2 deps puts the generated files it's expected for them to be stable and likely included in source control. As it's how the b2 conan integration works. Without that the automatic declaration of package targets just doesn't happen. Or, like the previous version of this, it means users needing to rerun the b2deps for each environment.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am still not sure about this. Generators also must work the same when a package is in editable mode for example, so the package is not in the cache at all, not in the CONAN_HOME. The same should happen for "deployed" artifacts, like https://blog.conan.io/2023/05/23/Conan-agnostic-deploy-dependencies.html, where the generated files will be used without Conan, and should be relocatable.

First.. There are multiple references to --deploy=full_deploy in that blog post which is incorrect. They should be --deployer=full_deploy (it confused me for a few minutes when I copy pasted commands and they errored.).

Second.. Both editable and full_deploy work just fine as is with this b2deps generator. The reason being that the use of conanhome only happens for packages that are installed to the home cache. For anything else the generated conanbuildinfo-*.jam files contain absolute paths to the packages (source path in editable or output/full_deploy). The reason being that it's a simple path replacement:

f'<search>"{b2_path(d.replace(self._conanhome, "$(CONAN_HOME)"))}"' for d in cpp_info.libdirs+cpp_info.bindirs
f' <include>"{b2_path(d.replace(self._conanhome, "$(CONAN_HOME)"))}"' for d in cpp_info.includedirs]

One thing that I could do is to make the full_deploy paths be relative to the conanbuildinfo*.jam generated files instead of absolute paths. That way they are fully relocatable. But as you said, it's better to start "simple" :-) And that's something I can work out later since I suspect full_deploy is a less common feature.

TLDR; It works as is.

Copy link
Member

@memsharded memsharded Jun 13, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thing is where b2 deps puts the generated files it's expected for them to be stable and likely included in source control.

I am afraid this is not the design of any Conan generators. They are not expected to be stable, but they are located in most cases inside the "build" folder (the generators folder inside build folder) and intended to be .gitignored and cleaned regularly. The reason for this is that these files can and will change for every single configuration change (architecture, shared, build_type), and not absolutely every binary variant can be captured in a file path of a generator.

I still think that this must be simplified and follow the rest of the generators design, allowing absolute paths in the generated files and managing them as non stable and not in version control.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am afraid this is not the design of any Conan generators. They are not expected to be stable, but they are located in most cases inside the "build" folder (the generators folder inside build folder) and intended to be .gitignored and cleaned regularly. The reason for this is that these files can and will change for every single configuration change (architecture, shared, build_type), and not absolutely every binary variant can be captured in a file path of a generator.

I still think that this must be simplified and follow the rest of the generators design, allowing absolute paths in the generated files and managing them as non stable and not in version control.

I guess there are a couple of different things in that..

  1. Stability of the generated files, in both names and content.
  2. Absolute or "relative" paths in the generated files.
  3. In version control.
  4. Where they are generated.

Each of those is separable from the others.

(1) I'm perfectly fine not calling them stable. After all it's just a label. They might or might not be stable.

(2) I can certainly do that. And then they are almost certainly not going to be stable.

(3) Sure, I don't have to say they can go in version control. Users might still put them there regardless though.

(4) I can't change this. If they are generated some place other than as a sibling to the conanfile.txt/py they will not. Specifically the B2 package manager integration will not load them. The declaration of the sub-targets will not work. Users would be forced to instruct/define/setup some manual way to define that "build/generator" folder to b2 instead of the automatic integration. Which goes very much against the b2 goals of removing complexity.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess I could abandon the idea of the B2Deps generator entirely and create something to read the conan packages directly, and entirely in b2. Which happens to be the way a pending b2 PR to support vcpkg integration works ATM.


def generate(self):
"""
This method will save the generated files to the conanfile.source_folder
"""
source_folder = self._conanfile.source_folder
self._conanfile.output.highlight(f"Writing B2Deps to {source_folder}")
with chdir(source_folder):
check_duplicated_generator(self, self._conanfile)
generator_files = self.content
for generator_file, content in generator_files.items():
self._conanfile.output.info(f"Saved B2Deps file {generator_file}")
save(generator_file, content)

@property
def content(self):
"""
Generates two content files: conanbuildinfo.jam and
conanbuildinfo-ID.jam. The former defines common package definition
function and include the latter conanbuildinfo-ID.jam files. The
conanbuildinfo-ID.jam files define sub-projects and targets for each
settings variation. The generated files have stable names and content.
Hence they can be added to source control.
"""
self._content = {}
self._content_conanbuildinfo_jam()
self._content_conanbuildinfo_variation_jam()
for ck in self._content.keys():
self._content[ck] = self._conanbuildinfo_header_text+"\n"+self._content[ck]
return self._content

def _content_conanbuildinfo_jam(self):
# Generate the common conanbuildinfo.jam which does four things:
#
# Defines common utility functions to make the rest of the code short
# and includes the conanbuildinfo-*.jam sub-files.
cbi = [self._conanbuildinfo_common_text]
# The combined text.
self._content['conanbuildinfo.jam'] = "\n".join(cbi)

def _content_conanbuildinfo_variation_jam(self):
# Generate the current build variation conanbuildinfo-/variation/.jam.
for require, dependency in self._conanfile.dependencies.items():
# Only generate defs for direct dependencies.
if not require.direct:
continue
# The base name of the dependency.
dep_name = dependency.ref.name
# B2 equivalent of the dependency name. We keep all names lower case.
dep_name_b2 = dep_name.lower()
# The dependency cpp_info. We need to consider that there's a
# "_depname" component. Such components are a kludge to appease
# cmake generators and will eventually go away. This special
# component holds the real root definitions of the dependency.
dep_cpp_info = dependency.cpp_info
if '_'+dep_name in dep_cpp_info.components:
dep_cpp_info = dep_cpp_info.components['_'+dep_name]
# The settings and options the dependency requires, i.e. finds relevant.
variant_settings = self._conanfile.settings
variant_options = dependency.options
# The variant specific file to add this dependency to.
dep_variant_jam = B2Deps._conanbuildinfo_variation_jam(
dep_name_b2, variant_settings, variant_options)
if not dep_variant_jam in self._content:
self._content[dep_variant_jam] = ""
# Declare/define the local project for the dependency.
cbiv = [
'#|',
f'{dependency.pref}',
'[settings]',
variant_settings.dumps(),
'[options]',
variant_options.dumps(),
'|#',
f'pkg-project {dep_name_b2} ;']
# Declare any system libs that we refer to (in usage requirements).
system_libs = set(dependency.cpp_info.system_libs)
for name, component in dependency.cpp_info.get_sorted_components().items():
system_libs |= set(component.system_libs)
cbiv += self._content_conanbuildinfo_variation_declare_syslibs(
dep_name_b2, system_libs, settings=variant_settings, options=variant_options)
# Declare any package libs for usage requirements. The first one is
# the main/global dependency.
cbiv += self._content_conanbuildinfo_variation_declare_libs(
dep_name_b2, dep_cpp_info, settings=variant_settings, options=variant_options)
# Followed by any components of the dependency. But skipping the
# special _depname component. As that is already declare as the
# main/global lib.
for name, component in dependency.cpp_info.get_sorted_components().items():
if name.lower() == '_'+dep_name_b2:
continue
cbiv += self._content_conanbuildinfo_variation_declare_libs(
dep_name_b2, component, settings=variant_settings, options=variant_options)
# Declare the main target of the dependency. This is an alias that
# refers to all the previous targets and adds all the defines,
# flags, etc for consumers.
cbiv += self._content_conanbuildinfo_variation_declare_target(
dep_name_b2, dep_name_b2,
dep_cpp_info,
settings=variant_settings, options=variant_options)
# Similarly declare the component targets, if any.
for name, component in dependency.cpp_info.get_sorted_components().items():
# Again, always skipping the kludge component as it's already
# defined.
if "_"+dep_name_b2 == name.lower():
continue
cbiv += self._content_conanbuildinfo_variation_declare_target(
dep_name_b2, name.lower(), component, settings=variant_settings, options=variant_options)
# Add the combined text.
self._content[dep_variant_jam] += "\n".join(cbiv)+"\n"

def _content_conanbuildinfo_variation_declare_libs(self, name, cpp_info, settings=None, options=None):
name = name.lower()
cbi_libs = []
variation = ' '.join(b2_features(b2_variation(settings, options)))
for lib in cpp_info.libs:
search = ' '.join(
[f'<search>"{b2_path(d.replace(self._conanhome, "$(CONAN_HOME)"))}"' for d in cpp_info.libdirs+cpp_info.bindirs])
# The lib targets are prefixed with "lib." to distinguish them
# from dependency main targets as it's often the case that the
# dependency has the same name as the library consumers link to.
cbi_libs += [
f'pkg-lib {name}//lib.{lib} : : <name>{lib}',
f' {variation}',
f' {search} ;']
return cbi_libs

def _content_conanbuildinfo_variation_declare_syslibs(self, name, systemlibs, settings=None, options=None):
name = name.lower()
cbi_libs = []
variation = ' '.join(b2_features(b2_variation(settings, options)))
for lib in systemlibs:
# Although system libs won't collide in the names. We still prefix
# the target names with "lib." for consistency and easier reference
# in the main targets.
cbi_libs += [
f'pkg-lib {name}//lib.{lib} : : <name>{lib}',
f' {variation} ;']
return cbi_libs

def _content_conanbuildinfo_variation_declare_target(self, name, target, cpp_info, settings=None, options=None):
cbi_target = []
# Target, no sources. The empty target is to catch incompatible build
# requirements matches by falling back to an unbuildable result.
cbi_target += [
f'pkg-alias {name}//{target} : : <build>no ;',
f'pkg-alias {name}//{target} : :']
# Requirements:
cbi_target += [
f' {" ".join(b2_features(b2_variation(settings, options)))}']
cbi_target += [
f' <source>lib.{l}' for l in cpp_info.libs+cpp_info.system_libs]
# No default-build:
cbi_target += [" : :"]
# Usage-requirements:
cbi_target += [
f' <include>"{b2_path(d.replace(self._conanhome, "$(CONAN_HOME)"))}"' for d in cpp_info.includedirs]
cbi_target += [f' <define>"{d}"' for d in cpp_info.defines]
cbi_target += [f' <cflags>"{f}"' for f in cpp_info.cflags]
cbi_target += [f' <cxxflags>"{f}"' for f in cpp_info.cxxflags]
cbi_target += [
f' <main-target-type>SHARED_LIB:<linkflags>"{f}"' for f in cpp_info.sharedlinkflags]
cbi_target += [f' <main-target-type>EXE:<linkflags>"{f}"' for f in cpp_info.exelinkflags]
cbi_target += [" ;"]
return cbi_target

@staticmethod
def _conanbuildinfo_variation_jam(name, settings, options=None):
return 'conanbuildinfo-{}-{}.jam'.format(
name, b2_variation_key(settings, options))

_conanbuildinfo_header_text = """\
#|
B2 definitions for Conan packages. This is a generated file.
Edit the corresponding conanfile.txt/py instead.
|#
"""

_conanbuildinfo_common_text = """\
import path ;
import project ;
import modules ;
import feature ;
import os ;

rule pkg-project ( id )
{
local id-mod = [ project.find $(id:L) : . ] ;
if ! $(id-mod)
{
local parent-prj = [ project.current ] ;
local parent-mod = [ $(parent-prj).project-module ] ;
local id-location = [ path.join
[ project.attribute $(parent-mod) location ]
$(id:L) ] ;
id-mod = [ project.load $(id-location) : synthesize ] ;
project.push-current [ project.current ] ;
project.initialize $(id-mod) : $(id-location) ;
project.pop-current ;
project.inherit-attributes $(id-mod) : $(parent-mod) ;
local attributes = [ project.attributes $(id-mod) ] ;
$(attributes).set parent-module : $(parent-mod) : exact ;
if [ project.is-jamroot-module $(parent-mod) ]
{
use-project /$(id:L) : $(id:L) ;
}
}
return $(id-mod) ;
}

rule pkg-target ( target : sources * : requirements * : default-build * : usage-requirements * )
{
target = [ MATCH "(.*)//(.*)" : $(target) ] ;
local id-mod = [ pkg-project $(target[1]) ] ;
project.push-current [ project.target $(id-mod) ] ;
local bt = [ BACKTRACE 1 ] ;
local rulename = [ MATCH "pkg-(.*)" : $(bt[4]) ] ;
modules.call-in $(id-mod) :
$(rulename) $(target[2]) : $(sources) : $(requirements) : $(default-build)
: $(usage-requirements) ;
project.pop-current ;
}

IMPORT $(__name__) : pkg-target : $(__name__) : pkg-alias ;
IMPORT $(__name__) : pkg-target : $(__name__) : pkg-lib ;

rule conan-home ( )
{
local conan_home = [ os.environ CONAN_HOME ] ;
if ! $(conan_home)
{
local conanrc = [ path.glob-in-parents [ path.join [ path.pwd ] "_" ] : ".conanrc" ] ;
if $(conanrc)
{
local conanrc_file = [ FILE_OPEN [ path.native $(conanrc) ] : t ] ;
conan_home = [ MATCH "^conan_home=(.*)
" ^conan_home=(.*) : $(conanrc_file) ] ;
conan_home = [ path.make $(conan_home[1]) ] ;
if [ MATCH ^(~/) : $(conan_home) ]
{
local home = [ os.home-directories ] ;
conan_home = [ path.join [ path.make $(home[1]) ] [ MATCH "^~/(.*)" : $(conan_home) ] ] ;
}
else if ! [ path.is-rooted $(conan_home) ]
{
conan_home = [ path.root $(conan_home) $(conanrc:D) ] ;
}
}
}
if ! $(conan_home)
{
local home = [ os.home-directories ] ;
conan_home = [ path.join [ path.make $(home[1]) ] .conan2 ] ;
}
return $(conan_home) ;
}

path-constant CONAN_HOME : [ conan-home ] ;

if ! ( relwithdebinfo in [ feature.values variant ] )
{
variant relwithdebinfo : : <optimization>speed <debug-symbols>on <inlining>full <runtime-debugging>off ;
}
if ! ( minsizerel in [ feature.values variant ] )
{
variant minsizerel : : <optimization>space <debug-symbols>off <inlining>full <runtime-debugging>off ;
}

for local __cbi__ in [ GLOB $(__file__:D) : conanbuildinfo-*.jam ]
{
include $(__cbi__) ;
}
"""
Loading