Skip to content

Latest commit

 

History

History
329 lines (264 loc) · 11.3 KB

Code_Generation_Guide.md

File metadata and controls

329 lines (264 loc) · 11.3 KB

Code Generation Guide

This guide explains the C++ code generated by the hpp-proto plugin of the Protocol Buffer compiler for any given .proto schema file.

Compiler Invocation

Compiler Invocation from CMake

Once the hpp-proto package has been added via find_package or FetchContent, the Protocol Buffer compiler can be invoked using the protobuf_generate_hpp function in the following format:

protobuf_generate_hpp(
    TARGET <TargetName> 
    [IMPORT_DIRS <dirs>]
    [PROTOC_OUT_DIR <output_dir>]
    [PROTOS <protobuf_files>]
    [PLUGIN_OPTIONS <plugin_options>])
  • IMPORT_DIRS: Specifies common parent directories for schema files.
  • PROTOC_OUT_DIR: Output directory for generated source files. Defaults to CMAKE_CURRENT_BINARY_DIR.
  • PROTOS: List of proto schema files. If omitted, then every source file ending in proto of TARGET will be used.
  • PLUGIN_OPTIONS: A comma-separated string forwarded to the protoc-gen-hpp plugin to customize code generation. Options include:
    • namespace_prefix=: Specifies a namespace prefix to be added to the generated C++ code in addition to the package namespace. If the specified prefix contains multiple components, they should be separated by a dot (.) instead of double colons (::).
    • directory_prefix=: Prepends a directory to included non-system dependencies.
    • proto2_explicit_presence=: For Proto2 only, makes optional scalar fields implicitly present except for specified scopes. This option can be specified multiple times. For example: the option proto2_explicit_presence=.pkg1.msg1.field1,proto2_explicit_presence=.pkg1.msg2 instructs the code generator that explicit presence is only applicable for the field1 of pkg1.msg1 and all fields of pkg1.msg2.
    • non_owning: Generates non-owning messages.
    • string_keyed_map: Uses an alternative type template, such as std.unordered_map, for generated string-keyed maps. Note: The namespace should be separated by a dot (.) instead of double colons (::) due to the protoc option parsing mechanism. This option applies only to regular mode map fields.
    • numeric_keyed_map: Uses an alternative type template, such as 'std.map', for generated numeric keyed map. Note: The namespace should be separated by a dot(.) instead of double colons (::) due to protoc option parsing mechanism. This option applies only to regular mode map fields.

The compiler generates several header files for each .proto file, transforming the filename and extension as follows:

  • The .proto extension is replaced with .msg.hpp, .pb.hpp, .glz.hpp and .desc.hpp for different header file types.
  • The proto path (specified with --proto_path= or -I flags) is replaced with the output path (specified with the --hpp_out flag).

Example CMake configuration:

add_library(non_owning_unittest_proto3_proto_lib INTERFACE)
target_include_directories(non_owning_unittest_proto3_proto_lib INTERFACE ${CMAKE_CURRENT_BINARY_DIR})
protobuf_generate_hpp(
    TARGET non_owning_unittest_proto3_proto_lib
    IMPORT_DIRS ${CMAKE_CURRENT_SOURCE_DIR}
    PROTOC_OUT_DIR ${CMAKE_CURRENT_BINARY_DIR}
    PROTOS ${CMAKE_CURRENT_SOURCE_DIR}/google/protobuf/unittest_proto3.proto
    PLUGIN_OPTIONS non_owning,namespace_prefix=non_owning,directory_prefix=non_owning)

Compiler Invocation from command line

You can invoke the hpp-proto plugin from the command line with the following command:

protoc --hpp_out=<output_dir> [--hpp_opt=<plugin_option>] [--proto_path=<dir>] <protobuf_files...>

Packages

If a .proto file contains a package declaration, the entire contents are placed in a corresponding C++ namespace. For example:

package foo.bar;

All declarations in the file will reside in the foo::bar namespace.

If you need to use both hpp-proto and the official Google Protobuf library in the same program, you can use the namespace_prefix option to customize the top-level namespace in the generated files. For example:

protoc --proto_path=src --hpp_out build/gen  --hpp_opt=namespace_prefix=baz src/foo.proto

This command will place all declarations in the baz::foo::bar namespace.

Non-owning mode only for specific files, messages or fields.

In addition to using the protoc plugin option to enable non-owning mode globally, the hpp_options extension can be used to apply non-owning mode selectively to specific files, messages, or fields. The file hpp_options_test.proto provides an example of how to mix regular and non-owning mode code generation within a single proto file.

Messages

Given a simple message declaration like:

message Foo {}

The protocol buffer compiler generates a struct called Foo.

Fields

For each field defined in a .proto message, the compiler generates corresponding member variables in the C++ struct.

Implicit Presence Fields (proto3)

syntax = "proto3"; 
message Foo {
    int32  f1 = 1;
    string f2 = 2;
    bytes  f3 = 3;
}

The compiler will generate the following struct:

Regular Mode 

struct Foo {
    int32_t f1;
    std::string f2;
    std::vector<std::byte> f3;
};

Non-owning Mode 

struct Foo {
    int32_t f1;
    std::string f2;
    hpp::proto::equality_comparable_span<const std::byte> f3;
};

The hpp::proto::equality_comparable_span is a template class which inherits std::span and adds an additional equality comparison operator.

Explicit Presence Fields (proto2)

For a Proto2 message with optional fields:

syntax = "proto2"; 
message Foo {
    optional int32  f1 = 1;
    optional string f2 = 2;
    optional bytes  f3 = 3;
    optional int64  f4 = 4 [default = 100];
}

The compiler will generate the following struct:

Regular Mode 

struct Foo {
    hpp::proto::optional<int32_t> f1;
    hpp::proto::optional<std::string> f2;
    hpp::proto::optional<std::vector<std::byte>> f3;
    hpp::proto::optional<int64_t, 100> f4;
};

Non-owning Mode 

struct Foo {
    hpp::proto::optional<int32_t> f1;
    hpp::proto::optional<std::string_view> f2;
    hpp::proto::optional<hpp::proto::equality_comparable_span<const std::byte>> f3;
    hpp::proto::optional<int64_t, 100> f4;
};

The hpp::proto::optional<T, DefaultValue> type has the same interface with std::optional<T> except the value() and operator*() member functions return the default value if the field is not present. Furthermore, the specialization for hpp::proto::optional<bool> deletes type conversion to bool to avoid ambiguity between a missing value and a false value.

Optional Embedded Message Fields (proto2 and proto3)

Given Bar is a message, any of these field definitions like:

//proto2
optional Bar foo = 1;

//proto3
Bar foo = 1;

The compiler will generate:

std::optional<Bar> foo;

Repeated Fields

For repeated fields, the compiler generates std::vector<T> in regular mode and hpp::proto::equality_comparable_span<const T> in non-owning mode.

Oneof fields

For oneof fields:

message TestOneof {
  oneof foo {
    int32 foo_int = 1;
    string foo_string = 2;
    NestedMessage foo_message = 3;
  }
  message NestedMessage {
    double required_double = 1;
  }
}

The compiler maps the oneof field to std::variant, with the first alternative being std::monostate:

struct TestOneof {
  struct NestedMessage {
    double required_double = {};

    bool operator == (const NestedMessage&) const = default;
  };

  enum foo_oneof_case : int {
    foo_int = 1,
    foo_string = 2,
    foo_message = 3
  };

  std::variant<std::monostate, int32_t, std::string, NestedMessage> foo;

  bool operator == (const TestOneof&) const = default;
};

Map fields

For map fields:

import "hpp_proto/hpp_options.proto";

message TestMap {
  map<int32, int32> map1 = 1;
  map<string, int32> map2 = 2 [(hpp.proto.hpp_field_opts).string_keyed_map = 'std::unordered_map'];
  map<int32, int32> map3 = 3 [(hpp.proto.hpp_field_opts).numeric_keyed_map = 'std::map'];;
}

By default, in regular mode, the compiler generates hpp::proto::flat_map<key_type, mapped_type> for map fields. Alternatively, a template class conforming to the std::map interface can be used by specifying either string_keyed_map or numeric_keyed_map in the protoc plugin options globally, or by annotating a specific file, message, or field with the hpp_options extension.

In non-owning mode, the compiler always generates hpp::proto::equality_comparable_span<std::pair<key_type, mapped_type>> for map fields. Additionally, the library does not handle key deduplication during serialization or deserialization. If a deserialized message contains duplicate map keys, users must ensure that only the last occurrence of each key is considered valid.

Regular Mode 

struct TestMap {
  hpp::proto::flat_map<int32_t,int32_t> map1;
  std::unordered_map<std::string, int32_t> map2;
  std::map<int32_t, int32_t> map3;
  bool operator == (const TestMap&) const = default;
};

Non-owning Mode 

struct TestMap {
  hpp::proto::equality_comparable_span<std::pair<int32_t,int32_t>> map1;
  hpp::proto::equality_comparable_span<std::pair<std::string,int32_t>> map2;
  hpp::proto::equality_comparable_span<std::pair<int32_t,int32_t>> map3;
  bool operator == (const TestMap&) const = default;
};

### Any Type For google.protobuf.Any fields: 
message TestAny {
  google.protobuf.Any any_value = 2;
}

The compiler generates ::google::protobuf::Any.

struct TestAny {
  std::optional<::google::protobuf::Any> any_value;
  bool operator == (const TestAny&) const = default;
};

To read or write Any fields, use hpp::proto::unpack_any and hpp::proto::pack_any.

Regular Mode 

    TestAny message;
    using namespace std::string_literals;
    google::protobuf::FieldMask fm{.paths = {"/usr/share"s, "/usr/local/share"s}};
    assert(hpp::proto::pack_any(message.any_value.emplace(), fm).ok());

    std::vector<char> buf;
    assert(hpp::proto::write_proto(message, buf).ok());

    TestAny message2;
    assert(hpp::proto::read_proto(message2, buf).ok());
    google::protobuf::FieldMask fm2;
    assert(hpp::proto::unpack_any(message2.any_value.value(), fm2).ok());

Non-owning Mode 

    TestAny message;
    using namespace std::string_view_literals;
    std::array<std::string_view, 2> paths = {"/usr/share"sw, "/usr/local/share"sw};
    google::protobuf::FieldMask fm;
    fm.paths = paths;
    std::pmr::monotonic_buffer_resource pool;
    hpp::proto::pb_context ctx{pool};

    assert(hpp::proto::pack_any(message.any_value.emplace(), fm, ctx).ok());

    std::pmr::vector<std::byte> buffer{&pool};
    assert(hpp::proto::write_proto(message, buffer).ok());

    TestAny message2;
    assert(hpp::proto::read_proto(message2, buf, ctx).ok());
    google::protobuf::FieldMask fm2;
    assert(hpp::proto::unpack_any(message2.any_value.value(), fm2, ctx).ok());