v24.2.18

Bug Fixes

• #23363 #25028 rpk: fixes a bug where rpk incorrectly handles IPv6 by adding extra brackets. by @r-vasquez in #25067
• #25008 Schema Registry/Protobuf: Fix a regression with maps. by @BenPope in #25010

Improvements

• #25081 Use streaming parsing of transaction range manifests in tiered storage to avoid large allocations which lead to OOMs with heavy use of transactions. by @nvartolomei in #25083
• PR #25013 [v24.2.x] k/fetch: fix selecting replica with highest high watermark by @mmaslankaprv
• PR #25086 [v24.2.x] pandaproxy/sr: Improve schema lookup by @IoannisRP

Full Changelog: v24.2.17...v24.2.18

v24.3.5

Features

• rpk: rpk group describe now supports printing instance IDs by @daisukebe in #24908

Bug Fixes

• Schema Registry: Fix a regression with protobuf map fields by @BenPope in #24975

Improvements

• rpk security user is now available for users with Cloud profiles. by @r-vasquez in #24951

Full Changelog: v24.3.4...v24.3.5

v24.2.17

Bug Fixes

• Fixes a crash during partition shutdown. This can happen during partition moves (cross core/broker) or at broker shutdown. by @bharathv in #24938
• Schema Registry: Fix a regression with protobuf map fields by @BenPope in #24974
• #24915 Fixes integer overflow issues when given a schema via the POST /subject/{subject}/version where version was > INT_MAX or a negative value was provided. by @michael-redpanda in #24917
• #25008 Schema Registry/Protobuf: Fix a regression with maps. by @BenPope in #25010

Full Changelog: v24.2.16...v24.2.17

v24.3.4

Features

• Introduces the node config crash_loop_sleep_sec, which sets the time the broker sleeps before terminating the process when the limit on the number of consecutive times a broker can crash has been reached. This is most useful in Kubernetes environments where setting this value allows customers to have ssh access into a crash looping pod for a short window of time. by @pgellert in #24825

Bug Fixes

• Avoid large allocations for the kafka response sequencing map. by @pgellert in #24743
• Fixes Iceberg metadata serialization to avoid writing an extraneous empty Avro block. This would previously prevent some query engines (e.g. BigQuery) from reading tables created by Iceberg Topics. by @andrwng in #24920
• Fixes a bug where failing to audit an authentication event could lead to a broker crash. by @pgellert in #24738
• Fixes a crash during partition shutdown. This can happen during partition moves (cross core/broker) or at broker shutdown. by @bharathv in #24939
• Fixes an issue where transactions incorrectly timeout due incorrect cleanup of evicted producers. by @bharathv in #24879
• Remove partial kvstore snapshots at startup. by @ztlpn in #24845
• #24914 Fixes integer overflow issues when given a schema via the POST /subject/{subject}/version where version was > INT_MAX or a negative value was provided. by @michael-redpanda in #24916

Improvements

• stable leadership under load by @mmaslankaprv in #24773
• PR #24672 [v24.3.x] raft: Make load_snapshot exception safe by @Lazin
• PR #24747 [v24.3.x] [CORE-8485] Reset translation state on snapshot by @mmaslankaprv
• PR #24764 [v24.3.x] [CORE-8787] Schema Registry: Support normalize=true for protobuf by @BenPope
• PR #24765 [v24.3.x] [CORE-8754] Handle new TLS error code by @michael-redpanda
• PR #24783 [CORE-8787] [v24.3.x] schema_registry: Normalization improvements by @BenPope
• PR #24785 [v24.3.x] bazel: Update protobuf to v29.0 by @BenPope
• PR #24786 [v24.3.x] bazel: add rp_util and compat by @rockwotj
• PR #24790 [v24.3.x] c/topic_table: notify ntp delta waiters in batches by @ztlpn
• PR #24799 [v24.3.x] [CORE-8450] schema_registry/protobuf: Optimize construction of iobuf by @BenPope
• PR #24805 [v24.3.x] rm_stm: improved logging related to producer eviction by @bharathv
• PR #24809 Manual backport 24772 v24.3.x 935 by @michael-redpanda
• PR #24836 [v24.3.x] ducktape: bump spark version by @ztlpn
• PR #24841 [v24.3.x] r/stm_manager: stop state machines before waiting for gate by @mmaslankaprv
• PR #24872 [v24.3.x] pandaproxy/sr: Fix normalized rendering for custom options by @IoannisRP
• PR #24903 [v24.3.x] CORE-8804 Make brokers field no longer required by @michael-redpanda
• PR #24906 [v24.3.x] Fix partitions local summary by @bashtanov
• PR #24910 [v24.3.x] pandaproxy/sr: Add rendering support for extension range options by @IoannisRP

Full Changelog: v24.3.3...v24.3.4

v24.2.16

Features

• #24826 Introduces the node config crash_loop_sleep_sec, which sets the time the broker sleeps before terminating the process when the limit on the number of consecutive times a broker can crash has been reached. This is most useful in Kubernetes environments where setting this value allows customers to have ssh access into a crash looping pod for a short window of time. by @pgellert in #24846

Bug Fixes

• Avoid large allocations for the kafka response sequencing map. by @pgellert in #24742
• Fixes a bug where failing to audit an authentication event could lead to a broker crash. by @pgellert in #24739
• Fixes an issue where transactions incorrectly timeout due incorrect cleanup of evicted producers. by @bharathv in #24878
• Remove partial kvstore snapshots at startup. by @ztlpn in #24843
• #24915 Fixes integer overflow issues when given a schema via the POST /subject/{subject}/version where version was > INT_MAX or a negative value was provided. by @michael-redpanda in #24917

Improvements

• #24668 stable leadership under load by @mmaslankaprv in #24708

Full Changelog: v24.2.15...v24.2.16

v24.3.3

Bug Fixes

• Fixes a bug in Redpanda's Iceberg manifest list Avro definition that previously resulted in an end-of-file (EOF) error when reading manifest list Avro files written by other engines. This could previously crash Redpanda or block Redpanda from appending Iceberg data, and could also prevent certain query engines from successfully reading Iceberg data written by Redpanda. by @andrwng in #24650
• Fixes a bug which may lead to archival_metadata_stm inconsistencies when reconfiguring clusters with recovered compacted topics. by @mmaslankaprv in #24678
• #24684 Fixes an issue that blocked the compaction of consumer offsets with group transactions. by @bharathv in #24688
• fixes rare bug leading to offset translation inconsistency in recovered topics by @mmaslankaprv in #24628

Improvements

• Added metrics for pandaproxy resource usage. by @IoannisRP in #24603
• Adds logging to mention data removed by compaction. by @andrwng in #24736
• Move failed authorization log statements from the kafka logger to a new kafka/authz logger, allowing for fine grained control over log statements for failed authorization. by @rockwotj in #24718
• rpk now supports well-known protobuf types when encoding/decoding records using Schema Registry. by @r-vasquez in #24699
• PR #24591 [v24.3.x] pandaproxy: add missing internal metrics by @IoannisRP
• PR #24608 [v24.3.x] storage: add tombstones_removed metric to probe by @WillemKauf
• PR #24619 [v24.3.x] Offset translator consistency validation by @mmaslankaprv
• PR #24627 [v24.3.x] rpk remote debug bundle: job-id help text change by @r-vasquez
• PR #24705 [v24.3.x] kafka/client: replace std::vector with chunked vector by @IoannisRP
• PR #24729 [v24.3.x] rpk bundle: Fix race condition in SASL credential redaction by @r-vasquez

Full Changelog: v24.3.2...v24.3.3

v24.2.15

Bug Fixes

• Fixes a bug where failing to audit an authentication event could lead to a broker crash. by @pgellert in #24739
• Fixes a bug which may lead to archival_metadata_stm inconsistencies when reconfiguring clusters with recovered compacted topics. by @mmaslankaprv in #24680
• #24685 Fixes an issue that blocked the compaction of consumer offsets with group transactions. by @bharathv in #24689
• fixes rare bug leading to offset translation inconsistency in recovered topics by @mmaslankaprv in #24629

Full Changelog: v24.2.14...v24.2.15

v24.2.14

Bug Fixes

• Fixes a bug in which a segment being rolled and closed could race, leading to a triggered vassert. by @WillemKauf in #24559

Improvements

• Added metrics for pandaproxy resource usage. by @IoannisRP in #24604
• Show leader id in /v1/cluster/partitions response. by @ztlpn in #24584

Full Changelog: v24.2.13...v24.2.14

v24.3.2

Features

• Improve the user messages when the
  topic_partitions_reserve_shard0 cluster config is used and a user tries to create a topic with more partitions than the core-based partition limit. by @pgellert in #24461

Bug Fixes

• Ensure redpanda_cloud_storage_cloud_log_size metric consistent across all replicas. We used to update it seldomly from the leader replica only which lead to inconsistent/stale values. by @nvartolomei in #24364
• Fixed a bug in which sliding window compaction may become stuck on failing to build an index map for a single segment. by @WillemKauf in #24424
• Fixes a bug in which a segment being rolled and closed could race, leading to a triggered vassert. by @WillemKauf in #24560
• Fixes a bug in which segments which may have tombstones in them were not considered eligible for self-compaction. by @WillemKauf in #24500
• Fixes a bug that could prevent topic recovery on ABS object storage when there are objects in a bucket from multiple clusters (e.g. following a whole cluster restore). by @andrwng in #24455
• Fixes a bug where rpk wasn't parsing --help when used alongside --redpanda-id in rpk cloud <provider> byoc apply by @r-vasquez in #24396
• Fixes a bug where serializing manifests for Iceberg topics with decimal fields could cause Redpanda to crash or upload invalid manifests by @oleiman in #24467
• Fixes a crash resulting from incorrect cleanup of log readers used for iceberg translation. by @bharathv in #24576
• Fixes a race that could prevent Iceberg translation from happening following a leadership change. by @andrwng in #24562
• Fixes accounting of iceberg commit lag metric that can remain erroneously high in some cases even though the translation if fully caught up. Additionally the change ensures that only partition leaders emit lag metrics while followers emit 0 lag. by @bharathv in #24575
• If a discrete disk is used for cloud storage cache Redpanda previously rejected writes if that disk (cache disk) was full (in degraded state). This is incorrect since the cache disk isn't in the way of writes. From now on, reject writes only if the data disk is full (in degraded state). by @nvartolomei in #24486
• #24428 Schema Registry: fixes a bug in the Avro compatibility check reader_field_missing_default_value where it was too lenient for missing default values of null-able types. by @pgellert in #24430
• #24587 Redpanda will now permit topics to be created with redpanda.remote.[read|write] set to true when a license is expired or missing provided that the cluster config cloud_storage_enabled is set to false. by @michael-redpanda in #24588

Improvements

• Adds additional debug log messages in the datalake coordinator regarding files to be committed to Iceberg. by @andrwng in #24563
• Beta version of Iceberg support was incorrectly classified as "enterprise only". by @oleiman in #24443
• Leader balancer: don't treat each core as independent and balance total number of leaders on each node as well. by @ztlpn in #24440
• Show leader id in /v1/cluster/partitions response. by @ztlpn in #24585
• #24539 Disable datalake services in recovery mode by @ztlpn in #24549
• rpk topic describe now supports the --format flag to display the output in either JSON or YAML. by @r-vasquez in #24438

Full Changelog: v24.3.1...v24.3.2

v24.2.13

Features

• Improve the user messages when the
  topic_partitions_reserve_shard0 cluster config is used and a user tries to create a topic with more partitions than the core-based partition limit. by @pgellert in #24462

Bug Fixes

• Ensure redpanda_cloud_storage_cloud_log_size metric consistent across all replicas. We used to update it seldomly from the leader replica only which lead to inconsistent/stale values. by @nvartolomei in #24365
• Fixes a bug that could prevent topic recovery on ABS object storage when there are objects in a bucket from multiple clusters (e.g. following a whole cluster restore). by @andrwng in #24454
• Fixes a bug where rpk wasn't parsing --help when used alongside --redpanda-id in rpk cloud <provider> byoc apply by @r-vasquez in #24397
• If a discrete disk is used for cloud storage cache Redpanda previously rejected writes if that disk (cache disk) was full (in degraded state). This is incorrect since the cache disk isn't in the way of writes. From now on, reject writes only if the data disk is full (in degraded state). by @nvartolomei in #24484
• #24431 Schema Registry: fixes a bug in the Avro compatibility check reader_field_missing_default_value where it was too lenient for missing default values of null-able types. by @pgellert in #24432
• PR #24200 [v24.2.x] cst/cache: fix use-after-move caused by calling get_exception twice by @nvartolomei
• PR #24329 [v24.2.x] Fixed race condition between appends and prefix truncation by @mmaslankaprv
• PR #24335 rm_stm: remove always true assert on transaction_ga feature by @bharathv
• PR #24349 [v24.2.x] c/balancer_planner: check if topic exists in node count map by @mmaslankaprv
• PR #24372 [v24.2.x] c/controller_backend: allow shutdown_partition to fail on app shutdown by @bashtanov
• PR #24459 [v24.2.x] raft/c: fix an indefinite hang in transfer leadership by @bharathv

Full Changelog: v24.2.12...v24.2.13