[KYUUBI #6836] Ship kafka-clients in binary distribution tarball without compression libs

### Why are the changes needed?

I'd like to include `kafka-clients` in the Kyuubi binary distribution tarball to enable the out-of-box support for sinking Kyuubi events to Kafka.

- Kafka is an important component in modern data platforms, and is a defacto message queue implementation, especially in the big data domain
- `kafka-clients` is released under Apache License V2, has no legal issue
- `kafka-clients` is quite a light lib, has no third-party deps except for `slf4j-api` and a few optional compression libs
- `kafka-clients` uses "none" compression as default, in practice, "gzip"(delegate to JDK gzip algorithm, no additional libs are required) already performs well for non-extreme cases

Additionally, LOG4J2 has a built-in `KafkaAppender` that supports sinking logging to Kafka, which also requires `kafka-clients` in the classpath, I have some initial ideas to forward both Kyuubi server's and engine bootstrap processes log to Kafka in a structured format, will send another PR to add docs to guide users in configuring that.

### How was this patch tested?

Review

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #6836 from pan3793/kafka-lib.

Closes #6836

b069eb1 [Cheng Pan] Ship kafka-clients in binary distribution tarball without compression libs

Authored-by: Cheng Pan <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>

Author: pan3793

LICENSE-binary

@@ -306,8 +306,6 @@ io.vertx:vertx-grpc
 com.squareup.retrofit2:retrofit
 com.squareup.okhttp3:okhttp
 org.apache.kafka:kafka-clients
-org.lz4:lz4-java
-org.xerial.snappy:snappy-java
 org.xerial:sqlite-jdbc
 
 BSD
@@ -319,7 +317,6 @@ jline:jline
 com.thoughtworks.paranamer:paranamer
 com.google.protobuf:protobuf-java-util
 com.google.protobuf:protobuf-java
-com.github.luben:zstd-jni
 org.postgresql:postgresql
 
 Eclipse Distribution License - v 1.0

NOTICE-binary

@@ -1063,19 +1063,3 @@ which can be obtained at:
     * license/LICENSE.kafka.txt (Apache License 2.0)
   * HOMEPAGE:
     * https://github.com/apache/kafka
-
-This product optionally depends on 'snappy-java', Snappy compression and
-decompression for Java, which can be obtained at:
-
-  * LICENSE:
-    * license/LICENSE.snappy-java.txt (Apache License 2.0)
-  * HOMEPAGE:
-    * https://github.com/xerial/snappy-java
-
-This product optionally depends on 'lz4-java', Lz4 compression and
-decompression for Java, which can be obtained at:
-
-  * LICENSE:
-    * license/LICENSE.lz4-java.txt (Apache License 2.0)
-  * HOMEPAGE:
-    * https://github.com/lz4/lz4-java

dev/dependencyList

@@ -128,7 +128,6 @@ log4j-api/2.24.2//log4j-api-2.24.2.jar
 log4j-core/2.24.2//log4j-core-2.24.2.jar
 log4j-slf4j-impl/2.24.2//log4j-slf4j-impl-2.24.2.jar
 logging-interceptor/3.12.12//logging-interceptor-3.12.12.jar
-lz4-java/1.8.0//lz4-java-1.8.0.jar
 metrics-core/4.2.26//metrics-core-4.2.26.jar
 metrics-jmx/4.2.26//metrics-jmx-4.2.26.jar
 metrics-json/4.2.26//metrics-json-4.2.26.jar
@@ -173,7 +172,6 @@ simpleclient_tracer_otel_agent/0.16.0//simpleclient_tracer_otel_agent-0.16.0.jar
 slf4j-api/1.7.36//slf4j-api-1.7.36.jar
 snakeyaml-engine/2.7//snakeyaml-engine-2.7.jar
 snakeyaml/2.2//snakeyaml-2.2.jar
-snappy-java/1.1.10.5//snappy-java-1.1.10.5.jar
 sqlite-jdbc/3.46.1.3//sqlite-jdbc-3.46.1.3.jar
 swagger-annotations/2.2.1//swagger-annotations-2.2.1.jar
 swagger-core/2.2.1//swagger-core-2.2.1.jar
@@ -185,4 +183,3 @@ units/1.7//units-1.7.jar
 vertx-core/4.5.3//vertx-core-4.5.3.jar
 vertx-grpc/4.5.3//vertx-grpc-4.5.3.jar
 zjsonpatch/0.3.0//zjsonpatch-0.3.0.jar
-zstd-jni/1.5.5-1//zstd-jni-1.5.5-1.jar

pom.xml

@@ -1053,7 +1053,20 @@
                 <groupId>org.apache.kafka</groupId>
                 <artifactId>kafka-clients</artifactId>
                 <version>${kafka.version}</version>
-                <optional>true</optional>
+                <exclusions>
+                    <exclusion>
+                        <groupId>com.github.luben</groupId>
+                        <artifactId>zstd-jni</artifactId>
+                    </exclusion>
+                    <exclusion>
+                        <groupId>org.lz4</groupId>
+                        <artifactId>lz4-java</artifactId>
+                    </exclusion>
+                    <exclusion>
+                        <groupId>org.xerial.snappy</groupId>
+                        <artifactId>snappy-java</artifactId>
+                    </exclusion>
+                </exclusions>
             </dependency>
 
             <dependency>