Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] can't create topic in k8s ipv6. broker #23843

Open
2 of 3 tasks
zey1996 opened this issue Jan 13, 2025 · 9 comments
Open
2 of 3 tasks

[Bug] can't create topic in k8s ipv6. broker #23843

zey1996 opened this issue Jan 13, 2025 · 9 comments
Labels
type/bug The PR fixed a bug or issue reported a bug

Comments

@zey1996
Copy link

zey1996 commented Jan 13, 2025

Search before asking

  • I searched in the issues and found nothing similar.

Read release policy

  • I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.

Version

k8s with ipv6
imag version:3.3.3
no client, just cli.

Minimal reproduce step

this is my broker deploy yaml file:

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: "my-release-pulsar-broker"
  namespace: milvus
  labels:
    app: pulsar
    chart: pulsar-2.7.8
    release: my-release
    heritage: Helm
    cluster: my-release-pulsar
    component: broker
spec:
  serviceName: "my-release-pulsar-broker"
  replicas: 3
  selector:
    matchLabels:
      app: pulsar
      release: my-release
      component: broker
  updateStrategy:
    type: RollingUpdate
  podManagementPolicy: Parallel
  template:
    metadata:
      labels:
        app: pulsar
        release: my-release
        cluster: my-release-pulsar
        component: broker
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
    spec:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app: broker
                    component: bookie
                topologyKey: kubernetes.io/hostname
              weight: 1
      serviceAccountName: "my-release-pulsar-broker-acct"
      terminationGracePeriodSeconds: 30
      initContainers:
      - name: wait-zookeeper-ready
        image: "registry.paas/eos-indexer/pulsar:3.3.3"
        imagePullPolicy: IfNotPresent
        command: ["sh", "-c"]
        args:
          - >-
            until bin/bookkeeper org.apache.zookeeper.ZooKeeperMain -server my-release-pulsar-zookeeper:2181 get /admin/clusters/my-release-pulsar; do
              echo "pulsar cluster my-release-pulsar isn't initialized yet ... check in 3 seconds ..." && sleep 3;
            done;
        volumeMounts:
      - name: wait-bookkeeper-ready
        image: "registry.paas/eos-indexer/pulsar:3.3.3"
        imagePullPolicy: IfNotPresent
        command: ["sh", "-c"]
        args:
          - >
            bin/apply-config-from-env.py conf/bookkeeper.conf;
            until bin/bookkeeper shell whatisinstanceid; do
              echo "bookkeeper cluster is not initialized yet. backoff for 3 seconds ...";
              sleep 3;
            done;
            echo "bookkeeper cluster is already initialized";
            bookieServiceNumber="$(nslookup -timeout=10 my-release-pulsar-bookie | grep Name | wc -l)";
            until [ ${bookieServiceNumber} -ge 2 ]; do
              echo "bookkeeper cluster my-release-pulsar isn't ready yet ... check in 10 seconds ...";
              sleep 10;
              bookieServiceNumber="$(nslookup -timeout=10 my-release-pulsar-bookie | grep Name | wc -l)";
            done;
            echo "bookkeeper cluster is ready";
        envFrom:
          - configMapRef:
              name: "my-release-pulsar-bookie"
        volumeMounts:
      containers:
      - name: "my-release-pulsar-broker"
        image: "registry.paas/eos-indexer/pulsar:3.3.3"
        imagePullPolicy: IfNotPresent
        resources:
          requests:
            cpu: 1.5
            memory: 4096Mi
        command: ["sh", "-c"]
        args:
        - >
          bin/apply-config-from-env.py conf/broker.conf;
          bin/gen-yml-from-env.py conf/functions_worker.yml;
          echo "OK" > status;
          bin/pulsar zookeeper-shell -server my-release-pulsar-zookeeper:2181 get /loadbalance/brokers/${HOSTNAME}.my-release-pulsar-broker.milvus.svc.cluster.local:8080;
          while [ $? -eq 0 ]; do
            echo "broker ${HOSTNAME}.my-release-pulsar-broker.milvus.svc.cluster.local znode still exists ... check in 10 seconds ...";
            sleep 10;
            bin/pulsar zookeeper-shell -server my-release-pulsar-zookeeper:2181 get /loadbalance/brokers/${HOSTNAME}.my-release-pulsar-broker.milvus.svc.cluster.local:8080;
          done;
          cat conf/pulsar_env.sh;
          OPTS="${OPTS} -Dlog4j2.formatMsgNoLookups=true" exec bin/pulsar broker;
        ports:
        - name: http
          containerPort: 8080
          hostPort: 8080
        - name: "pulsar"
          containerPort: 6650
          hostPort: 6650
        envFrom:
        - configMapRef:
            name: "my-release-pulsar-broker"
        volumeMounts:
      volumes:
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: "my-release-pulsar-broker"
  namespace: milvus
  labels:
    app: pulsar
    chart: pulsar-2.7.8
    release: my-release
    heritage: Helm
    cluster: my-release-pulsar
    component: broker
data:
  zookeeperServers: "my-release-pulsar-zookeeper:2181"
  configurationStoreServers: "my-release-pulsar-zookeeper:2181"
  clusterName: my-release-pulsar
  exposeTopicLevelMetricsInPrometheus: "true"
  numHttpServerThreads: "8"
  zooKeeperSessionTimeoutMillis: "30000"
  statusFilePath: "/pulsar/status"
  functionsWorkerEnabled: "false"
  webServicePort: "8080"
  brokerServicePort: "6650"
  PULSAR_GC: |
    -Dio.netty.leakDetectionLevel=disabled -Dio.netty.recycler.linkCapacity=1024 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=32 -XX:ConcGCThreads=32 -XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:-ResizePLAB -XX:+ExitOnOutOfMemoryError
  PULSAR_MEM: |
    -Xms4096m -Xmx4096m -XX:MaxDirectMemorySize=8192m
  backlogQuotaDefaultLimitGB: "32"
  backlogQuotaDefaultRetentionPolicy: producer_exception
  defaultRetentionSizeInMB: "-1"
  defaultRetentionTimeInMinutes: "10080"
  managedLedgerDefaultAckQuorum: "1"
  managedLedgerDefaultEnsembleSize: "1"
  managedLedgerDefaultWriteQuorum: "1"
  maxMessageSize: "104857600"
  subscriptionExpirationTimeMinutes: "3"
  ttlDurationDefaultInSeconds: "259200"
---
apiVersion: v1
kind: Service
metadata:
  name: "my-release-pulsar-broker"
  namespace: milvus
  labels:
    app: pulsar
    chart: pulsar-2.7.8
    release: my-release
    heritage: Helm
    cluster: my-release-pulsar
    component: broker
spec:
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: http
    port: 8080
  - name: "pulsar"
    port: 6650
  clusterIP: None
  selector:
    app: pulsar
    release: my-release
    component: broker

What did you expect to see?

I am a newer for pulsar, I just want to run a demo of pulsar cluster with some node.
first. I deploy my pods in k8s cluster. they are look good.
then, I want to create a topic, but I'm failed.

What did you see instead?

in pulsar-admin cli, I got this.

my-release-pulsar-proxy-0:/pulsar# pulsar-admin topics create public/default/test1
2025-01-13T07:41:27,924+0000 [AsyncHttpClient-7-1] WARN  org.apache.pulsar.client.admin.internal.BaseResource - [http://localhost:8080/admin/v2/persistent/public/default/test1] Failed to perform http put request: javax.ws.rs.ServerErrorException: HTTP 504 Gateway Timeout
HTTP 504 Gateway Timeout

Reason: HTTP 504 Gateway Timeout

then I found some exception log in broker pod:

2025-01-13T07:41:04,652+0000 [pulsar-io-3-12] WARN  org.apache.pulsar.client.impl.ConnectionHandler - [persistent://public/default/__change_events] [__system_reader-reader-f6cc6053d5] Could not get connection to broker: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: io.netty.resolver.dns.DnsResolveContext$SearchDomainUnknownHostException: Failed to resolve 'my-release-pulsar-broker-2.my-release-pulsar-broker.milvus.svc.cluster.local' [A(1)] and search domain query for configured domains failed as well: [milvus.svc.cluster.local, svc.cluster.local, cluster.local] -- Will try again in 6.085 s
2025-01-13T07:41:10,739+0000 [broker-client-shared-timer-8-1] INFO  org.apache.pulsar.client.impl.ConnectionHandler - [persistent://public/default/__change_events] [__system_reader-reader-f6cc6053d5] Reconnecting after connection was closed
2025-01-13T07:41:10,742+0000 [pulsar-io-3-13] WARN  org.apache.pulsar.client.impl.ConnectionPool - Failed to open connection to my-release-pulsar-broker-2.my-release-pulsar-broker.milvus.svc.cluster.local/<unresolved>:6650 : io.netty.resolver.dns.DnsResolveContext$SearchDomainUnknownHostException: Failed to resolve 'my-release-pulsar-broker-2.my-release-pulsar-broker.milvus.svc.cluster.local' [A(1)] and search domain query for configured domains failed as well: [milvus.svc.cluster.local, svc.cluster.local, cluster.local]
2025-01-13T07:41:10,743+0000 [pulsar-io-3-13] WARN  org.apache.pulsar.client.impl.BinaryProtoLookupService - [persistent://public/default/__change_events] lookup failed : java.util.concurrent.CompletionException: io.netty.resolver.dns.DnsResolveContext$SearchDomainUnknownHostException: Failed to resolve 'my-release-pulsar-broker-2.my-release-pulsar-broker.milvus.svc.cluster.local' [A(1)] and search domain query for configured domains failed as well: [milvus.svc.cluster.local, svc.cluster.local, cluster.local]
org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: io.netty.resolver.dns.DnsResolveContext$SearchDomainUnknownHostException: Failed to resolve 'my-release-pulsar-broker-2.my-release-pulsar-broker.milvus.svc.cluster.local' [A(1)] and search domain query for configured domains failed as well: [milvus.svc.cluster.local, svc.cluster.local, cluster.local]
	at org.apache.pulsar.client.impl.ConnectionPool.lambda$createConnection$13(ConnectionPool.java:333) ~[org.apache.pulsar-pulsar-client-original-3.3.3.jar:3.3.3]
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:408) ~[io.netty-netty-transport-classes-epoll-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at java.base/java.lang.Thread.run(Unknown Source) [?:?]
Caused by: java.util.concurrent.CompletionException: io.netty.resolver.dns.DnsResolveContext$SearchDomainUnknownHostException: Failed to resolve 'my-release-pulsar-broker-2.my-release-pulsar-broker.milvus.svc.cluster.local' [A(1)] and search domain query for configured domains failed as well: [milvus.svc.cluster.local, svc.cluster.local, cluster.local]
	at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) ~[?:?]
	at java.base/java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) ~[?:?]
	at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(Unknown Source) ~[?:?]
	at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?]
	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?]
	at org.apache.pulsar.client.impl.ConnectionPool.lambda$resolveName$18(ConnectionPool.java:409) ~[org.apache.pulsar-pulsar-client-original-3.3.3.jar:3.3.3]
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.DefaultPromise.setFailure(DefaultPromise.java:110) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.resolver.InetSocketAddressResolver$2.operationComplete(InetSocketAddressResolver.java:86) ~[io.netty-netty-resolver-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.resolver.dns.DnsResolveContext$1.operationComplete(DnsResolveContext.java:268) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.resolver.dns.DnsResolveContext.finishResolve(DnsResolveContext.java:1158) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.resolver.dns.DnsResolveContext.tryToFinishResolve(DnsResolveContext.java:1097) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.resolver.dns.DnsResolveContext.query(DnsResolveContext.java:455) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.resolver.dns.DnsResolveContext.access$700(DnsResolveContext.java:70) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.resolver.dns.DnsResolveContext$2.operationComplete(DnsResolveContext.java:524) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.resolver.dns.DnsQueryContext.finishFailure(DnsQueryContext.java:343) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.resolver.dns.DnsQueryContext.onQueryWriteCompletion(DnsQueryContext.java:264) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.resolver.dns.DnsQueryContext.access$600(DnsQueryContext.java:51) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.resolver.dns.DnsQueryContext$3.operationComplete(DnsQueryContext.java:255) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.resolver.dns.DnsQueryContext$3.operationComplete(DnsQueryContext.java:252) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.internal.PromiseNotificationUtil.tryFailure(PromiseNotificationUtil.java:64) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.channel.ChannelOutboundBuffer.safeFail(ChannelOutboundBuffer.java:754) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.channel.ChannelOutboundBuffer.remove0(ChannelOutboundBuffer.java:339) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.channel.ChannelOutboundBuffer.remove(ChannelOutboundBuffer.java:319) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.channel.epoll.EpollDatagramChannel.doWrite(EpollDatagramChannel.java:386) ~[io.netty-netty-transport-classes-epoll-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:929) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.flush0(AbstractEpollChannel.java:557) ~[io.netty-netty-transport-classes-epoll-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:893) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1319) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:935) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:921) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:907) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.channel.DefaultChannelPipeline.flush(DefaultChannelPipeline.java:916) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.channel.AbstractChannel.flush(AbstractChannel.java:253) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.resolver.dns.DnsResolveContext.internalResolve(DnsResolveContext.java:398) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.resolver.dns.DnsResolveContext.doSearchDomainQuery(DnsResolveContext.java:321) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.resolver.dns.DnsAddressResolveContext.doSearchDomainQuery(DnsAddressResolveContext.java:99) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.resolver.dns.DnsResolveContext.resolve(DnsResolveContext.java:283) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.resolver.dns.DnsNameResolver.resolveNow(DnsNameResolver.java:1302) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.resolver.dns.DnsNameResolver.doResolveAllUncached0(DnsNameResolver.java:1286) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.resolver.dns.DnsNameResolver.access$700(DnsNameResolver.java:96) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.resolver.dns.DnsNameResolver$8.run(DnsNameResolver.java:1233) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:405) ~[io.netty-netty-transport-classes-epoll-4.1.115.Final.jar:4.1.115.Final]
	... 4 more
Caused by: io.netty.resolver.dns.DnsResolveContext$SearchDomainUnknownHostException: Failed to resolve 'my-release-pulsar-broker-2.my-release-pulsar-broker.milvus.svc.cluster.local' [A(1)] and search domain query for configured domains failed as well: [milvus.svc.cluster.local, svc.cluster.local, cluster.local]
	at io.netty.resolver.dns.DnsResolveContext.finishResolve(DnsResolveContext.java:1150) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.resolver.dns.DnsResolveContext.tryToFinishResolve(DnsResolveContext.java:1097) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.resolver.dns.DnsResolveContext.query(DnsResolveContext.java:455) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.resolver.dns.DnsResolveContext.access$700(DnsResolveContext.java:70) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.resolver.dns.DnsResolveContext$2.operationComplete(DnsResolveContext.java:524) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.resolver.dns.DnsQueryContext.finishFailure(DnsQueryContext.java:343) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.resolver.dns.DnsQueryContext.onQueryWriteCompletion(DnsQueryContext.java:264) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.resolver.dns.DnsQueryContext.access$600(DnsQueryContext.java:51) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.resolver.dns.DnsQueryContext$3.operationComplete(DnsQueryContext.java:255) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.resolver.dns.DnsQueryContext$3.operationComplete(DnsQueryContext.java:252) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.internal.PromiseNotificationUtil.tryFailure(PromiseNotificationUtil.java:64) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.channel.ChannelOutboundBuffer.safeFail(ChannelOutboundBuffer.java:754) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.channel.ChannelOutboundBuffer.remove0(ChannelOutboundBuffer.java:339) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.channel.ChannelOutboundBuffer.remove(ChannelOutboundBuffer.java:319) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.channel.epoll.EpollDatagramChannel.doWrite(EpollDatagramChannel.java:386) ~[io.netty-netty-transport-classes-epoll-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:929) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.flush0(AbstractEpollChannel.java:557) ~[io.netty-netty-transport-classes-epoll-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:893) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1319) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:935) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:921) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:907) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.channel.DefaultChannelPipeline.flush(DefaultChannelPipeline.java:916) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.channel.AbstractChannel.flush(AbstractChannel.java:253) ~[io.netty-netty-transport-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.resolver.dns.DnsResolveContext.internalResolve(DnsResolveContext.java:398) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.resolver.dns.DnsResolveContext.doSearchDomainQuery(DnsResolveContext.java:321) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.resolver.dns.DnsAddressResolveContext.doSearchDomainQuery(DnsAddressResolveContext.java:99) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.resolver.dns.DnsResolveContext.resolve(DnsResolveContext.java:283) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.resolver.dns.DnsNameResolver.resolveNow(DnsNameResolver.java:1302) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.resolver.dns.DnsNameResolver.doResolveAllUncached0(DnsNameResolver.java:1286) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.resolver.dns.DnsNameResolver.access$700(DnsNameResolver.java:96) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.resolver.dns.DnsNameResolver$8.run(DnsNameResolver.java:1233) ~[io.netty-netty-resolver-dns-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) ~[io.netty-netty-common-4.1.115.Final.jar:4.1.115.Final]
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:405) ~[io.netty-netty-transport-classes-epoll-4.1.115.Final.jar:4.1.115.Final]
	... 4 more
Caused by: io.netty.resolver.dns.DnsNameResolverException: [4836: /[fd11:1111:1112:15:0:0:0:a]:53] DefaultDnsQuestion(my-release-pulsar-broker-2.my-release-pulsar-broker.milvus.svc.cluster.local. IN A) failed to send a query '4836' via UDP (no stack trace available)
Caused by: io.netty.channel.unix.Errors$NativeIoException: sendToAddress(..) failed: Address family not supported by protocol
2025-01-13T07:41:10,743+0000 [pulsar-io-3-13] WARN  org.apache.pulsar.client.impl.ConnectionHandler - [persistent://public/default/__change_events] [__system_reader-reader-f6cc6053d5] Error connecting to broker: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: io.netty.resolver.dns.DnsResolveContext$SearchDomainUnknownHostException: Failed to resolve 'my-release-pulsar-broker-2.my-release-pulsar-broker.milvus.svc.cluster.local' [A(1)] and search domain query for configured domains failed as well: [milvus.svc.cluster.local, svc.cluster.local, cluster.local]
2025-01-13T07:41:10,744+0000 [pulsar-io-3-13] WARN  org.apache.pulsar.client.impl.ConnectionHandler - [persistent://public/default/__change_events] [__system_reader-reader-f6cc6053d5] Could not get connection to broker: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: io.netty.resolver.dns.DnsResolveContext$SearchDomainUnknownHostException: Failed to resolve 'my-release-pulsar-broker-2.my-release-pulsar-broker.milvus.svc.cluster.local' [A(1)] and search domain query for configured domains failed as well: [milvus.svc.cluster.local, svc.cluster.local, cluster.local] -- Will try again in 12.786 s

It appears that pulsar cannot use ipv6 dns server?
I confirm the dns server in log is working.

my-release-pulsar-broker-0:/pulsar$ nslookup my-release-pulsar-broker-2.my-release-pulsar-broker.milvus.svc.cluster.local fd11:1111:1112:15:0:0:0:a
Server:		fd11:1111:1112:15:0:0:0:a
Address:	fd11:1111:1112:15::a#53

Name:	my-release-pulsar-broker-2.my-release-pulsar-broker.milvus.svc.cluster.local
Address: 10.222.55.247

Can I run pulsar in ipv6 k8s cluster?

Anything else?

No response

Are you willing to submit a PR?

  • I'm willing to submit a PR!
@zey1996 zey1996 added the type/bug The PR fixed a bug or issue reported a bug label Jan 13, 2025
@lhotari
Copy link
Member

lhotari commented Jan 13, 2025

Can I run pulsar in ipv6 k8s cluster?

@zey1996 We don't have tests in the Apache Pulsar project for this. One known detail is that Pulsar sets -Djava.net.preferIPv4Stack=true by default. You would need to pass -Djava.net.preferIPv4Stack=false in one of the environment variables that gets added to JVM arguments for each Pulsar component.

@zey1996
Copy link
Author

zey1996 commented Jan 14, 2025

I add -Djava.net.preferIPv4Stack=false opt in broker launch command. like this:

containers:
  - args:
    - |
      bin/apply-config-from-env.py conf/broker.conf; bin/gen-yml-from-env.py conf/functions_worker.yml; echo "OK" > status; bin/pulsar zookeeper-shell -server my-release-pulsar-zookeeper:2181 get /loadbalance/brokers/${HOSTNAME}.my-release-pulsar-broker.milvus.svc.cluster.local:8080; while [ $? -eq 0 ]; do
        echo "broker ${HOSTNAME}.my-release-pulsar-broker.milvus.svc.cluster.local znode still exists ... check in 10 seconds ...";
        sleep 10;
        bin/pulsar zookeeper-shell -server my-release-pulsar-zookeeper:2181 get /loadbalance/brokers/${HOSTNAME}.my-release-pulsar-broker.milvus.svc.cluster.local:8080;
      done; cat conf/pulsar_env.sh; OPTS="${OPTS} -Dlog4j2.formatMsgNoLookups=true -Djava.net.preferIPv4Stack=false" exec bin/pulsar broker;
    command:
    - sh
    - -c

and I still get exception.

Caused by: io.netty.resolver.dns.DnsNameResolverException: [13557: /[fd11:1111:1112:15:0:0:0:a]:53] DefaultDnsQuestion(my-release-pulsar-broker-0.my-release-pulsar-broker.milvus.svc.cluster.local. IN A) failed to send a query '13557' via UDP (no stack trace available)
Caused by: io.netty.channel.unix.Errors$NativeIoException: sendToAddress(..) failed: Address family not supported by protocol

It doesn't seem to work. Maybe I did was wrong,or it's a new problem?
@lhotari

@lhotari
Copy link
Member

lhotari commented Jan 14, 2025

It doesn't seem to work. Maybe I did was wrong,or it's a new problem? @lhotari

@zey1996 Due to the way how the scripts are structured, it's currently not supported to set -Djava.net.preferIPv4Stack=false in OPTS.

You will need to add it to PULSAR_EXTRA_OPTS/BOOKIE_EXTRA_OPTS. The problem in adding to PULSAR_EXTRA_OPTS/BOOKIE_EXTRA_OPTS is that it drops the defaults which are -Dpulsar.allocator.exit_on_oom=true -Dio.netty.recycler.maxCapacityPerThread=4096.

I'll create a separate PR to allow putting -Djava.net.preferIPv4Stack=false to OPTS, that is #23846.

In your case you could add PULSAR_EXTRA_OPTS="-Dpulsar.allocator.exit_on_oom=true -Dio.netty.recycler.maxCapacityPerThread=4096 -Djava.net.preferIPv4Stack=false" to keep the previous settings and to add the -Djava.net.preferIPv4Stack=false option.

@zey1996
Copy link
Author

zey1996 commented Jan 15, 2025

I try to set env PULSAR_EXTRA_OPTS="-Djava.net.preferIPv4Stack=false", but it's not work. I found launch command has two same argument

my-release-pulsar-broker-0:/pulsar$ echo $PULSAR_EXTRA_OPTS
-Djava.net.preferIPv4Stack=false
my-release-pulsar-broker-0:/pulsar$ ps -ef|grep java
pulsar        1      0 21 02:07 ?        00:00:24 /opt/jvm/bin/java -Dlog4j.shutdownHookEnabled=false -cp /pulsar/conf:::/pulsar/lib/*: -Dlog4j2.formatMsgNoLookups=true -Dorg.xerial.snappy.use.systemlib=true -Dlog4j.configurationFile=log4j2.yaml -Djute.maxbuffer=10485760 -Djava.net.preferIPv4Stack=true -Dzookeeper.clientTcpKeepAlive=true -Dio.netty.tryReflectionSetAccessible=true --add-opens java.base/java.io=ALL-UNNAMED --add-opens java.base/java.util.zip=ALL-UNNAMED --add-opens java.base/java.nio=ALL-UNNAMED --add-opens java.base/jdk.internal.misc=ALL-UNNAMED --add-opens java.base/sun.net=ALL-UNNAMED --add-opens java.management/sun.management=ALL-UNNAMED --add-opens jdk.management/com.sun.management.internal=ALL-UNNAMED --add-opens java.base/jdk.internal.platform=ALL-UNNAMED -Xms4096m -Xmx4096m -XX:MaxDirectMemorySize=8192m -Dio.netty.leakDetectionLevel=disabled -Dio.netty.recycler.linkCapacity=1024 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=32 -XX:ConcGCThreads=32 -XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:-ResizePLAB -XX:+ExitOnOutOfMemoryError -Xlog:async -Xlog:gc*,safepoint:/pulsar/logs/pulsar_gc_%p.log:time,uptime,tags:filecount=10,filesize=20M -Djava.net.preferIPv4Stack=false -Dpulsar.log.appender=RoutingAppender -Dpulsar.log.dir=/pulsar/logs -Dpulsar.log.level=info -Dpulsar.log.root.level=info -Dpulsar.log.immediateFlush=false -Dpulsar.routing.appender.default=Console -Dlog4j2.is.webapp=false -Dpulsar.functions.process.container.log.dir=/pulsar/logs -Dpulsar.functions.java.instance.jar=/pulsar/instances/java-instance.jar -Dpulsar.functions.python.instance.file=/pulsar/instances/python-instance/python_instance_main.py -Dpulsar.functions.extra.dependencies.dir=/pulsar/instances/deps -Dpulsar.functions.instance.classpath=/pulsar/conf:::/pulsar/lib/*: -Dpulsar.functions.log.conf=/pulsar/conf/functions_log4j2.xml -Dbookkeeper.metadata.bookie.drivers=org.apache.pulsar.metadata.bookkeeper.PulsarMetadataBookieDriver -Dbookkeeper.metadata.client.drivers=org.apache.pulsar.metadata.bookkeeper.PulsarMetadataClientDriver -Dpulsar.log.file=pulsar-broker.log org.apache.pulsar.PulsarBrokerStarter --broker-conf /pulsar/conf/broker.conf

I'm learning the start-up process. It may take a while.
What is the correct way to configure parameters?

@lhotari
Copy link
Member

lhotari commented Jan 15, 2025

I try to set env PULSAR_EXTRA_OPTS="-Djava.net.preferIPv4Stack=false", but it's not work. I found launch command has two same argument

Yes, that is expected. The last parameter "wins".
You can use the jinfo -sysprops 1 or jcmd 1 VM.system_properties commands in the shell to check effective values.

@zey1996
Copy link
Author

zey1996 commented Jan 16, 2025

Yes, that is expected. The last parameter "wins".
You can use the jinfo -sysprops 1 or jcmd 1 VM.system_properties commands in the shell to check effective values.

I do this. but I can't understand, It looks like it could work.

my-release-pulsar-broker-2:/pulsar$ jinfo -sysprops 1|grep IP
java.net.preferIPv4Stack=false
my-release-pulsar-broker-2:/pulsar$ jcmd 1 VM.system_properties|grep IP
java.net.preferIPv4Stack=false

I found -Djava.net.preferIPv4Stack=true be set in bin/pulsar, and there's no switch. Can I overwrite it without modifying the image?
Maybe we can add a switch?

@lhotari
Copy link
Member

lhotari commented Jan 16, 2025

Yes, that is expected. The last parameter "wins".
You can use the jinfo -sysprops 1 or jcmd 1 VM.system_properties commands in the shell to check effective values.

I do this. but I can't understand, It looks like it could work.

my-release-pulsar-broker-2:/pulsar$ jinfo -sysprops 1|grep IP
java.net.preferIPv4Stack=false
my-release-pulsar-broker-2:/pulsar$ jcmd 1 VM.system_properties|grep IP
java.net.preferIPv4Stack=false

As you can see, the value is false as expected.


I found -Djava.net.preferIPv4Stack=true be set in bin/pulsar, and there's no switch. Can I overwrite it without modifying the image?
Maybe we can add a switch?

There is no need to add a switch since the value is false as expected.

@zey1996
Copy link
Author

zey1996 commented Jan 16, 2025

sorry I don't understand. Let me sort it out。
First I get a exception. then I try to set java.net.preferIPv4Stack=false to resolve it.
I set env PULSAR_EXTRA_OPTS.

my-release-pulsar-broker-0:/pulsar$ echo $PULSAR_EXTRA_OPTS
-Djava.net.preferIPv4Stack=false
my-release-pulsar-broker-0:/pulsar$ jinfo -sysprops 1|grep IP
java.net.preferIPv4Stack=false
my-release-pulsar-broker-0:/pulsar$ jcmd 1 VM.system_properties|grep IP
java.net.preferIPv4Stack=false
my-release-pulsar-broker-0:/pulsar$  ps -ef|grep java
pulsar        1      0  0 Jan15 ?        00:09:07 /opt/jvm/bin/java -Dlog4j.shutdownHookEnabled=false -cp /pulsar/conf:::/pulsar/lib/*: -Dlog4j2.formatMsgNoLookups=true -Dorg.xerial.snappy.use.systemlib=true -Dlog4j.configurationFile=log4j2.yaml -Djute.maxbuffer=10485760 -Djava.net.preferIPv4Stack=true -Dzookeeper.clientTcpKeepAlive=true -Dio.netty.tryReflectionSetAccessible=true --add-opens java.base/java.io=ALL-UNNAMED --add-opens java.base/java.util.zip=ALL-UNNAMED --add-opens java.base/java.nio=ALL-UNNAMED --add-opens java.base/jdk.internal.misc=ALL-UNNAMED --add-opens java.base/sun.net=ALL-UNNAMED --add-opens java.management/sun.management=ALL-UNNAMED --add-opens jdk.management/com.sun.management.internal=ALL-UNNAMED --add-opens java.base/jdk.internal.platform=ALL-UNNAMED -Xms4096m -Xmx4096m -XX:MaxDirectMemorySize=8192m -Dio.netty.leakDetectionLevel=disabled -Dio.netty.recycler.linkCapacity=1024 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=32 -XX:ConcGCThreads=32 -XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:-ResizePLAB -XX:+ExitOnOutOfMemoryError -Xlog:async -Xlog:gc*,safepoint:/pulsar/logs/pulsar_gc_%p.log:time,uptime,tags:filecount=10,filesize=20M -Djava.net.preferIPv4Stack=false -Dpulsar.log.appender=RoutingAppender -Dpulsar.log.dir=/pulsar/logs -Dpulsar.log.level=info -Dpulsar.log.root.level=info -Dpulsar.log.immediateFlush=false -Dpulsar.routing.appender.default=Console -Dlog4j2.is.webapp=false -Dpulsar.functions.process.container.log.dir=/pulsar/logs -Dpulsar.functions.java.instance.jar=/pulsar/instances/java-instance.jar -Dpulsar.functions.python.instance.file=/pulsar/instances/python-instance/python_instance_main.py -Dpulsar.functions.extra.dependencies.dir=/pulsar/instances/deps -Dpulsar.functions.instance.classpath=/pulsar/conf:::/pulsar/lib/*: -Dpulsar.functions.log.conf=/pulsar/conf/functions_log4j2.xml -Dbookkeeper.metadata.bookie.drivers=org.apache.pulsar.metadata.bookkeeper.PulsarMetadataBookieDriver -Dbookkeeper.metadata.client.drivers=org.apache.pulsar.metadata.bookkeeper.PulsarMetadataClientDriver -Dpulsar.log.file=pulsar-broker.log org.apache.pulsar.PulsarBrokerStarter --broker-conf /pulsar/conf/broker.conf
pulsar     1360   1330  0 06:01 pts/1    00:00:00 grep java

in ps -ef I get two java.net.preferIPv4Stack, ont is true and the other is false, but in jinfo or jcmd, java.net.preferIPv4Stack=false
And I still get the original exception.
I find OPTS="$OPTS -Djute.maxbuffer=10485760 -Djava.net.preferIPv4Stack=true" in bin/pulsar, I think it is the reason of I get two java.net.preferIPv4Stack.


now I still get exception.
Maybe java.net.preferIPv4Stack=false can't resolve my problem?
Or I didn't configure it properly?

@lhotari
Copy link
Member

lhotari commented Jan 16, 2025

now I still get exception.
Maybe java.net.preferIPv4Stack=false can't resolve my problem?
Or I didn't configure it properly?

It is needed in all Pulsar components. It's possible that there are also some other gaps in IPv6 support since we don't have tests as part of CI to ensure that IPv6 is supported. It's up to contributors like you to test and address possible issues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/bug The PR fixed a bug or issue reported a bug
Projects
None yet
Development

No branches or pull requests

2 participants