Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to authenticate the request: invalid bearer token #5622

Open
4 tasks done
fruworg opened this issue Feb 25, 2025 · 0 comments
Open
4 tasks done

Unable to authenticate the request: invalid bearer token #5622

fruworg opened this issue Feb 25, 2025 · 0 comments
Labels
bug Something isn't working

Comments

@fruworg
Copy link

fruworg commented Feb 25, 2025

Before creating an issue, make sure you've checked the following:

  • You are running the latest released version of k0s
  • Make sure you've searched for existing issues, both open and closed
  • Make sure you've searched for PRs too, a fix might've been merged already
  • You're looking at docs for the released version, "main" branch docs are usually ahead of released versions.

Platform

Linux 5.14.0-503.26.1.el9_5.aarch64 #1 SMP PREEMPT_DYNAMIC Mon Feb 10 11:01:35 UTC 2025 aarch64 GNU/Linux
NAME="Red Hat Enterprise Linux"
VERSION="9.5 (Plow)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="9.5"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Red Hat Enterprise Linux 9.5 (Plow)"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9"
BUG_REPORT_URL="https://issues.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_BUGZILLA_PRODUCT_VERSION=9.5
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.5"

Version

v1.32.1+k0s.0

Sysinfo

`k0s sysinfo` 
Total memory: 11.3 GiB (pass)
File system of /var/lib/k0s: xfs (pass)
Disk space available for /var/lib/k0s: 20.3 GiB (pass)
Relative disk space available for /var/lib/k0s: 41% (pass)
Name resolution: localhost: [::1 127.0.0.1] (pass)
Operating system: Linux (pass)
  Linux kernel release: 5.14.0-503.26.1.el9_5.aarch64 (pass)
  Max. file descriptors per process: current: 524288 / max: 524288 (pass)
  AppArmor: unavailable (pass)
  Executable in PATH: modprobe: /usr/sbin/modprobe (pass)
  Executable in PATH: mount: /usr/bin/mount (pass)
  Executable in PATH: umount: /usr/bin/umount (pass)
  /proc file system: mounted (0x9fa0) (pass)
  Control Groups: version 2 (pass)
    cgroup controller "cpu": available (is a listed root controller) (pass)
    cgroup controller "cpuacct": available (via cpu in version 2) (pass)
    cgroup controller "cpuset": available (is a listed root controller) (pass)
    cgroup controller "memory": available (is a listed root controller) (pass)
    cgroup controller "devices": available (device filters attachable) (pass)
    cgroup controller "freezer": available (cgroup.freeze exists) (pass)
    cgroup controller "pids": available (is a listed root controller) (pass)
    cgroup controller "hugetlb": available (is a listed root controller) (pass)
    cgroup controller "blkio": available (via io in version 2) (pass)
  CONFIG_CGROUPS: Control Group support: built-in (pass)
    CONFIG_CGROUP_FREEZER: Freezer cgroup subsystem: built-in (pass)
    CONFIG_CGROUP_PIDS: PIDs cgroup subsystem: built-in (pass)
    CONFIG_CGROUP_DEVICE: Device controller for cgroups: built-in (pass)
    CONFIG_CPUSETS: Cpuset support: built-in (pass)
    CONFIG_CGROUP_CPUACCT: Simple CPU accounting cgroup subsystem: built-in (pass)
    CONFIG_MEMCG: Memory Resource Controller for Control Groups: built-in (pass)
    CONFIG_CGROUP_HUGETLB: HugeTLB Resource Controller for Control Groups: built-in (pass)
    CONFIG_CGROUP_SCHED: Group CPU scheduler: built-in (pass)
      CONFIG_FAIR_GROUP_SCHED: Group scheduling for SCHED_OTHER: built-in (pass)
        CONFIG_CFS_BANDWIDTH: CPU bandwidth provisioning for FAIR_GROUP_SCHED: built-in (pass)
    CONFIG_BLK_CGROUP: Block IO controller: built-in (pass)
  CONFIG_NAMESPACES: Namespaces support: built-in (pass)
    CONFIG_UTS_NS: UTS namespace: built-in (pass)
    CONFIG_IPC_NS: IPC namespace: built-in (pass)
    CONFIG_PID_NS: PID namespace: built-in (pass)
    CONFIG_NET_NS: Network namespace: built-in (pass)
  CONFIG_NET: Networking support: built-in (pass)
    CONFIG_INET: TCP/IP networking: built-in (pass)
      CONFIG_IPV6: The IPv6 protocol: built-in (pass)
    CONFIG_NETFILTER: Network packet filtering framework (Netfilter): built-in (pass)
      CONFIG_NETFILTER_ADVANCED: Advanced netfilter configuration: built-in (pass)
      CONFIG_NF_CONNTRACK: Netfilter connection tracking support: module (pass)
      CONFIG_NETFILTER_XTABLES: Netfilter Xtables support: built-in (pass)
        CONFIG_NETFILTER_XT_TARGET_REDIRECT: REDIRECT target support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_COMMENT: "comment" match support: module (pass)
        CONFIG_NETFILTER_XT_MARK: nfmark target and match support: module (pass)
        CONFIG_NETFILTER_XT_SET: set target and match support: module (pass)
        CONFIG_NETFILTER_XT_TARGET_MASQUERADE: MASQUERADE target support: module (pass)
        CONFIG_NETFILTER_XT_NAT: "SNAT and DNAT" targets support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: "addrtype" address type match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_CONNTRACK: "conntrack" connection tracking match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_MULTIPORT: "multiport" Multiple port match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_RECENT: "recent" match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_STATISTIC: "statistic" match support: module (pass)
      CONFIG_NETFILTER_NETLINK: module (pass)
      CONFIG_NF_NAT: module (pass)
      CONFIG_IP_SET: IP set support: module (pass)
        CONFIG_IP_SET_HASH_IP: hash:ip set support: module (pass)
        CONFIG_IP_SET_HASH_NET: hash:net set support: module (pass)
      CONFIG_IP_VS: IP virtual server support: module (pass)
        CONFIG_IP_VS_NFCT: Netfilter connection tracking: built-in (pass)
        CONFIG_IP_VS_SH: Source hashing scheduling: module (pass)
        CONFIG_IP_VS_RR: Round-robin scheduling: module (pass)
        CONFIG_IP_VS_WRR: Weighted round-robin scheduling: module (pass)
      CONFIG_NF_CONNTRACK_IPV4: IPv4 connection tracking support (required for NAT): unknown (warning)
      CONFIG_NF_REJECT_IPV4: IPv4 packet rejection: module (pass)
      CONFIG_NF_NAT_IPV4: IPv4 NAT: unknown (warning)
      CONFIG_IP_NF_IPTABLES: IP tables support: module (pass)
        CONFIG_IP_NF_FILTER: Packet filtering: module (pass)
          CONFIG_IP_NF_TARGET_REJECT: REJECT target support: module (pass)
        CONFIG_IP_NF_NAT: iptables NAT support: module (pass)
        CONFIG_IP_NF_MANGLE: Packet mangling: module (pass)
      CONFIG_NF_DEFRAG_IPV4: module (pass)
      CONFIG_NF_CONNTRACK_IPV6: IPv6 connection tracking support (required for NAT): unknown (warning)
      CONFIG_NF_NAT_IPV6: IPv6 NAT: unknown (warning)
      CONFIG_IP6_NF_IPTABLES: IP6 tables support: module (pass)
        CONFIG_IP6_NF_FILTER: Packet filtering: module (pass)
        CONFIG_IP6_NF_MANGLE: Packet mangling: module (pass)
        CONFIG_IP6_NF_NAT: ip6tables NAT support: module (pass)
      CONFIG_NF_DEFRAG_IPV6: module (pass)
    CONFIG_BRIDGE: 802.1d Ethernet Bridging: module (pass)
      CONFIG_LLC: module (pass)
      CONFIG_STP: module (pass)
  CONFIG_EXT4_FS: The Extended 4 (ext4) filesystem: module (pass)
  CONFIG_PROC_FS: /proc file system support: built-in (pass)

What happened?

After deploying the cluster via k0sctl, only one node is available.

Steps to reproduce

  1. copy k0sctl.yaml
  2. k0sctl apply
  3. k0s kubectl get nodes

Expected behavior

something like this:

NAME STATUS ROLES AGE VERSION
<first-REDACTED> Ready control-plane 22m v1.32.1+k0s
<second-REDACTED> Ready control-plane 22m v1.32.1+k0s

Actual behavior

NAME STATUS ROLES AGE VERSION
<first-REDACTED> Ready control-plane 22m v1.32.1+k0s

Screenshots and logs

kosctl.yaml:

apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
metadata:
  name: <cluster-REDACTED>
spec:
  hosts:
  - role: controller+worker
    noTaints: true
    privateInterface: wt0
    ssh:
      address: <first-REDACTED>.netbird.cloud
      user: root
      keyPath: ~/.ssh/id_rsa
  - role: controller+worker
    noTaints: true
    privateInterface: wt0
    ssh:
      address: <second-REDACTED>.netbird.cloud
      user: root
      keyPath: ~/.ssh/id_rsa
  k0s:
    version: 1.32.1+k0s.0
    config:
      spec:
        storage:
          type: kine
          kine:
            dataSource: mysql://k0s:<passwd-REDACTED>@tcp(127.0.0.1:4000)/k0s
        telemetry:
          enabled: false
        api:
          externalAddress: <first-REDACTED>.netbird.cloud
          sans:
          - <first-REDACTED>.netbird.cloud
          - <second-REDACTED>.netbird.cloud

k0sctl apply output:


⠀⣿⣿⡇⠀⠀⢀⣴⣾⣿⠟⠁⢸⣿⣿⣿⣿⣿⣿⣿⡿⠛⠁⠀⢸⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠀█████████ █████████ ███
⠀⣿⣿⡇⣠⣶⣿⡿⠋⠀⠀⠀⢸⣿⡇⠀⠀⠀⣠⠀⠀⢀⣠⡆⢸⣿⣿⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀███          ███    ███
⠀⣿⣿⣿⣿⣟⠋⠀⠀⠀⠀⠀⢸⣿⡇⠀⢰⣾⣿⠀⠀⣿⣿⡇⢸⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠀███          ███    ███
⠀⣿⣿⡏⠻⣿⣷⣤⡀⠀⠀⠀⠸⠛⠁⠀⠸⠋⠁⠀⠀⣿⣿⡇⠈⠉⠉⠉⠉⠉⠉⠉⠉⢹⣿⣿⠀███          ███    ███
⠀⣿⣿⡇⠀⠀⠙⢿⣿⣦⣀⠀⠀⠀⣠⣶⣶⣶⣶⣶⣶⣿⣿⡇⢰⣶⣶⣶⣶⣶⣶⣶⣶⣾⣿⣿⠀█████████    ███    ██████████
k0sctl v0.22.0 Copyright 2023, k0sctl authors.
By continuing to use k0sctl you agree to these terms:
https://k0sproject.io/licenses/eula
INFO ==> Running phase: Connect to hosts
INFO [ssh] <second-REDACTED>.netbird.cloud:22: connected
INFO [ssh] <first-REDACTED>.netbird.cloud:22: connected
INFO ==> Running phase: Detect host operating systems
INFO [ssh] <second-REDACTED>.netbird.cloud:22: is running Red Hat Enterprise Linux 9.5 (Plow)
INFO [ssh] <first-REDACTED>.netbird.cloud:22: is running Red Hat Enterprise Linux 9.5 (Plow)
INFO ==> Running phase: Acquire exclusive host lock
INFO ==> Running phase: Prepare hosts
INFO ==> Running phase: Gather host facts
INFO [ssh] <first-REDACTED>.netbird.cloud:22: using <first-REDACTED> as hostname
INFO [ssh] <second-REDACTED>.netbird.cloud:22: using <second-REDACTED> as hostname
INFO [ssh] <first-REDACTED>.netbird.cloud:22: discovered 100.93.80.238 as private address
INFO [ssh] <second-REDACTED>.netbird.cloud:22: discovered 100.93.131.91 as private address
INFO ==> Running phase: Validate hosts
INFO ==> Running phase: Gather k0s facts
INFO [ssh] <second-REDACTED>.netbird.cloud:22: found existing configuration
INFO [ssh] <first-REDACTED>.netbird.cloud:22: found existing configuration
INFO ==> Running phase: Validate facts
INFO [ssh] <first-REDACTED>.netbird.cloud:22: validating configuration
INFO [ssh] <second-REDACTED>.netbird.cloud:22: validating configuration
INFO ==> Running phase: Configure k0s
INFO [ssh] <first-REDACTED>.netbird.cloud:22: installing new configuration
INFO [ssh] <second-REDACTED>.netbird.cloud:22: installing new configuration
INFO ==> Running phase: Initialize the k0s cluster
INFO [ssh] <first-REDACTED>.netbird.cloud:22: installing k0s controller
INFO [ssh] <first-REDACTED>.netbird.cloud:22: waiting for the k0s service to start
INFO [ssh] <first-REDACTED>.netbird.cloud:22: wait for kubernetes to reach ready state
INFO ==> Running phase: Install controllers
INFO [ssh] <first-REDACTED>.netbird.cloud:22: generate join token for [ssh] <second-REDACTED>.netbird.cloud:22
INFO [ssh] <second-REDACTED>.netbird.cloud:22: validating api connection to https://<first-REDACTED>.netbird.cloud:9443
INFO [ssh] <second-REDACTED>.netbird.cloud:22: writing join token to /etc/k0s/k0stoken
INFO [ssh] <second-REDACTED>.netbird.cloud:22: installing k0s controller
INFO [ssh] <second-REDACTED>.netbird.cloud:22: starting service
INFO [ssh] <second-REDACTED>.netbird.cloud:22: waiting for the k0s service to start
INFO ==> Running phase: Release exclusive host lock
INFO ==> Running phase: Disconnect from hosts
INFO ==> Finished in 57s
INFO k0s cluster version v1.32.1+k0s.0 is now installed
INFO Tip: To access the cluster you can now fetch the admin kubeconfig using:                         
INFO      k0sctl kubeconfig

first node:

Feb 25 16:44:22 <first-REDACTED> k0s[3037]: time="2025-02-25 16:44:22" level=info msg="E0225 16:44:22.773817    3511 dns.go:153] \"Nameserver limits exceeded\" err=\"Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 100.93.80.238 9.9.9.9 169.254.169.254\"" component=kubelet stream=stderr
Feb 25 16:44:24 <first-REDACTED> k0s[3037]: time="2025-02-25 16:44:24" level=info msg="E0225 16:44:24.347739    3211 authentication.go:74] \"Unable to authenticate the request\" err=\"invalid bearer token\"" component=kube-apiserver stream=stderr

second node:

Feb 25 16:42:20 <second-REDACTED> k0s[3495]: time="2025-02-25 16:42:20" level=info msg="I0225 16:42:20.589621    3581 serving.go:386] Generated self-signed cert in-memory" component=kube-controller-manager stream=stderr
Feb 25 16:42:20 <second-REDACTED> k0s[3495]: time="2025-02-25 16:42:20" level=info msg="I0225 16:42:20.784147    3579 serving.go:386] Generated self-signed cert in-memory" component=kube-scheduler stream=stderr
Feb 25 16:42:21 <second-REDACTED> k0s[3495]: time="2025-02-25 16:42:21" level=info msg="E0225 16:42:21.073287    3571 server.go:528] \"Failed to get a backend\" err=\"No agent available\" dialID=6780906097258402204" component=konnectivity stream=stderr
Feb 25 16:42:21 <second-REDACTED> k0s[3495]: time="2025-02-25 16:42:21" level=info msg="E0225 16:42:21.073439    3571 server.go:528] \"Failed to get a backend\" err=\"No agent available\" dialID=7692605151483548974" component=konnectivity stream=stderr
Feb 25 16:42:21 <second-REDACTED> k0s[3495]: time="2025-02-25 16:42:21" level=info msg="E0225 16:42:21.073854    3571 server.go:528] \"Failed to get a backend\" err=\"No agent available\" dialID=365363991137775338" component=konnectivity stream=stderr
Feb 25 16:42:21 <second-REDACTED> k0s[3495]: time="2025-02-25 16:42:21" level=info msg="E0225 16:42:21.074566    3571 server.go:528] \"Failed to get a backend\" err=\"No agent available\" dialID=8837144582146544687" component=konnectivity stream=stderr
Feb 25 16:42:21 <second-REDACTED> k0s[3495]: time="2025-02-25 16:42:21" level=info msg="E0225 16:42:21.074870    3571 server.go:528] \"Failed to get a backend\" err=\"No agent available\" dialID=413944914470519235" component=konnectivity stream=stderr
Feb 25 16:42:21 <second-REDACTED> k0s[3495]: time="2025-02-25 16:42:21" level=info msg="E0225 16:42:21.075320    3538 remote_available_controller.go:448] \"Unhandled Error\" err=\"v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.103.86.202:443/apis/metrics.k8s.io/v1beta1: Get \\\"https://10.103.86.202:443/apis/metrics.k8s.io/v1beta1\\\": No agent available\" logger=\"UnhandledError\"" component=kube-apiserver stream=stderr
Feb 25 16:42:21 <second-REDACTED> k0s[3495]: I0225 16:42:21.427069    3495 bootstrap.go:296] "Failed to connect to apiserver" err="the server has asked for the client to provide credentials"

Additional context

No response
@fruworg fruworg added the bug Something isn't working label Feb 25, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@fruworg