Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sentry: conn_executor.go:958: runtime error: invalid memory address or nil pointer dereference (1) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeCo... #143071

Closed
cockroach-sentry opened this issue Mar 18, 2025 · 6 comments
Labels
branch-release-24.1 Used to mark GA and release blockers, technical advisories, and bugs for 24.1 C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)

Comments

@cockroach-sentry
Copy link
Collaborator

cockroach-sentry commented Mar 18, 2025

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/6421359249/?referrer=webhooks_plugin

Panic Message:

conn_executor.go:958: runtime error: invalid memory address or nil pointer dereference
(1) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:958
  | [...repeated from below...]
Wraps: (2) while executing: WITH _ AS (SELECT _, crdb_internal.pb_to_json('_', _, _) AS _ FROM _ WHERE _ IN (_,)), _ AS (SELECT _, json_set(json_set(json_remove_path(json_set(_, ARRAY[_, _], json_build_array(json_set(((((_->'_')->'_')->_)->'_')->'_', ARRAY[_], _::JSONB))), ARRAY[_, _, __more1_10__]), ARRAY[_, _], (((_->'_')->>'_')::INT8 + _)::STRING::JSONB), ARRAY[_, _], json_build_object(_, ((extract(_, now()) * _)::INT8 * _)::STRING)) AS _ FROM _) SELECT crdb_internal.unsafe_upsert_descriptor(_, crdb_internal.json_to_pb('_', _), _) FROM _
Wraps: (3) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:958
  | runtime.gopanic
  | 	GOROOT/src/runtime/panic.go:770
  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:120
  | runtime.gopanic
  | 	GOROOT/src/runtime/panic.go:770
  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:120
  | runtime.gopanic
  | 	GOROOT/src/runtime/panic.go:770
  | runtime.panicmem
  | 	GOROOT/src/runtime/panic.go:261
  | runtime.sigpanic
  | 	GOROOT/src/runtime/signal_unix.go:881
  | github.com/cockroachdb/cockroach/pkg/sql/catalog/descpb.(*TableDescriptor_CheckConstraint).MarshalToSizedBuffer
  | 	github.com/cockroachdb/cockroach/pkg/sql/catalog/descpb/bazel-out/k8-opt/bin/pkg/sql/catalog/descpb/descpb_go_proto_/github.com/cockroachdb/cockroach/pkg/sql/catalog/descpb/structured.pb.go:8755
  | github.com/cockroachdb/cockroach/pkg/sql/catalog/descpb.(*TableDescriptor).MarshalToSizedBuffer
  | 	github.com/cockroachdb/cockroach/pkg/sql/catalog/descpb/bazel-out/k8-opt/bin/pkg/sql/catalog/descpb/descpb_go_proto_/github.com/cockroachdb/cockroach/pkg/sql/catalog/descpb/structured.pb.go:8610
  | github.com/cockroachdb/cockroach/pkg/sql/catalog/descpb.(*Descriptor_Table).MarshalToSizedBuffer
  | 	github.com/cockroachdb/cockroach/pkg/sql/catalog/descpb/bazel-out/k8-opt/bin/pkg/sql/catalog/descpb/descpb_go_proto_/github.com/cockroachdb/cockroach/pkg/sql/catalog/descpb/structured.pb.go:10273
  | github.com/cockroachdb/cockroach/pkg/sql/catalog/descpb.(*Descriptor_Table).MarshalTo
  | 	github.com/cockroachdb/cockroach/pkg/sql/catalog/descpb/bazel-out/k8-opt/bin/pkg/sql/catalog/descpb/descpb_go_proto_/github.com/cockroachdb/cockroach/pkg/sql/catalog/descpb/structured.pb.go:10266
  | github.com/cockroachdb/cockroach/pkg/sql/catalog/descpb.(*Descriptor).MarshalToSizedBuffer
  | 	github.com/cockroachdb/cockroach/pkg/sql/catalog/descpb/bazel-out/k8-opt/bin/pkg/sql/catalog/descpb/descpb_go_proto_/github.com/cockroachdb/cockroach/pkg/sql/catalog/descpb/structured.pb.go:10256
  | github.com/cockroachdb/cockroach/pkg/util/protoutil.MarshalToSizedBuffer
  | 	github.com/cockroachdb/cockroach/pkg/util/protoutil/pkg/util/protoutil/marshal.go:59
  | github.com/cockroachdb/cockroach/pkg/util/protoutil.Marshal
  | 	github.com/cockroachdb/cockroach/pkg/util/protoutil/pkg/util/protoutil/marshal.go:28
  | github.com/cockroachdb/cockroach/pkg/sql/protoreflect.JSONBMarshalToMessage
  | 	github.com/cockroachdb/cockroach/pkg/sql/protoreflect/utils.go:125
  | github.com/cockroachdb/cockroach/pkg/sql/sem/builtins.init.func591
  | 	github.com/cockroachdb/cockroach/pkg/sql/sem/builtins/builtins.go:4348
  | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*defaultBuiltinFuncOperator).Next.func1
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/builtin_funcs.go:72
  | github.com/cockroachdb/cockroach/pkg/sql/colmem.(*Allocator).PerformOperation
  | 	github.com/cockroachdb/cockroach/pkg/sql/colmem/allocator.go:455
  | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*defaultBuiltinFuncOperator).Next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/builtin_funcs.go:49
  | github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils.(*vectorTypeEnforcer).Next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils/operator.go:152
  | github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase.constBoolOp.Next
  | 	github.com/cockroachdb/cockroach/bazel-out/k8-opt/bin/pkg/sql/colexec/colexecbase/const.eg.go:169
  | github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils.(*vectorTypeEnforcer).Next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils/operator.go:152
  | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*defaultBuiltinFuncOperator).Next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/builtin_funcs.go:41
  | github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase.(*simpleProjectOp).Next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase/simple_project.go:113
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:113
  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:145
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:121
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).nextAdapter
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:245
  | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
  | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:145
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).next
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:249
  | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).Run
  | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:281
Wraps: (4) runtime error: invalid memory address or nil pointer dereference
Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *withstack.withStack (4) runtime.errorString
-- report composition:
runtime.errorString
conn_executor.go:958: *withstack.withStack (top exception)
*safedetails.withSafeDetails: while executing: WITH _ AS (SELECT _, crdb_internal.pb_to_json('_', _, _) AS _ FROM _ WHERE _ IN (_,)), _ AS (SELECT _, json_set(json_set(json_remove_path(json_set(_, ARRAY[_, _], json_build_array(json_set(((((_->'_')->'_')->_)->'_')->'_', ARRAY[_], _::JSONB))), ARRAY[_, _, __more1_10__]), ARRAY[_, _], (((_->'_')->>'_')::INT8 + _)::STRING::JSONB), ARRAY[_, _], json_build_object(_, ((extract(_, now()) * _)::INT8 * _)::STRING)) AS _ FROM _) SELECT crdb_internal.unsafe_upsert_descriptor(_, crdb_internal.json_to_pb('_', _), _) FROM _
conn_executor.go:958: *withstack.withStack (1)
(check the extra data payloads)
Stacktrace (expand for inline code snippets):

for status == execinfra.NeedMoreRows {
err := f.next()
if err != nil {

func (f *BatchFlowCoordinator) next() error {
return colexecerror.CatchVectorizedRuntimeError(f.nextAdapter)
}

}()
operation()
return retErr

func (f *BatchFlowCoordinator) nextAdapter() {
f.batch = f.input.Root.Next()
}

// execution time (e.g. in the statement bundle).
err := colexecerror.CatchVectorizedRuntimeError(bic.next)
bic.stopwatch.Stop()

}()
operation()
return retErr

func (bic *batchInfoCollector) next() {
bic.batch = bic.Input.Next()
}

func (d *simpleProjectOp) Next() coldata.Batch {
batch := d.Input.Next()
if batch.Length() == 0 {

func (b *defaultBuiltinFuncOperator) Next() coldata.Batch {
batch := b.Input.Next()
n := batch.Length()

func (e *vectorTypeEnforcer) Next() coldata.Batch {
b := e.Input.Next()
if b.Length() == 0 {

https://github.com/cockroachdb/cockroach/blob/c8459c085e2258f589949e970a527a0342871276/bazel-out/k8-opt/bin/pkg/sql/colexec/colexecbase/const.eg.go#L168-L170
func (e *vectorTypeEnforcer) Next() coldata.Batch {
b := e.Input.Next()
if b.Length() == 0 {

output := batch.ColVec(b.outputIdx)
b.allocator.PerformOperation(
[]coldata.Vec{output},

// caused by this approach is ok.
operation()
after := getVecsMemoryFootprint(destVecs)

res, err = b.funcExpr.ResolvedOverload().
Fn.(eval.FnOverload)(b.Ctx, b.evalCtx, b.row)
if err != nil {

}
data, err := protoreflect.JSONBMarshalToMessage(tree.MustBeDJSON(args[1]).JSON, msg)
if err != nil {

}
data, err := protoutil.Marshal(target)
if err != nil {

https://github.com/cockroachdb/cockroach/blob/c8459c085e2258f589949e970a527a0342871276/pkg/util/protoutil/pkg/util/protoutil/marshal.go#L27-L29
https://github.com/cockroachdb/cockroach/blob/c8459c085e2258f589949e970a527a0342871276/pkg/util/protoutil/pkg/util/protoutil/marshal.go#L58-L60
https://github.com/cockroachdb/cockroach/blob/c8459c085e2258f589949e970a527a0342871276/pkg/sql/catalog/descpb/bazel-out/k8-opt/bin/pkg/sql/catalog/descpb/descpb_go_proto_/github.com/cockroachdb/cockroach/pkg/sql/catalog/descpb/structured.pb.go#L10255-L10257
https://github.com/cockroachdb/cockroach/blob/c8459c085e2258f589949e970a527a0342871276/pkg/sql/catalog/descpb/bazel-out/k8-opt/bin/pkg/sql/catalog/descpb/descpb_go_proto_/github.com/cockroachdb/cockroach/pkg/sql/catalog/descpb/structured.pb.go#L10265-L10267
https://github.com/cockroachdb/cockroach/blob/c8459c085e2258f589949e970a527a0342871276/pkg/sql/catalog/descpb/bazel-out/k8-opt/bin/pkg/sql/catalog/descpb/descpb_go_proto_/github.com/cockroachdb/cockroach/pkg/sql/catalog/descpb/structured.pb.go#L10272-L10274
https://github.com/cockroachdb/cockroach/blob/c8459c085e2258f589949e970a527a0342871276/pkg/sql/catalog/descpb/bazel-out/k8-opt/bin/pkg/sql/catalog/descpb/descpb_go_proto_/github.com/cockroachdb/cockroach/pkg/sql/catalog/descpb/structured.pb.go#L8609-L8611
https://github.com/cockroachdb/cockroach/blob/c8459c085e2258f589949e970a527a0342871276/pkg/sql/catalog/descpb/bazel-out/k8-opt/bin/pkg/sql/catalog/descpb/descpb_go_proto_/github.com/cockroachdb/cockroach/pkg/sql/catalog/descpb/structured.pb.go#L8754-L8756
GOROOT/src/runtime/signal_unix.go#L880-L882
GOROOT/src/runtime/panic.go#L260-L262
GOROOT/src/runtime/panic.go#L769-L771
// an illegal state that might persist even after this statement finishes.
panic(panicObj)
}

GOROOT/src/runtime/panic.go#L769-L771
// an illegal state that might persist even after this statement finishes.
panic(panicObj)
}

GOROOT/src/runtime/panic.go#L769-L771
r := recover()
h.ex.closeWrapper(ctx, r)
}(ctx, h)

pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*BatchFlowCoordinator).Run at line 281
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*BatchFlowCoordinator).next at line 249
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 145
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*BatchFlowCoordinator).nextAdapter at line 245
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).Next at line 121
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError at line 145
pkg/sql/colflow/stats.go in pkg/sql/colflow.(*batchInfoCollector).next at line 113
pkg/sql/colexec/colexecbase/simple_project.go in pkg/sql/colexec/colexecbase.(*simpleProjectOp).Next at line 113
pkg/sql/colexec/builtin_funcs.go in pkg/sql/colexec.(*defaultBuiltinFuncOperator).Next at line 41
pkg/sql/colexec/colexecutils/operator.go in pkg/sql/colexec/colexecutils.(*vectorTypeEnforcer).Next at line 152
bazel-out/k8-opt/bin/pkg/sql/colexec/colexecbase/const.eg.go in pkg/sql/colexec/colexecbase.constBoolOp.Next at line 169
pkg/sql/colexec/colexecutils/operator.go in pkg/sql/colexec/colexecutils.(*vectorTypeEnforcer).Next at line 152
pkg/sql/colexec/builtin_funcs.go in pkg/sql/colexec.(*defaultBuiltinFuncOperator).Next at line 49
pkg/sql/colmem/allocator.go in pkg/sql/colmem.(*Allocator).PerformOperation at line 455
pkg/sql/colexec/builtin_funcs.go in pkg/sql/colexec.(*defaultBuiltinFuncOperator).Next.func1 at line 72
pkg/sql/sem/builtins/builtins.go in pkg/sql/sem/builtins.init.func591 at line 4348
pkg/sql/protoreflect/utils.go in pkg/sql/protoreflect.JSONBMarshalToMessage at line 125
pkg/util/protoutil/pkg/util/protoutil/marshal.go in pkg/util/protoutil.Marshal at line 28
pkg/util/protoutil/pkg/util/protoutil/marshal.go in pkg/util/protoutil.MarshalToSizedBuffer at line 59
pkg/sql/catalog/descpb/bazel-out/k8-opt/bin/pkg/sql/catalog/descpb/descpb_go_proto_/github.com/cockroachdb/cockroach/pkg/sql/catalog/descpb/structured.pb.go in pkg/sql/catalog/descpb.(*Descriptor).MarshalToSizedBuffer at line 10256
pkg/sql/catalog/descpb/bazel-out/k8-opt/bin/pkg/sql/catalog/descpb/descpb_go_proto_/github.com/cockroachdb/cockroach/pkg/sql/catalog/descpb/structured.pb.go in pkg/sql/catalog/descpb.(*Descriptor_Table).MarshalTo at line 10266
pkg/sql/catalog/descpb/bazel-out/k8-opt/bin/pkg/sql/catalog/descpb/descpb_go_proto_/github.com/cockroachdb/cockroach/pkg/sql/catalog/descpb/structured.pb.go in pkg/sql/catalog/descpb.(*Descriptor_Table).MarshalToSizedBuffer at line 10273
pkg/sql/catalog/descpb/bazel-out/k8-opt/bin/pkg/sql/catalog/descpb/descpb_go_proto_/github.com/cockroachdb/cockroach/pkg/sql/catalog/descpb/structured.pb.go in pkg/sql/catalog/descpb.(*TableDescriptor).MarshalToSizedBuffer at line 8610
pkg/sql/catalog/descpb/bazel-out/k8-opt/bin/pkg/sql/catalog/descpb/descpb_go_proto_/github.com/cockroachdb/cockroach/pkg/sql/catalog/descpb/structured.pb.go in pkg/sql/catalog/descpb.(*TableDescriptor_CheckConstraint).MarshalToSizedBuffer at line 8755
GOROOT/src/runtime/signal_unix.go in runtime.sigpanic at line 881
GOROOT/src/runtime/panic.go in runtime.panicmem at line 261
GOROOT/src/runtime/panic.go in runtime.gopanic at line 770
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1 at line 120
GOROOT/src/runtime/panic.go in runtime.gopanic at line 770
pkg/sql/colexecerror/error.go in pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1 at line 120
GOROOT/src/runtime/panic.go in runtime.gopanic at line 770
pkg/sql/conn_executor.go in pkg/sql.(*Server).ServeConn.func1 at line 958

Tags

Tag Value
Command server
Environment v24.1.14
Go Version go1.22.5 X:nocoverageredesign
Platform linux amd64
Distribution CCL
Cockroach Release v24.1.14
Cockroach SHA c8459c0
# of CPUs 4
# of Goroutines 528

Jira issue: CRDB-48644

@cockroach-sentry cockroach-sentry added branch-release-24.1 Used to mark GA and release blockers, technical advisories, and bugs for 24.1 C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. labels Mar 18, 2025
Copy link

blathers-crl bot commented Mar 18, 2025

CC'ing via the CODEOWNERS-based sentry heuristic failed:

issue #143071, cause pkg/sql/catalog/descpb/bazel-out/k8-opt/bin/pkg/sql/catalog/descpb/descpb_go_proto_/github.com/cockroachdb/cockroach/pkg/sql/catalog/descpb/structured.pb.go, owners [{cockroachdb/unowned map[cockroachdb/rfc-prs:other] 0 false}]: didn't find contacts

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

@blathers-crl blathers-crl bot added the X-blathers-untriaged blathers was unable to find an owner label Mar 18, 2025
@yuzefovich yuzefovich added T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) and removed X-blathers-untriaged blathers was unable to find an owner labels Mar 19, 2025
@cockroachdb cockroachdb deleted a comment from blathers-crl bot Mar 19, 2025
@lunevalex
Copy link
Collaborator

blathers auth-check

Copy link
Collaborator

You shall pass

@rail
Copy link
Member

rail commented Mar 19, 2025

blathers auth-check

Copy link
Member

rail commented Mar 19, 2025

You shall pass

@rafiss
Copy link
Collaborator

rafiss commented Mar 25, 2025

@rafiss rafiss closed this as completed Mar 25, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
branch-release-24.1 Used to mark GA and release blockers, technical advisories, and bugs for 24.1 C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)
Projects
None yet
Development

No branches or pull requests

5 participants