[SPARK-51660][CORE] Gracefully handle when MDC is not supported

[robreeves]

What changes were proposed in this pull request?

This improves the handling when MDC is not supported. The previous handling only works when one task runs on an executor. After this fix multiple tasks can run on an executor when MDC is not supported.

Why are the changes needed?

In #35141 it added handling to gracefully handle when MDC is not available. It catches NoSuchFieldError, which is thrown during MDC initialization. This will work for the first task. If a second task runs it will not throw a NoSuchFieldError. Instead it throws NoClassDefFoundError. Then the job will hang indefinitely. This improves the fix so it works for all tasks instead of just the first one.

Does this PR introduce any user-facing change?

Yes, it fixes a bug. It also changes a log message to include the exception for easier debugging.

How was this patch tested?

I tested it manually in spark-shell by manually throwing NoSuchFieldError. It logs this message and does not hang.

25/03/28 14:11:48 INFO Executor: MDC is not supported.
java.lang.NoSuchFieldError: this is a test
    at org.apache.spark.executor.Executor.liftedTree1$1(Executor.scala:118)
    at org.apache.spark.executor.Executor.mdcIsSupported$lzycompute(Executor.scala:114)
    at org.apache.spark.executor.Executor.mdcIsSupported(Executor.scala:113)
    at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$setMDCForTask(Executor.scala:948)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:584)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:833)

Was this patch authored or co-authored using generative AI tooling?

No

[robreeves]

@dongjoon-hyun please take a look

[mridulm]

The fix looks good to me.
+CC @dongjoon-hyun as well.

[robreeves]

@dongjoon-hyun gentle ping on this

[robreeves]

@viirya can you take a look since you reviewed the original PR #35141?

[mridulm]

Merged to master.
Thanks for fixing this @robreeves !