unexcepted org.eclipse.jetty.io.EofException is thrown when server serving requests

[zhaochun-ma]

Jetty version(s)

9.4.54

Jetty Environment

Java version/vendor (use: java -version)
java 11

OS type/version
linux

Description

How to reproduce?

the below stacktrace was logged in the application when handling a request. The application was under heavy load at the time and request were experiencing long latency. This org.eclipse.jetty.io.EofException propagates all the way up to one of the ExceptionMappers registered with jetty. We didn't particularly handle this exception, so it was eventually returned as 500 error.

org.eclipse.jetty.io.EofException
	at org.eclipse.jetty.http2.server.HTTP2ServerConnectionFactory$HTTPServerSessionListener.onFailure(HTTP2ServerConnectionFactory.java:167)
	at org.eclipse.jetty.http2.HTTP2Stream.notifyFailure(HTTP2Stream.java:716)
	at org.eclipse.jetty.http2.HTTP2Stream.onFailure(HTTP2Stream.java:434)
	at org.eclipse.jetty.http2.HTTP2Stream.process(HTTP2Stream.java:335)
	at org.eclipse.jetty.http2.HTTP2Session.failStream(HTTP2Session.java:626)
	at org.eclipse.jetty.http2.HTTP2Session.onFailure(HTTP2Session.java:601)
	at org.eclipse.jetty.http2.HTTP2Session.abort(HTTP2Session.java:588)
	at org.eclipse.jetty.http2.HTTP2Session$StreamsState.onShutdown(HTTP2Session.java:1819)
	at org.eclipse.jetty.http2.HTTP2Session$StreamsState.access$1500(HTTP2Session.java:1488)
	at org.eclipse.jetty.http2.HTTP2Session.onShutdown(HTTP2Session.java:973)
	at org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:306)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:362)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:186)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:137)
	at org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:193)
	at org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:148)
	at org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:371)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555)
	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410)
	at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.base/java.lang.Thread.run(Thread.java:840)
	Suppressed: java.lang.Throwable: HttpInput failure
		at org.eclipse.jetty.server.HttpInput.failed(HttpInput.java:895)
		at org.eclipse.jetty.http2.server.HttpChannelOverHTTP2.onFailure(HttpChannelOverHTTP2.java:321)
		at org.eclipse.jetty.http2.server.HTTP2ServerConnection.onStreamFailure(HTTP2ServerConnection.java:209)
		at org.eclipse.jetty.http2.server.HTTP2ServerConnectionFactory$HTTPServerSessionListener.onFailure(HTTP2ServerConnectionFactory.java:173)
		at org.eclipse.jetty.http2.server.HTTP2ServerConnectionFactory$HTTPServerSessionListener.onFailure(HTTP2ServerConnectionFactory.java:168)
		... 31 more
Caused by: java.nio.channels.ClosedChannelException
	at org.eclipse.jetty.http2.HTTP2Session$StreamsState.onShutdown(HTTP2Session.java:1780)
	... 24 more

I did some local debugging to mimic the scenario:
enable jetty server debug level log. start up a client to send a request and stop the client before it receives the response. The same stacktrace was logged with

[2024-04-26 09:08:05,303] DEBUG Commit failed (org.eclipse.jetty.server.HttpChannel:1391)
org.eclipse.jetty.io.EofException
.... <same stacktrace>

but in application log, the exception wasn't caught by ExceptionMappers and the request was returned as HTTP 200.
I'm not sure if we should have an ExceptionMapper to catch this exception.

[zhaochun-ma]

if I run the server with debugger, start up a client to send the same request, I set a breakpoint where the httpconnection is reading from input, while the request is being read by server, I stopped the client, then I continue the server. (this probably simulates the request was read incompletely). This time the EofException propagates all the way up to one of the ExceptionMappers registered with jetty. and 500 was returned.

[2024-04-26 11:01:51,451] ERROR Unhandled exception resulting in internal server error response 
org.eclipse.jetty.io.EofException
	at org.eclipse.jetty.http2.server.HTTP2ServerConnectionFactory$HTTPServerSessionListener.onFailure(HTTP2ServerConnectionFactory.java:167)
	.... same stacktrace ....

I’m not sure what is the expected behavior for jetty here. Should we just add a exception mapper similar to this to handle this kind of exception?: https://github.com/SDA-SE/sda-dropwizard-commons/blob/1464533eda1061cf8d374ac8d511[…]dase/commons/server/jackson/errors/EarlyEofExceptionMapper.java

[janbartel]

@zhaochun-ma jetty-9.4 is at end of community support. In case you missed the announcement, including your options going forward, here it is: https://www.eclipse.org//lists/jetty-announce/msg00168.html

[zhaochun-ma]

While we upgrade our jetty library, is this issue fixed on newer version? would you suggest adding an exception mapper to handle this kind of exception as a temporary workaround?

[joakime]

There have been tens of thousands of commits of difference between Jetty 9.4.x and 12.0.x

We don't have reports of similar concerns in Jetty 12.0.x, that's about as close to an answer we can provide.
We suggest that you upgrade your version of Jetty and don't use/rely on workarounds for End of Community Support versions of Jetty.

Note: Jetty 11 and older are all now either at End of Life or End of Community Support.

• EoCS (End of Community Support) for Jetty 10 / Jetty 11 - January 2024 #10485
• EoCS (End of Community Support) for Jetty 9 - June 2022 #7958
• Jetty Version Table - https://eclipse.dev/jetty/download.php#version-table
• Jetty Security Advisories - https://eclipse.dev/jetty/security_reports.php

You should be running Jetty 12 at this point in time.