Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PS-9328: Fix for failures to shutdown server after execution of threapool_debug test (8.4 version). #5426

Merged
merged 2 commits into from
Sep 12, 2024

Conversation

dlenev
Copy link
Contributor

@dlenev dlenev commented Sep 12, 2024

Problem:

MTR was unable to properly shutdown Percona Server after execution of
threadpool_debug test and had to resort to killing it with SIGKILL.

Analysis:

threadpool_debug test simulates OOM situation when creating new THD objects
while processing new connections using threadpool plugin. This is done
by injecting and error in Thread_pool_connection_handler::add_connection()
method. The problem occurred because we failed to correctly decrement
connections count before returning from this method in case of error, like
it is done in add_connection() methods for other connection handlers.
As result during server shutdown we tried to wait indefinitely until
total number of connections will become 0, which never happened since
the connections which we failed to create due to simulated OOM were
counted as existing (even though they didn't really).

Note that the above means that failure to properly shutdown probably could
have been observed on user systems which use threadpool plugin in cases when
establishment of new connections failed due to OOM.
OTOH Percona Server/MySQL is likely to hit more serious problems in case of
OOM anyway.

Solution:

Thread_pool_connection_handler::add_connection() now decrements connection
count in case of its failure as implementations of add_connection() for other
connection handlers do.

Fix is applied to both 8.0 and 8.4 as both branches are affected by this issue.

…dpool_debug test.

Problem:
========
MTR was unable to properly shutdown Percona Server after execution of
threadpool_debug test and had to resort to killing it with SIGKILL.

Analysis:
=========
threadpool_debug test simulates OOM situation when creating new THD objects
while processing new connections using threadpool plugin. This is done
by injecting and error in Thread_pool_connection_handler::add_connection()
method. The problem occurred because we failed to correctly decrement
connections count before returning from this method in case of error, like
it is done in add_connection() methods for other connection handlers.
As result during server shutdown we tried to wait indefinitely until
total number of connections will become 0, which never happened since
the connections which we failed to create due to simulated OOM were
counted as existing (even though they didn't really).

Note that the above means that failure to properly shutdown probably could
have been observed on user systems which use threadpool plugin in cases when
establishment of new connections failed due to OOM.
OTOH Percona Server/MySQL is likely to hit more serious problems in case of
OOM anyway.

Solution:
=========
Thread_pool_connection_handler::add_connection() now decrements connection
count in case of its failure as implementations of add_connection() for other
connection handlers do.

Fix is applied to both 8.0 and 8.4 as both branches are affected by this issue.
@dlenev dlenev requested review from percona-ysorokin and removed request for percona-ysorokin September 12, 2024 16:15
@dlenev
Copy link
Contributor Author

dlenev commented Sep 12, 2024

See @percona-ysorokin approval for #5425

@dlenev dlenev merged commit 4b00629 into percona:release-8.4.2-2 Sep 12, 2024
10 of 24 checks passed
@dlenev dlenev deleted the ps-8.4-threadpool_debug-fix branch September 12, 2024 17:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant