Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flyway migration fails with the latest MSSQL RHEL server (most often on RHEL) #46083

Open
michalvavrik opened this issue Feb 4, 2025 · 11 comments
Labels
area/flyway kind/bug-thirdparty Bugs that are caused by third-party components and not causing a major dysfunction of core Quarkus.

Comments

@michalvavrik
Copy link
Member

michalvavrik commented Feb 4, 2025

Describe the bug

I presume this will get closed, but I can't with a good conscience decide for myself that this is 100 % not related to Quarkus. I experience bug only on RHEL8, when the same thing is executed on RHEL9 or Fedora, all is working. Very rarely I can't reproduce it on RHEL8, so maybe it is little flaky, but almost always it fails. And it doesn't happen with non-RHEL SQL server.

TL;DR; migration fails with "connection closed" and I am not sure if it is not related to Agroal or if it is only Flyway & MSSQL issue.

Expected behavior

Migration pass.

Actual behavior

Application fails to start and exception is thrown:

2025-02-04 19:28:12,257 ERROR [org.fly.cor.int.jdb.TransactionalExecutionTemplate] (main) Unable to rollback transaction: com.microsoft.sqlserver.jdbc.SQLServerException: The connection is closed.
	at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDriverError(SQLServerException.java:242)
	at com.microsoft.sqlserver.jdbc.SQLServerConnection.checkClosed(SQLServerConnection.java:1848)
	at com.microsoft.sqlserver.jdbc.SQLServerConnection.rollback(SQLServerConnection.java:4722)
	at io.agroal.pool.wrapper.ConnectionWrapper.rollback(ConnectionWrapper.java:208)
	at org.flywaydb.core.internal.jdbc.TransactionalExecutionTemplate.execute(TransactionalExecutionTemplate.java:75)
	at org.flywaydb.core.internal.command.DbMigrate.applyMigrations(DbMigrate.java:282)
	at org.flywaydb.core.internal.command.DbMigrate.migrateGroup(DbMigrate.java:255)
	at org.flywaydb.core.internal.command.DbMigrate.lambda$migrateAll$0(DbMigrate.java:153)
	at org.flywaydb.database.sqlserver.SQLServerApplicationLockTemplate.execute(SQLServerApplicationLockTemplate.java:65)
	at org.flywaydb.database.sqlserver.SQLServerConnection.lock(SQLServerConnection.java:115)
	at org.flywaydb.core.internal.schemahistory.JdbcTableSchemaHistory.lock(JdbcTableSchemaHistory.java:149)
	at org.flywaydb.core.internal.command.DbMigrate.migrateAll(DbMigrate.java:153)
	at org.flywaydb.core.internal.command.DbMigrate.migrate(DbMigrate.java:104)
	at org.flywaydb.core.Flyway.lambda$migrate$1(Flyway.java:232)
	at org.flywaydb.core.FlywayExecutor.execute(FlywayExecutor.java:213)
	at org.flywaydb.core.Flyway.migrate(Flyway.java:178)
	at io.quarkus.flyway.runtime.FlywayRecorder.doStartActions(FlywayRecorder.java:142)
	at io.quarkus.runner.recorded.FlywayProcessor$startActions2099152139.deploy_0(Unknown Source)
	at io.quarkus.runner.recorded.FlywayProcessor$startActions2099152139.deploy(Unknown Source)
	at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source)
	at io.quarkus.runtime.Application.start(Application.java:101)
	at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:121)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:77)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:48)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:137)
	at io.quarkus.runner.GeneratedMain.main(Unknown Source)
	at io.quarkus.bootstrap.runner.QuarkusEntryPoint.doRun(QuarkusEntryPoint.java:68)
	at io.quarkus.bootstrap.runner.QuarkusEntryPoint.main(QuarkusEntryPoint.java:36)

2025-02-04 19:28:12,257 ERROR [org.fly.cor.int.jdb.TransactionalExecutionTemplate] (main) Unable to restore autocommit to original value for connection: com.microsoft.sqlserver.jdbc.SQLServerException: The connection is closed.
	at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDriverError(SQLServerException.java:242)
	at com.microsoft.sqlserver.jdbc.SQLServerConnection.checkClosed(SQLServerConnection.java:1848)
	at com.microsoft.sqlserver.jdbc.SQLServerConnection.getAutoCommit(SQLServerConnection.java:4673)
	at io.agroal.pool.wrapper.ConnectionWrapper.setAutoCommit(ConnectionWrapper.java:162)
	at org.flywaydb.core.internal.jdbc.TransactionalExecutionTemplate.execute(TransactionalExecutionTemplate.java:90)
	at org.flywaydb.core.internal.command.DbMigrate.applyMigrations(DbMigrate.java:282)
	at org.flywaydb.core.internal.command.DbMigrate.migrateGroup(DbMigrate.java:255)
	at org.flywaydb.core.internal.command.DbMigrate.lambda$migrateAll$0(DbMigrate.java:153)
	at org.flywaydb.database.sqlserver.SQLServerApplicationLockTemplate.execute(SQLServerApplicationLockTemplate.java:65)
	at org.flywaydb.database.sqlserver.SQLServerConnection.lock(SQLServerConnection.java:115)
	at org.flywaydb.core.internal.schemahistory.JdbcTableSchemaHistory.lock(JdbcTableSchemaHistory.java:149)
	at org.flywaydb.core.internal.command.DbMigrate.migrateAll(DbMigrate.java:153)
	at org.flywaydb.core.internal.command.DbMigrate.migrate(DbMigrate.java:104)
	at org.flywaydb.core.Flyway.lambda$migrate$1(Flyway.java:232)
	at org.flywaydb.core.FlywayExecutor.execute(FlywayExecutor.java:213)
	at org.flywaydb.core.Flyway.migrate(Flyway.java:178)
	at io.quarkus.flyway.runtime.FlywayRecorder.doStartActions(FlywayRecorder.java:142)
	at io.quarkus.runner.recorded.FlywayProcessor$startActions2099152139.deploy_0(Unknown Source)
	at io.quarkus.runner.recorded.FlywayProcessor$startActions2099152139.deploy(Unknown Source)
	at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source)
	at io.quarkus.runtime.Application.start(Application.java:101)
	at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:121)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:77)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:48)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:137)
	at io.quarkus.runner.GeneratedMain.main(Unknown Source)
	at io.quarkus.bootstrap.runner.QuarkusEntryPoint.doRun(QuarkusEntryPoint.java:68)
	at io.quarkus.bootstrap.runner.QuarkusEntryPoint.main(QuarkusEntryPoint.java:36)

2025-02-04 19:28:12,258 ERROR [org.fly.cor.int.com.DbMigrate] (main) Migration of schema [msdb] to version "1.0.1 - load default data" failed! Changes successfully rolled back.
2025-02-04 19:28:12,258 ERROR [org.fly.dat.sql.SQLServerApplicationLockTemplate] (main) Unable to release SQL Server application lock: com.microsoft.sqlserver.jdbc.SQLServerException: The connection is closed.
	at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDriverError(SQLServerException.java:242)
	at com.microsoft.sqlserver.jdbc.SQLServerConnection.checkClosed(SQLServerConnection.java:1848)
	at com.microsoft.sqlserver.jdbc.SQLServerConnection.prepareStatement(SQLServerConnection.java:5013)
	at com.microsoft.sqlserver.jdbc.SQLServerConnection.prepareStatement(SQLServerConnection.java:4620)
	at io.agroal.pool.wrapper.ConnectionWrapper.prepareStatement(ConnectionWrapper.java:647)
	at org.flywaydb.core.internal.jdbc.JdbcTemplate.prepareStatement(JdbcTemplate.java:313)
	at org.flywaydb.core.internal.jdbc.JdbcTemplate.execute(JdbcTemplate.java:189)
	at org.flywaydb.database.sqlserver.SQLServerConnection.setCurrentDatabase(SQLServerConnection.java:91)
	at org.flywaydb.database.sqlserver.SQLServerApplicationLockTemplate.execute(SQLServerApplicationLockTemplate.java:78)
	at org.flywaydb.database.sqlserver.SQLServerConnection.lock(SQLServerConnection.java:115)
	at org.flywaydb.core.internal.schemahistory.JdbcTableSchemaHistory.lock(JdbcTableSchemaHistory.java:149)
	at org.flywaydb.core.internal.command.DbMigrate.migrateAll(DbMigrate.java:153)
	at org.flywaydb.core.internal.command.DbMigrate.migrate(DbMigrate.java:104)
	at org.flywaydb.core.Flyway.lambda$migrate$1(Flyway.java:232)
	at org.flywaydb.core.FlywayExecutor.execute(FlywayExecutor.java:213)
	at org.flywaydb.core.Flyway.migrate(Flyway.java:178)
	at io.quarkus.flyway.runtime.FlywayRecorder.doStartActions(FlywayRecorder.java:142)
	at io.quarkus.runner.recorded.FlywayProcessor$startActions2099152139.deploy_0(Unknown Source)
	at io.quarkus.runner.recorded.FlywayProcessor$startActions2099152139.deploy(Unknown Source)
	at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source)
	at io.quarkus.runtime.Application.start(Application.java:101)
	at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:121)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:77)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:48)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:137)
	at io.quarkus.runner.GeneratedMain.main(Unknown Source)
	at io.quarkus.bootstrap.runner.QuarkusEntryPoint.doRun(QuarkusEntryPoint.java:68)
	at io.quarkus.bootstrap.runner.QuarkusEntryPoint.main(QuarkusEntryPoint.java:36)

2025-02-04 19:28:12,274 ERROR [io.qua.run.Application] (main) Failed to start application: java.lang.RuntimeException: Failed to start quarkus
	at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source)
	at io.quarkus.runtime.Application.start(Application.java:101)
	at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:121)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:77)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:48)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:137)
	at io.quarkus.runner.GeneratedMain.main(Unknown Source)
	at io.quarkus.bootstrap.runner.QuarkusEntryPoint.doRun(QuarkusEntryPoint.java:68)
	at io.quarkus.bootstrap.runner.QuarkusEntryPoint.main(QuarkusEntryPoint.java:36)
Caused by: org.flywaydb.core.internal.exception.FlywayMigrateException: Script V1.0.1__load_default_data.sql failed
-------------------------------------------
SQL State  : 08S01
Error Code : 0
Message    : SQL Server did not return a response. The connection has been closed.
Location   : db/migration/V1.0.1__load_default_data.sql (/home/hudson/test2/test2/code-with-quarkus/file:/home/hudson/test2/test2/code-with-quarkus/target/quarkus-app/app/code-with-quarkus-1.0.0-SNAPSHOT.jar!/db/migration/V1.0.1__load_default_data.sql)
Line       : 1
Statement  : Run Flyway with -X option to see the actual statement causing the problem

	at org.flywaydb.core.internal.command.DbMigrate.doMigrateGroup(DbMigrate.java:399)
	at org.flywaydb.core.internal.command.DbMigrate.lambda$applyMigrations$1(DbMigrate.java:283)
	at org.flywaydb.core.internal.jdbc.TransactionalExecutionTemplate.execute(TransactionalExecutionTemplate.java:59)
	at org.flywaydb.core.internal.command.DbMigrate.applyMigrations(DbMigrate.java:282)
	at org.flywaydb.core.internal.command.DbMigrate.migrateGroup(DbMigrate.java:255)
	at org.flywaydb.core.internal.command.DbMigrate.lambda$migrateAll$0(DbMigrate.java:153)
	at org.flywaydb.database.sqlserver.SQLServerApplicationLockTemplate.execute(SQLServerApplicationLockTemplate.java:65)
	at org.flywaydb.database.sqlserver.SQLServerConnection.lock(SQLServerConnection.java:115)
	at org.flywaydb.core.internal.schemahistory.JdbcTableSchemaHistory.lock(JdbcTableSchemaHistory.java:149)
	at org.flywaydb.core.internal.command.DbMigrate.migrateAll(DbMigrate.java:153)
	at org.flywaydb.core.internal.command.DbMigrate.migrate(DbMigrate.java:104)
	at org.flywaydb.core.Flyway.lambda$migrate$1(Flyway.java:232)
	at org.flywaydb.core.FlywayExecutor.execute(FlywayExecutor.java:213)
	at org.flywaydb.core.Flyway.migrate(Flyway.java:178)
	at io.quarkus.flyway.runtime.FlywayRecorder.doStartActions(FlywayRecorder.java:142)
	at io.quarkus.runner.recorded.FlywayProcessor$startActions2099152139.deploy_0(Unknown Source)
	at io.quarkus.runner.recorded.FlywayProcessor$startActions2099152139.deploy(Unknown Source)
	... 9 more
Caused by: org.flywaydb.core.internal.sqlscript.FlywaySqlScriptException: Script V1.0.1__load_default_data.sql failed
-------------------------------------------
SQL State  : 08S01
Error Code : 0
Message    : SQL Server did not return a response. The connection has been closed.
Location   : db/migration/V1.0.1__load_default_data.sql (/home/hudson/test2/test2/code-with-quarkus/file:/home/hudson/test2/test2/code-with-quarkus/target/quarkus-app/app/code-with-quarkus-1.0.0-SNAPSHOT.jar!/db/migration/V1.0.1__load_default_data.sql)
Line       : 1
Statement  : Run Flyway with -X option to see the actual statement causing the problem

	at org.flywaydb.core.internal.sqlscript.DefaultSqlScriptExecutor.handleException(DefaultSqlScriptExecutor.java:256)
	at org.flywaydb.core.internal.sqlscript.DefaultSqlScriptExecutor.executeStatement(DefaultSqlScriptExecutor.java:217)
	at org.flywaydb.core.internal.sqlscript.DefaultSqlScriptExecutor.execute(DefaultSqlScriptExecutor.java:137)
	at org.flywaydb.core.internal.resolver.sql.SqlMigrationExecutor.executeOnce(SqlMigrationExecutor.java:75)
	at org.flywaydb.core.internal.resolver.sql.SqlMigrationExecutor.lambda$execute$0(SqlMigrationExecutor.java:66)
	at org.flywaydb.core.internal.database.DefaultExecutionStrategy.execute(DefaultExecutionStrategy.java:31)
	at org.flywaydb.core.internal.resolver.sql.SqlMigrationExecutor.execute(SqlMigrationExecutor.java:65)
	at org.flywaydb.core.internal.command.DbMigrate.doMigrateGroup(DbMigrate.java:391)
	... 25 more
Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: SQL Server did not return a response. The connection has been closed.
	at com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:4271)
	at com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:4260)
	at com.microsoft.sqlserver.jdbc.TDSReader.readPacket(IOBuffer.java:6902)
	at com.microsoft.sqlserver.jdbc.TDSCommand.startResponse(IOBuffer.java:8145)
	at com.microsoft.sqlserver.jdbc.SQLServerStatement.doExecuteStatement(SQLServerStatement.java:927)
	at com.microsoft.sqlserver.jdbc.SQLServerStatement$StmtExecCmd.doExecute(SQLServerStatement.java:823)
	at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7745)
	at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:4391)
	at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:276)
	at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:246)
	at com.microsoft.sqlserver.jdbc.SQLServerStatement.execute(SQLServerStatement.java:796)
	at io.agroal.pool.wrapper.StatementWrapper.execute(StatementWrapper.java:220)
	at org.flywaydb.core.internal.jdbc.JdbcTemplate.executeStatement(JdbcTemplate.java:210)
	at org.flywaydb.core.internal.sqlscript.ParsedSqlStatement.execute(ParsedSqlStatement.java:88)
	at org.flywaydb.core.internal.sqlscript.DefaultSqlScriptExecutor.executeStatement(DefaultSqlScriptExecutor.java:212)
	... 31 more

How to Reproduce?

Steps to reproduce the behavior:

# for success
docker run -e "ACCEPT_EULA=Y" -e "MSSQL_SA_PASSWORD=My1337p@ssworD" -p 1433:1433 -d mcr.microsoft.com/mssql/server:2022-latest
# for failure
docker run -e "ACCEPT_EULA=Y" -e "MSSQL_SA_PASSWORD=My1337p@ssworD" -p 1433:1433 -d mcr.microsoft.com/mssql/rhel/server:2022-latest

quarkus create app -x=jdbc-mssql,flyway
cd code-with-quarkus/
sed -i -e "s|<artifactId>quarkus-flyway</artifactId>|<artifactId>quarkus-flyway</artifactId></dependency><dependency><groupId>org.flywaydb</groupId><artifactId>flyway-sqlserver</artifactId>|" pom.xml
cat << 'EOF' > ./src/main/resources/application.properties
quarkus.datasource.db-kind=mssql
quarkus.datasource.jdbc.additional-jdbc-properties.trustServerCertificate=true
quarkus.datasource.username=sa
quarkus.datasource.password=My1337p@ssworD
quarkus.flyway.schemas=msdb
quarkus.flyway.migrate-at-start=true
quarkus.flyway.locations=db/migration
quarkus.datasource.jdbc.url=jdbc:sqlserver://localhost:1433
EOF
mkdir -p src/main/resources/db/migration
wget -O src/main/resources/db/migration/V1.0.1__load_default_data.sql https://raw.githubusercontent.com/quarkus-qe/quarkus-test-suite/refs/heads/main/sql-db/vertx-sql/src/main/resources/db/migration/common/V1.0.1__load_default_data.sql
wget -O src/main/resources/db/migration/V1.0.0__init.sql https://raw.githubusercontent.com/quarkus-qe/quarkus-test-suite/refs/heads/main/sql-db/vertx-sql/src/main/resources/db/migration/mssql/V1.0.0__init.sql
mvn clean package
java -jar target/quarkus-app/quarkus-run.jar

There are other ways to reproduce this, for example you can run Quarkus QE Test Suite vertx-sql module https://github.com/quarkus-qe/quarkus-test-suite/actions/runs/13191616175/job/36825489234.

Output of uname -a or ver

Red Hat Enterprise Linux release 8.10 (Ootpa)

Output of java -version

OpenJDK Runtime Environment (Red_Hat-17.0.12.0.7-1) (build 17.0.12+7-LTS)

Quarkus version or git rev

3.15,3.18,999-SNAPSHOT (in short: all I tried)

Build tool (ie. output of mvnw --version or gradlew --version)

Apache Maven 3.9.9

Additional information

Docker container image: mcr.microsoft.com/mssql/rhel/server@sha256:af5b907277cbe8f82a8986d7abd5c954459cbc77a0358ad0eab80471b53e311b

Works with older RHEL-based SQL Server images like http://mcr.microsoft.com/mssql/rhel/server:2022-CU10-rhel-9.1

Feel free to close the issue.

@michalvavrik michalvavrik added area/flyway kind/bug-thirdparty Bugs that are caused by third-party components and not causing a major dysfunction of core Quarkus. labels Feb 4, 2025
Copy link

quarkus-bot bot commented Feb 4, 2025

/cc @cristhiank (flyway), @gastaldi (flyway), @geoand (flyway), @gsmet (flyway)

@rsvoboda
Copy link
Member

rsvoboda commented Feb 4, 2025

@michalvavrik there was an idea that the failure was reproducible on JDK 17 and not on 21. Was that confirmed or JDK version doesn't have any effect?

@michalvavrik
Copy link
Member Author

@michalvavrik there was an idea that the failure was reproducible on JDK 17 and not on 21. Was that confirmed or JDK version doesn't have any effect?

I never said anything like that, I tried every possible combination, it is not related to JDK version.

@michalvavrik
Copy link
Member Author

Maybe @jcarranzan mentioned JDK version? anyway, not related.

@michalvavrik
Copy link
Member Author

I found it, you are referring to passing JDK 21 Job, I tried it and reproduced it with 21. I think there is a race somewhere and even a little things can influence that. But the reproducer I provider is stable, because I only failed to reproduce it once in many runs. I tried it from scratch with new instances more than once.

@michalvavrik
Copy link
Member Author

I rerun that job created by @jcarranzan and it failed with 21 as well. Sending you link via DM.

@jcarranzan
Copy link
Contributor

When I tested last week, this failed for openJDK17 and passed with openJDK21, that's what I said. But probably as @michalvavrik mentioned, it is possible that it was a race issue and it confused me...
Anyway, I also see in another (different) issue with Mssql that failed on openJDK17 (#40813).
Thank you @michalvavrik for investigating and get reproducing it

@rsvoboda
Copy link
Member

rsvoboda commented Feb 5, 2025

Thanks Michal for the thorough checks and confirmation it's not JDK related.

@michalvavrik
Copy link
Member Author

Updated reproducer, now failure also happens sometimes in GH CI, so it is combination of Ubuntu and RHEL9

@michalvavrik michalvavrik changed the title Flyway migration fails with the latest MSSQL RHEL server only on RHEL8 Flyway migration fails with the latest MSSQL RHEL server (most often on RHEL) Feb 7, 2025
@gsmet
Copy link
Member

gsmet commented Feb 7, 2025

TBH, I'm not sure we can do anything about this on the Quarkus side.

@michalvavrik
Copy link
Member Author

TBH, I'm not sure we can do anything about this on the Quarkus side.

I expected that, I just needed to check. Once you are certain, please close this issue. Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/flyway kind/bug-thirdparty Bugs that are caused by third-party components and not causing a major dysfunction of core Quarkus.
Projects
None yet
Development

No branches or pull requests

4 participants