Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IMPORT INTO + global sort didn't handle UK conflict correctly #59650

Open
lance6716 opened this issue Feb 19, 2025 · 2 comments · May be fixed by #59659
Open

IMPORT INTO + global sort didn't handle UK conflict correctly #59650

lance6716 opened this issue Feb 19, 2025 · 2 comments · May be fixed by #59659
Labels
affects-8.1 This bug affects the 8.1.x(LTS) versions. affects-8.5 This bug affects the 8.5.x(LTS) versions. severity/major type/bug The issue is confirmed as a bug.

Comments

@lance6716
Copy link
Contributor

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

diff --git a/tests/realtikvtest/importintotest4/global_sort_test.go b/tests/realtikvtest/importintotest4/global_sort_test.go
index 6734ff65ee..3563ad5c89 100644
--- a/tests/realtikvtest/importintotest4/global_sort_test.go
+++ b/tests/realtikvtest/importintotest4/global_sort_test.go
@@ -156,3 +156,28 @@ func (s *mockGCSSuite) TestGlobalSortMultiFiles() {
 	s.tk.MustQuery(importSQL)
 	s.tk.MustQuery("select * from t").Sort().Check(testkit.Rows(allData...))
 }
+
+func (s *mockGCSSuite) TestGlobalSortUniqueKeyConflict() {
+	var allData []string
+	for i := 0; i < 10; i++ {
+		var content []byte
+		keyCnt := 1000
+		for j := 0; j < keyCnt; j++ {
+			idx := i*keyCnt + j
+			content = append(content, []byte(fmt.Sprintf("%d,test-%d\n", idx, j))...)
+		}
+		s.server.CreateObject(fakestorage.Object{
+			ObjectAttrs: fakestorage.ObjectAttrs{BucketName: "gs-multi-files-uk", Name: fmt.Sprintf("t.%d.csv", i)},
+			Content:     content,
+		})
+	}
+	slices.Sort(allData)
+	s.prepareAndUseDB("gs_multi_files")
+	s.server.CreateBucketWithOpts(fakestorage.CreateBucketOpts{Name: "sorted"})
+	s.tk.MustExec("create table t (a bigint primary key , b varchar(100) unique key);")
+	// 1 subtask, encoding 10 files using 4 threads.
+	sortStorageURI := fmt.Sprintf("gs://sorted/gs_multi_files?endpoint=%s", gcsEndpoint)
+	importSQL := fmt.Sprintf(`import into t FROM 'gs://gs-multi-files-uk/t.*.csv?endpoint=%s'
+		with cloud_storage_uri='%s', __max_engine_size='1', thread=8`, gcsEndpoint, sortStorageURI)
+	s.tk.MustQuery(importSQL)
+}

2. What did you expect to see? (Required)

has reasonable output

3. What did you see instead (Required)

in the log we can find

[2025/02/19 20:55:53.969 +08:00] [WARN] [local.go:1255] ["meet retryable error when writing to TiKV"] [category=ddl-ingest] [error="peer 31, store 1, region 30, epoch conf_ver:1 version:11 , when send data: rpc error: code = Unknown desc = EngineTraits(Engine(Status { code: IoError, sub_code: None, sev: NoError, state: \"Invalid argument: Keys must be added in strict ascending order.\" }))"] ["job stage"=needRescan]

4. What is your TiDB version? (Required)

@lance6716
Copy link
Contributor Author

lance6716 commented Feb 19, 2025

should also port #59611 to master

And check if DDL has this problem

@lance6716 lance6716 added affects-8.1 This bug affects the 8.1.x(LTS) versions. affects-8.5 This bug affects the 8.5.x(LTS) versions. labels Feb 19, 2025
@D3Hunter
Copy link
Contributor

D3Hunter commented Feb 19, 2025

Ensure that the data to be imported does not contain any records with primary key or non-null unique index conflicts. Otherwise, the conflicts can result in import task failures.

https://docs.pingcap.com/tidb/dev/sql-statement-import-into

by design, we don't support conflict data, but we should fail with a more clear error in this case

@lance6716 lance6716 linked a pull request Feb 20, 2025 that will close this issue
13 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affects-8.1 This bug affects the 8.1.x(LTS) versions. affects-8.5 This bug affects the 8.5.x(LTS) versions. severity/major type/bug The issue is confirmed as a bug.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants