[DB] Import or load data from an external CSV file to a table

[authorjapps]

AC1:

Given a datafile in CSV format, matching the Database table columns,
provide mechanism to load these data(rows and columns) in the that table.

Example:

• Table "players" has three columns as below:

ID:
NAME:
AGE:

and data in CSV is as below:

1001, Ronaldo, 23
1002, Devaldo, 24
1003, Trevaldo, 35

Note- There are no headers present in the above CSV

AC2:

Same as AC1, but with headers.

Example:

and data in CSV is as below:

ID, AGE, NAME <----------- 1st row as header
1001, 23, Ronaldo
1002, 24, Devaldo
1003, 35, Trevaldo

AC3 :

Required for Postgres DB (v9 to v12 should be supported )

AC4: (optional)

File could be present in the local files system(eg /test/resources) or in a git repo

[nirmalchandra]

Suggestion: (WITHOUT HEADERS)

Try this for "WITH HEADERS": (replace the relevant line)

String sql = "COPY " + tableName + " FROM STDIN WITH (FORMAT csv, HEADER true)";

import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.io.IOException;
import java.sql.Connection;
import java.sql.SQLException;
import java.util.Objects;

import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.stereotype.Service;
import org.springframework.beans.factory.annotation.Autowired;

@Service
public class ImportCsvToPostgres {

    @Autowired
    private JdbcTemplate jdbcTemplate;

    public void importCsvToTable(String fileName, String tableName) {
        Connection connection = null;
        BufferedReader reader = null;

        // COPY command without the HEADER true option
        String sql = "COPY " + tableName + " FROM STDIN WITH (FORMAT csv)";

        try {
            // Load CSV file from resources
            ClassLoader classLoader = getClass().getClassLoader();
            reader = new BufferedReader(new InputStreamReader(
                    Objects.requireNonNull(classLoader.getResourceAsStream(fileName))));

            // Get PostgreSQL connection and execute the COPY command
            connection = jdbcTemplate.getDataSource().getConnection();
            org.postgresql.copy.CopyManager copyManager = connection.unwrap(org.postgresql.PGConnection.class).getCopyManagerAPI();
            copyManager.copyIn(sql, reader);

        } catch (IOException e) {
            throw new RuntimeException("IOException reading the file from resources: " + fileName, e);
        } catch (SQLException e) {
            throw new RuntimeException("SQLException during COPY command execution", e);
        } finally {
            // Close the resources manually //
            if (reader != null) {
                try {
                    reader.close();
                } catch (IOException e) {
                    e.printStackTrace();
                }
            }
            if (connection != null) {
                try {
                    connection.close();
                } catch (SQLException e) {
                    e.printStackTrace();
                }
            }
        }
    }
}

[authorjapps]

Worth adding to this Page:
https://github.com/authorjapps/zerocode/wiki/Sample-DB-SQL-Executor