The changes in the README.md file involve reordering and updating the status of various vector databases in the "In Progress" and "Not Supported" sections.

The changes introduce the ExportPGVector class to handle exporting data from PGVector tables in a PostgreSQL database, including methods for argument parsing, data retrieval, and metadata generation.

Sweep Found These Issues

• The get_all_schemas and get_all_table_names methods use self.conn.execute which is not a valid method for a psycopg2 connection object; it should be self.conn.cursor().execute.

vector-io/src/vdf_io/export_vdf/pgvector_export.py

Lines 69 to 81 in cb7e7ff

	def get_all_schemas(self):
	schemas = self.conn.execute(
	"SELECT schema_name FROM information_schema.schemata"
	)
	self.all_schemas = [schema[0] for schema in schemas]
	return [schema[0] for schema in schemas]
	def get_all_table_names(self):
	tables = self.conn.execute(
	"SELECT table_name FROM information_schema.tables WHERE table_schema='public'"
	)
	self.all_tables = [table[0] for table in tables]
	return [table[0] for table in tables]


View Diff

Introduced a new class ImportPGVector for importing data into a PGVector database, including methods for database connection, schema and table retrieval, and data upsertion from Parquet files.

Sweep Found These Issues

• The get_all_schemas and get_all_table_names methods use self.conn.execute which is not a valid method for a psycopg2 connection object; it should use a cursor object to execute SQL queries.

vector-io/src/vdf_io/import_vdf/pgvector_import.py

Lines 77 to 87 in cb7e7ff

	schemas = self.conn.execute(
	"SELECT schema_name FROM information_schema.schemata"
	)
	self.all_schemas = [schema[0] for schema in schemas]
	return [schema[0] for schema in schemas]
	def get_all_table_names(self):
	tables = self.conn.execute(
	"SELECT table_name FROM information_schema.tables WHERE table_schema='public'"
	)
	self.all_tables = [table[0] for table in tables]


View Diff
• The upsert_data method assumes that self.vdf_meta is already populated, but there is no code to load or initialize this attribute, which may lead to AttributeError.

vector-io/src/vdf_io/import_vdf/pgvector_import.py

Lines 96 to 99 in cb7e7ff

	indexes_content: Dict[str, List[NamespaceMeta]] = self.vdf_meta["indexes"]
	index_names: List[str] = list(indexes_content.keys())
	if len(index_names) == 0:
	raise ValueError("No indexes found in VDF_META.json")


View Diff

A new class attribute PGVECTOR was added to the DBNames class to include the "pgvector" database.

The changes include reordering the execution of code cells, updating the file path for loading data, modifying the DataFrame display headers and data, and adding new functionality to calculate DataFrame length, display a summary, add a new column, and save the DataFrame as a Parquet file.

Sweep Found These Issues

• The change in the file path for the jsonl_file variable may introduce issues if the new file does not exist or is not accessible, leading to a FileNotFoundError or similar error.

vector-io/src/vdf_io/notebooks/jsonl_to_parquet.ipynb

Line 38 in cb7e7ff

"jsonl_file = '/Users/dhruvanand/Downloads/579aac087579b5acbc881164eb7af8b8-662d020154513ab1ef571e325a8fc649ebf64561/o200k_tokens.jsonl'\n"


View Diff

Introduced a new module pgvector_util.py with functions to create a command-line argument parser and prompt for Postgres connection details.

Sweep Found These Issues

• The function set_pgv_args_from_prompt does not validate the connection string format, which could lead to runtime errors if an invalid connection string is provided.

vector-io/src/vdf_io/pgvector_util.py

Lines 42 to 47 in cb7e7ff

	set_arg_from_input(
	args,
	"connection_string",
	"Enter the connection string to Postgres instance: ",
	str,
	)


View Diff
• The default password "postgres" is set if the user does not provide one, which could be a security risk.

vector-io/src/vdf_io/pgvector_util.py

Lines 61 to 63 in cb7e7ff

	if not args.get("password"):
	# If password is not provided, set it to "postgres"
	args["password"] = "postgres"


View Diff