Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DMS] March release branch, release Thursday 27 #6603

Merged
merged 23 commits into from
Mar 27, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
910235c
DMS: March release branch
gvasquezvargas Mar 11, 2025
03446eb
chore(ET-1872): add config_writer.mdx
yidian1997 Mar 11, 2025
19a6240
Update advocacy_docs/edb-postgres-ai/migration-etl/data-migration-ser…
yidian1997 Mar 12, 2025
00079dc
Update advocacy_docs/edb-postgres-ai/migration-etl/data-migration-ser…
yidian1997 Mar 12, 2025
bbbd1b5
Update advocacy_docs/edb-postgres-ai/migration-etl/data-migration-ser…
yidian1997 Mar 12, 2025
da9b04e
Update advocacy_docs/edb-postgres-ai/migration-etl/data-migration-ser…
yidian1997 Mar 12, 2025
dd29add
Update advocacy_docs/edb-postgres-ai/migration-etl/data-migration-ser…
yidian1997 Mar 12, 2025
2b17183
Update advocacy_docs/edb-postgres-ai/migration-etl/data-migration-ser…
yidian1997 Mar 12, 2025
c942590
Update advocacy_docs/edb-postgres-ai/migration-etl/data-migration-ser…
yidian1997 Mar 12, 2025
54af3ef
Update advocacy_docs/edb-postgres-ai/migration-etl/data-migration-ser…
yidian1997 Mar 12, 2025
a424492
Initial code changes to support CDC Writer installation
dwicinas Mar 12, 2025
c939f91
Removed SLES 12 artifact
dwicinas Mar 13, 2025
a2a8969
Generated file changes needed to support CDC writer
dwicinas Mar 13, 2025
9d46c84
Cross-linked DMS Writer installation to getting started
gvasquezvargas Mar 17, 2025
d720621
Rewording the rest of DMS documentation to include writer information
gvasquezvargas Mar 17, 2025
d550baa
Updated terminology page with new terms, added Oracle 21c note, updat…
gvasquezvargas Mar 19, 2025
c37e762
Edited note on RW_SERVICE_HOST
gvasquezvargas Mar 24, 2025
c35b0da
Code changes to generate install files for DMS reader and writer
dwicinas Mar 25, 2025
1995809
Generated file changes for Ubuntu 24 install topics for DMS reader an…
dwicinas Mar 25, 2025
4a341d6
Change config.yaml to remove ubuntu 20 from dms reader and writer and…
dwicinas Mar 25, 2025
b42db55
Apply suggestions from Matt's review
gvasquezvargas Mar 26, 2025
58bc8e1
Removed ubuntu 20 file for DMS writer
gvasquezvargas Mar 26, 2025
2e76fa0
Reader/writer ID info
gvasquezvargas Mar 26, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -121,11 +121,17 @@ Consider the following ID guidelines:

- The maximum character length for the ID is 255 characters.
- You can use lowercase and uppercase characters, numbers, underscores(_) and hyphens(-) for the ID. Other special characters are not supported.
- The ID must be unique. The source instances cannot have the same ID.
- The ID must be unique.

### `RW_SERVICE_HOST`

Specifies the URL of the service that will host the migration. `transporter-rw-service` is always https://transporter-rw-service.biganimal.com.
Specifies the URL of the service that will host the migration. Set `RW_SERVICE_HOST` to the domain name or host associated with the HCP ingress.

Derive `RW_SERVICE_HOST` from the `TRANSPORTER_RW_SERVICE_DOMAIN_NAME` that was assigned by the administrators or installers of your HCP instance via the `values.yaml` file, and add `/transporter`.

Alternatively, derive `RW_SERVICE_HOST` from the URL you use to access the HCP portal. For example, if the URL is `https://portal-transporter.foo.network`, extract the domain name only, and add `/transporter` at the end.

In this example, you'd have to set the `RW_SERVICE_HOST` to `transporter.foo.network/transporter`.

### `TLS_PRIVATE_KEY_PATH`

Expand Down Expand Up @@ -190,7 +196,7 @@ Specify the name of the bucket you want to use for the migration. The bucket mus

This is a list of source database information you require for the EDB DMS Reader be able to read the correct source database information for the migration.

You can configure the EDB DMS Reader to migrate multiple databases. The `DBZ_DATABASES_0__TYPE` section delimits the information for the first database. You can use `DBZ_DATABASES_1__TYPE` to provide data for a second database. Add more sections to the EDB DMS Reader (`DBZ_DATABASES_2__TYPE`, `DBZ_DATABASES_3__TYPE`) by increasing the index manully.
You can configure the EDB DMS Reader to migrate multiple databases. The `DBZ_DATABASES_0__TYPE` section delimits the information for the first database. You can use `DBZ_DATABASES_1__TYPE` to provide data for a second database. Add more sections to the EDB DMS Reader (`DBZ_DATABASES_2__TYPE`, `DBZ_DATABASES_3__TYPE`) by increasing the index manually.

#### `DBZ_DATABASES_0__TYPE`

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,222 @@
---
title: "Configuring and running the EDB DMS Writer"
deepToC: true
redirects:
- /purl/dms/configure_target
---

This step is optional. Configure and run the EDB DMS Writer only if the destination for your migration is a self-managed Postgres database.

## Getting credentials

1. Access the Hybrid Control Plane Portal (HCP Portal) and log in with your EDB Postgres AI credentials.

1. Select the HCP project you are using for your migration activities.

1. Within your project, select **Migrate** > **Credentials**.

1. Select **Create Migration Credential** > **Download Credential**.

1. Unzip the credentials folder and copy it to the host where the writer is installed.

## Configuring the writer

1. Open the EDB DMS writer located in `/opt/cdcwriter/run-cdcwriter.sh` and ensure you have write permissions.

1. Set the variables according to your environment. Uncomment the lines as you edit or review them, the DMS can only perform migrations when all necessary values are provided. See [parameters](#parameters) for further guidance. The script is reproduced below.

```shell
#!/bin/bash -e

### set the following environment variables:

##########################################
# Transporter Cloud Configuration #
##########################################

# This ID is used to identify DMS Writer
# and is specified by the user.
#export DBCONFIG_ID=

# Supported options include: appliance (the hybrid PG AI platform), aws
#export CLOUD_PROVIDER=

# This is the DMS backend service used by the Writer
# If your CLOUD_PROVIDER is `appliance`, consult your system administrators
# The default value supports the `aws` CLOUD_PROVIDER
#export RW_SERVICE_HOST=https://transporter-rw-service.biganimal.com

# You need to create migration credentials in EDB Postgres AI platform
# and set these fields with the path of the credential files
#export TLS_PRIVATE_KEY_PATH=$HOME/credentials/client-key.pem
#export TLS_CERTIFICATE_PATH=$HOME/credentials/client-cert.pem
#export TLS_CA_PATH=$HOME/credentials/int.crt
#export APICURIOREQUEST_CLIENT_KEYSTORE_LOCATION=$HOME/credentials/client.keystore.p12
#export APICURIOREQUEST_TRUSTSTORE_LOCATION=$HOME/credentials/int.truststore.p12
#export KAFKASECURITY_CLIENT_KEYSTORE_LOCATION=$HOME/credentials/client.keystore.p12
#export KAFKASECURITY_TRUSTSTORE_LOCATION=$HOME/credentials/int.truststore.p12

##########################################
# DMS Writer Target DB Configuration #
##########################################

# A sample configuration to create a single postgres database connection:
#export DBCONFIG_DATABASES_0__TYPE=POSTGRES
#export DBCONFIG_DATABASES_0__HOSTNAME=localhost
#export DBCONFIG_DATABASES_0__PORT=5332
# The CATALOG is the database name
#export DBCONFIG_DATABASES_0__CATALOG=target
#export DBCONFIG_DATABASES_0__USERNAME=postgres
# The password env can be set without specifying it here
# but the env structure looks like this
#export DBCONFIG_DATABASES_0__PASSWORD=password

##############################################################################
# DMS Writer Object storage config, only necessary for appliance/local #
##############################################################################
#export AWS_ACCESS_KEY_ID=
#export AWS_SECRET_ACCESS_KEY=
#export AWS_REGION=
#export AWS_CA_BUNDLE=
#export AWS_ENDPOINT_URL_S3=
#export BUCKET_NAME=


##########################################
# Optional parameters below #
##########################################

# Configure logging
# Generic loglevel
#export QUARKUS_LOG_LEVEL=DEBUG
# Loglevel for a single package
#export QUARKUS_LOG_CATEGORY__COM_ENTERPRISEDB__LEVEL=DEBUG

cd $(dirname "$0")
java "${JAVA_OPTS}" -jar quarkus-run.jar
```

## Parameters

### `DBCONFIG_ID`

This is the name you assign to identify a destination. This name will later appear as a _destination_ in the **Migrate** > **Destinations** section of the HCP Portal.

Consider the following ID guidelines:

- The maximum character length for the ID is 255 characters.
- You can use lowercase and uppercase characters, numbers, underscores(_) and hyphens(-) for the ID. Other special characters are not supported.
- The ID must be unique.

### `RW_SERVICE_HOST`

Specifies the URL of the service that will host the migration. Set `RW_SERVICE_HOST` to the domain name or host associated with the HCP ingress.

Derive `RW_SERVICE_HOST` from the `TRANSPORTER_RW_SERVICE_DOMAIN_NAME` that was assigned by the administrators or installers of your HCP instance via the `values.yaml` file, and add `/transporter`.

Alternatively, derive `RW_SERVICE_HOST` from the URL you use to access the HCP portal. For example, if the URL is `https://portal-transporter.foo.network`, extract the domain name only, and add `/transporter` at the end.

In this example, you'd have to set the `RW_SERVICE_HOST` to `transporter.foo.network/transporter`.

### `TLS_PRIVATE_KEY_PATH`

Directory path to the `client-key.pem` private key you downloaded from the HCP Portal.

The HTTP client of the EDB DMS writer uses it to perform mTLS authentication with the `transporter-rw-service`.

### `TLS_CERTIFICATE_PATH`

Directory path to the X509 `client-cert.pem` certificate you downloaded from the HCP Portal.

The HTTP client of the EDB DMS writer uses it to perform mTLS authentication with the `transporter-rw-service`.

### `TLS_CA_PATH`

Directory path to the `int.cert` Certificate Authority you downloaded from the HCP Portal.

It signs the certificate configured in `TLS_CERTIFICATE_PATH`.

### `APICURIOREQUEST_CLIENT_KEYSTORE_LOCATION`

Directory path to the `client-keystore.p12` keystore location file you downloaded from the HCP Portal.
It is created from the private key and certificate configured in `TLS_PRIVATE_KEY_PATH` and `TLS_CERTIFICATE_PATH`.

The Apicurio client uses it to perform mTLS authentication with the `transporter-rw-service`.

### `APICURIOREQUEST_TRUSTSTORE_LOCATION`

Created from the Certificate Authority configured in `TLS_CA_PATH`.

The Apicurio client uses it to perform mTLS authentication with the `transporter-rw-service`.

### `AWS_ACCESS_KEY_ID`
1) If you are using AWS S3, set this value to an access key ID that has read and write permissions for the S3 bucket.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a setting for something that is in a user's environment or in the HCP deployment environment? If the latter, is this a value that a user of the HCP will know or will they have to get this from an administrator? This should be clearer. And, if it is something that is specified in the HCP configuration, the admin's, who may not have set up the HCP, will need to know how find it.

@jperozoa @yidian1997 Settings like this and the RW_SERVICE_HOST have me wondering if there is a way for a user to have an environment specific run-cdcreader.sh and run-cdcwriter.sh files generated for them that would populate some of these values for them based on the HCP deployment.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think those access key related values should get from administrator.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Who can we ask more about this? If an admin needs to provide this information, where does he/she get it?

Do they check in their AWS console? Is there a command they can run to fetch this and all subsequent values that start with AWS_? Or maybe all these values are related to something established during the installation of the HCP?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, in a conversation I had with @jperozoa earlier today, it's my understanding that these S3 related settings are only required if you have large rows in your source database that needs to be migrated. As I understand it, transporter can actually migrate the large rows, but setting these variables will improve performance of migrating large rows. If they are optional, we should note that they are optional but recommended and provide their purpose/benefit.


2) If you are using S3-compatible storage, like MinIO, set this value to the access key for your object storage.

### `AWS_SECRET_ACCESS_KEY`
1) If you are using AWS S3, set this value to a secret access key that has read and write permissions for the specified S3 bucket.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have the same question here as I did in my comment under to the AWS_ACCESS_KEY_ID setting.


2) If you are using S3-compatible storage, like MinIO, set this value to the secret key for your object storage.

### `AWS_REGION` (Optional)
1) If you are using AWS S3, set this value to match the region specified in `AWS_ENDPOINT_URL_S3`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have the same question here as I did in my comment under to the AWS_ACCESS_KEY_ID setting.


2) If you are using S3-compatible storage, such as MinIO, this setting is not required.

### `AWS_CA_BUNDLE` (Optional)
Set this value only if you are using a self-signed certificate for your object storage. Specify the path to the CA bundle file.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have the same question here as I did in my comment under to the AWS_ACCESS_KEY_ID setting.


This setting is not required if you are using AWS S3 or S3-compatible storage with publicly trusted certificate authorities (CAs).

### `AWS_ENDPOINT_URL_S3`
1) If you are using AWS S3, you can find the endpoint URL in the AWS documentation: [AWS S3 Endpoints](https://docs.aws.amazon.com/general/latest/gr/s3.html).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have the same question here as I did in my comment under to the AWS_ACCESS_KEY_ID setting.


2) If you are using S3-compatible storage, like MinIO, set this value according to your object storage configuration.

### `BUCKET_NAME`
Specify the name of the bucket you want to use for the migration. The bucket must be created before starting the migration.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have the same question here as I did in my comment under to the AWS_ACCESS_KEY_ID setting.


### `DBCONFIG_DATABASES`

This is a list of target database information you require for the EDB DMS writer be able to read the correct target database information for the migration.

You can configure the EDB DMS writer to migrate multiple databases. The `DBCONFIG_DATABASES_0__TYPE` section delimits the information for the first database. You can use `DBCONFIG_DATABASES_1__TYPE` to provide data for a second database. Add more sections to the EDB DMS writer (`DBCONFIG_DATABASES_2__TYPE`, `DBCONFIG_DATABASES_3__TYPE`) by increasing the index manually.

#### `DBCONFIG_DATABASES_0__TYPE`

This is the target database type. EDB DMS writer supports `POSTGRES`.

#### `DBCONFIG_DATABASES_0__HOSTNAME`

The hostname of the target database.

#### `DBCONFIG_DATABASES_0__PORT`

The port of the target database.

#### `DBCONFIG_DATABASES_0__CATALOG`

The database name in the target database server.

#### `DBCONFIG_DATABASES_0__USERNAME`

The database username of the target database.

#### `DBCONFIG_DATABASES_0__PASSWORD`

The password for the database username of the target database.


## Running the EDB DMS Writer

1. Start the migration:

```shell
cd /opt/cdcwriter
./run-cdcwriter.sh
```

1. Go to the HCP Portal, and verify that a destination with the `DBCONFIG_ID` name is displayed in **Migrate** > **Destinations**.

You can select this destination for your [migration](create_migration).
Original file line number Diff line number Diff line change
Expand Up @@ -5,25 +5,29 @@ title: "Creating a migration"
After you use the EDB DMS Reader to read the source database, create a new migration in the EDB Postgres® AI Console.
This establishes a sync between the source database and a target cluster in the EDB Postgres AI Console.

1. Access the [EDB Postgres AI Console](https://portal.biganimal.com) and log in with your EDB Postgres AI Database Cloud Service credentials.
1. Access your Hybrid Control Plane portal and log in with your EDB Postgres AI credentials.

1. Select the project where you created the database cluster.
1. Select the project from which you want to manage the migration.

1. Within your project, select **Migrate** > **Migrations**.

1. In the **Migrations** page, select **Create New Migration** > **To Managed Postgres**.
1. In the **Migrations** page, select **Create New Migration**.

1. In the **Create Migration** page, assign a **Name** to the migration.

1. Select the **Source** of the migration. The ID for the EDB DMS Reader is listed in the drop-down menu.

1. Under **Destination**, select a target cluster for the migration and enter the name of the database where you want the migration to copy data and select **Next**.
1. Select the migration type, [**Snapshot**](/edb-postgres-ai/migration-etl/data-migration-service/terminology/#snapshot-migration) or [**Snapshot + Streaming**](/edb-postgres-ai/migration-etl/data-migration-service/terminology/#snapshot--streaming-migration).

1. Under **Destination Type**, select a target for the migration. This can be a [**Managed Postgres**](/edb-postgres-ai/migration-etl/data-migration-service/terminology/#managed-postgres) or [**External Postgres**](/edb-postgres-ai/migration-etl/data-migration-service/terminology/#external-postgres) instance. Select the destination cluster and database accordingly and select **Next**.

1. Select the tables and columns to migrate. Modify the table and column names if needed.

1. Select **Create Migration**.

The EDB Postgres AI Console now displays a new migration. The EDB DMS Reader is constantly streaming data when the migration displays the **Running** state. Changes to data are replicated from the source to the target database as long as the migration is running.
The Hybrid Control Plane portal now displays a new migration.

If you chose the **Snapshot + Streaming** option, the EDB DMS Reader is constantly streaming data when the migration displays the **Running** state. Changes to data are replicated from the source to the target database as long as the migration is running.

!!!note
The EDB DMS Reader streams data changes. It does not stream changes in the DDL objects.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,10 @@ navigation:
- create_database
- prepare_schema
- installing
- installing2
- preparing_db
- config_reader
- config_writer
- create_migration
- mark_completed
- apply_constraints
Expand All @@ -22,9 +24,13 @@ Creating a migration from an Oracle or Postgres database to EDB Postgres AI invo

1. **[Install the EDB DMS Reader](installing)**: In your source machine, install the EDB DMS Reader from the EDB repository.

1. ** Optional - [Install the EDB DMS Writer](installing2)**: Perform this step only if you want to migrate either HCP-managed databases or other external, self-managed databases to a different database outside of the Hybrid Control Plane. In your target machine, install the EDB DMS Writer from the EDB repository.

1. **[Prepare your source Oracle or Postgres database](preparing_db)**: In your source machine, prepare the source database by altering settings and creating users that are required for the migration. Ensure your source database can accept SSL connections.

1. **[Configure the EDB DMS Reader](config_reader)**: In the EDB Postgres AI Console, download dedicated migration credentials. In your source machine, configure the EDB DMS Reader by exporting environment variables that allow the Reader to connect to the source. Execute the Reader.
1. **[Configure the EDB DMS Reader](config_reader)**: In the Hybrid Control Plane portal, download dedicated migration credentials. In your source machine, configure the EDB DMS Reader by exporting environment variables that allow the Reader to connect to the source. Execute the Reader.

1. **Optional - [Configure the EDB DMS Writer](config_writer)**: Perform this step only if you want to migrate either HCP-managed databases or other external, self-managed databases to a different database outside of the Hybrid Control Plane. In your target machine, configure the EDB DMS Writer by exporting environment variables that allow the Writer to connect to the target. Execute the Writer.

1. **[Create a new migration](create_migration)**: In the EDB Postgres AI Console, create a new migration by selecting the source generated by the Reader in the Console, and selecting the target database you created for this purpose.

Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
navTitle: Installing
navTitle: Installing reader
title: Installing EDB Data Migration Service Reader on Linux
indexCards: none

Expand Down Expand Up @@ -27,7 +27,7 @@ Select a link to access the applicable installation instructions:

### Debian and derivatives

- [Ubuntu 22.04](linux_x86_64/edb-dms-reader_ubuntu_22), [Ubuntu 20.04](linux_x86_64/edb-dms-reader_ubuntu_20)
- [Ubuntu 24.04](linux_x86_64/edb-dms-reader_ubuntu_24), [Ubuntu 22.04](linux_x86_64/edb-dms-reader_ubuntu_22)

- [Debian 12](linux_x86_64/edb-dms-reader_debian_12), [Debian 11](linux_x86_64/edb-dms-reader_debian_11)

Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
navTitle: Ubuntu 20.04
title: Installing EDB Data Migration Service Reader on Ubuntu 20.04 x86_64
navTitle: Ubuntu 24.04
title: Installing EDB Data Migration Service Reader on Ubuntu 24.04 x86_64
---

## Prerequisites
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@ navigation:
- edb-dms-reader_rhel_8
- edb-dms-reader_other_linux_9
- edb-dms-reader_sles_15
- edb-dms-reader_ubuntu_24
- edb-dms-reader_ubuntu_22
- edb-dms-reader_ubuntu_20
- edb-dms-reader_debian_12
- edb-dms-reader_debian_11
---
Expand All @@ -36,9 +36,9 @@ Operating system-specific install instructions are described in the correspondin

### Debian and derivatives

- [Ubuntu 22.04](edb-dms-reader_ubuntu_22)
- [Ubuntu 24.04](edb-dms-reader_ubuntu_24)

- [Ubuntu 20.04](edb-dms-reader_ubuntu_20)
- [Ubuntu 22.04](edb-dms-reader_ubuntu_22)

- [Debian 12](edb-dms-reader_debian_12)

Expand Down
Loading
Loading