Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update the upsert api to handle multiple columns as unique index #75

Closed
ozgrakkurt opened this issue Jul 12, 2023 · 3 comments
Closed
Assignees

Comments

@ozgrakkurt
Copy link
Contributor

We want the api to look like this:

apiVersion: 0.1.0
meta:
  version: 0.3.1
  name: json-sql-connector
  type: sql-sink
  topic: sql-topic
  create-topic: true
  secrets:
    - name: DATABASE_URL
sql:
  url: ${{ secrets.DATABASE_URL }}
transforms:
  - uses: infinyon/json-sql
    with:
      mapping:
        operation: "upsert"
        table: "topic_message"
        unique-columns: 
          - device_id
          - user_id
        map-columns:
          "device_id":
            json-key: "device.device_id"
            value:
              type: "int"
              default: "0"
              required: true
          "record":
            json-key: "$"
            value:
              type: "jsonb"
              required: true

@galibey maybe we can find a better name for unique-columns? since the combination of them should be unique and they don't need to be unique individually.

@ajhunyady
Copy link

ajhunyady commented Jul 12, 2023

@ozgrakkurt I like parallel keywords, and since we already have map-columns, unique-columns makes sense. It also eliminates the confusion between json-key and column name.

@ozgrakkurt
Copy link
Contributor Author

implemented in #74 but the implementation is hacky, need to update fluvio-model-sql and sql-sink to handle it as array instead of single variable named uniq_idx and add actual tests, and update docs

@ozgrakkurt ozgrakkurt self-assigned this Jul 13, 2023
@ozgrakkurt
Copy link
Contributor Author

opened follow-up issue here: #79

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants