source-chameleon contribution from btkcodedev

[btkcodedev]

Closes https://github.com/airbytehq/airbyte-internal-issues/issues/9874
Auto-generated PR by the Connector Builder for source-chameleon

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Sep 19, 2024 6:30pm

[btkcodedev]

/format-fix

Format-fix job started... Check job output.

✅ Changes applied successfully. (13427b4)

[natikgadzhi]

• Let's add survey responses!
• Can you pretty please make the icon square?

[natikgadzhi]

@btkcodedev can you please add survey responses? https://developers.chameleon.io/#/apis/survey-responses

We (@nataliekwong and I and @lmossman and the whole team) are very interested in that connector ourselves and will definitely use it ;)

[btkcodedev]

Got 200 ok for the survey_responses stream ✔️
Added it with inc sync and error handler, but couldn't seed it currently, thus has_records is false,
I hope it returns all things correctly.
I'll be ready to improve it in future if needed :)

[btkcodedev]

/format-fix

Format-fix job started... Check job output.

✅ Changes applied successfully. (984e333)

[natikgadzhi]

Merging to ship to prod, @nataliekwong @alex-gron lets dogfood this!

[nataliekwong]

I set up a connection in our internal workspace! https://cloud.airbyte.com/workspaces/bed3b473-1518-4461-a37f-730ea3d3a848/connections/821262c9-7036-4abb-97d0-a5a94a0fb643/status

We haven't had the urgency to build this connector since the data comes to us through Segment already. But it's not bad to have another avenue to it!

Only feedback I have so far is: it's odd that I was asked for an End Date in the connector configuration. I don't want to have to go back and edit the configuration every time I want the latest results. I wouldn't expect for it to be required (optional is fine).

[btkcodedev]

Oh, sure, Will make the end date optional

[btkcodedev]

#45708
Made end date optional (Defaults to current date)

[nataliekwong]

I have another piece of feedback, but this time on the sync experience. I tested Chameleon sending data to both BigQuery and Google Sheets. The sync failed both times, because of the same error: Unrecognized name: id

My best guess of why is that this is likely because survey_responses has the primary key id. I do not see a id field in the field selection, and it doesn't exist, so I guess we are trying to pull it or ask the destination to run typing & deduping on it but it cannot because it does not exist.

If this indeed the reason it's failing, I'm a bit surprised our testing didn't catch this. Connector should be updated to pull it in if available.

Also, just curious, why is pagination not implemented? Is it because it's too difficult, not available, other?

[btkcodedev]

I couldn't seed the survey responses stream,
#45658 (comment)
that might lead to wrong primary key guesses,
I'll fix it in #45708

For pagination, it is kind of a problem, Ref: https://developers.chameleon.io/#/concepts/pagination
the docs didn't say anything about offset, page or cursor type, but some filtering with before and after,
Only limiting the records is available.
Need comment on how we could implement the pagination,
Thanks!! 🙇

[nataliekwong]

Got it! Would it help if I shared the API key that has survey responses to help with seeding? I added an entry in integration tests folder in Lastpass - do you have access to that?

[btkcodedev]

The fields are added to the schema in #45708 according to the documentation and merged.
It should work correctly now.
If it doesn't, I would require a seeded account via lastpass

[nataliekwong]

I was able to run this again for v0.0.2. The sync errored here:

Encountered Exception on unsafe SQL for stream chameleon survey_responses with suffix , attempting with error handling com.google.cloud.bigquery.BigQueryException: Query error: Unrecognized name: updated_at at [88:32]

And I believe it's because we declare that updated_at is the cursor, but updated_at is not in the schema, so the destination gets confused trying to run typing & deduping on the field.

I already added an entry to Lastpass if you need a seeded account with data for survey_responses. Let me know if you have any trouble accessing that.

[btkcodedev]

I don't have access to the official LastPass, Could you share on my shared vault Shared-Community Connectors-BTK, Email: [email protected]

[nataliekwong]

Hello, I do not have access to that folder, but I have asked our IT to help with access so I can put it on the shared vault!

[nataliekwong]

Shared the Chameleon API key with you!

[btkcodedev]

Got it,
On it

[btkcodedev]

Fix: #46248