New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add AssemblyAI Plugin #687

Open

oconnoob wants to merge 24 commits into livekit:main from oconnoob:oconnoob/add-assemblyai-plugin

+512 −1

oconnoob commented Aug 30, 2024

Extended from PR 419 - #419

cch41 and others added 13 commits

July 5, 2024 16:04


          Add AssemblyAI STT plugin

ef6a5b5


          Merge changes from PR 419 into oconnoob/add-assemblyai-plugin

dad69a7

See: livekit#419


          add error resolution details to ValueError in STT initialization

afb7cf1


          use session property rather than function in STT

147bebc


          fix type hint for buffer size in STT

aa2cf67


          replace type hint union with Optional

ddd8106

to improve backwards compatibility


          make encoding type hint specify possible values

8b478a6


          word boost type hint: str -> List[str]

2e8cde5


          "Assembly AI" -> "AssemblyAI"

c53f254


          fix buffer size type hint

ab67cc4


          add check for mono audio

5bdaa64


          moved buffer variables from instance to _run_ws method

194dda9


          corrected logic error and improved readability

72adccd

1. added dictionary to map from encoding type to bytes per frame
2. clarified the calculation of buffer duration

changeset-bot bot commented Aug 30, 2024 •

edited

Loading

⚠️ No Changeset found

Latest commit: dd8ec5c

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

CLAassistant commented Aug 30, 2024

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ cch41
❌ ryan-assemblyai
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

oconnoob commented

View reviewed changes

Author

oconnoob left a comment

@ploeber could you review when you get a chance?

livekit-plugins/livekit-plugins-assemblyai/livekit/plugins/assemblyai/stt.py Show resolved Hide resolved

livekit-plugins/livekit-plugins-assemblyai/livekit/plugins/assemblyai/stt.py Outdated

Comment on lines 78 to 87

+                      self._opts = STTOptions(
+                          sample_rate=sample_rate,
+                          word_boost=word_boost,
+                          encoding=encoding,
+                          disable_partial_transcripts=disable_partial_transcripts,
+                          enable_extra_session_information=enable_extra_session_information,
+                          buffer_size_seconds=buffer_size_seconds,
+                          token_expires_in=token_expires_in,
+                          end_utterance_silence_threshold=end_utterance_silence_threshold,
+                      )

Author

oconnoob Aug 30, 2024

Why not use dependency injection? Because higher-level class than e.g. SpeechStream which does use DI?

ploeber Sep 4, 2024

Not sure what you mean?

Author

oconnoob Sep 5, 2024

Why don't we just pass in STTOptions directly as we do in SpeechStream's init?

livekit-plugins/livekit-plugins-assemblyai/livekit/plugins/assemblyai/stt.py Outdated Show resolved Hide resolved

livekit-plugins/livekit-plugins-assemblyai/livekit/plugins/assemblyai/stt.py Outdated Show resolved Hide resolved

ploeber suggested changes

View reviewed changes

ploeber left a comment •

edited

Loading

We also need to update the base README.md and list our plugin there

livekit-plugins/livekit-plugins-assemblyai/README.md Outdated Show resolved Hide resolved

livekit-plugins/livekit-plugins-assemblyai/README.md Outdated Show resolved Hide resolved

tests/test_stt.py Outdated Show resolved Hide resolved

livekit-plugins/livekit-plugins-assemblyai/livekit/plugins/assemblyai/stt.py Show resolved Hide resolved

livekit-plugins/livekit-plugins-assemblyai/livekit/plugins/assemblyai/stt.py Outdated Show resolved Hide resolved

livekit-plugins/livekit-plugins-assemblyai/livekit/plugins/assemblyai/stt.py Outdated Show resolved Hide resolved

livekit-plugins/livekit-plugins-assemblyai/livekit/plugins/assemblyai/stt.py Outdated Show resolved Hide resolved

livekit-plugins/livekit-plugins-assemblyai/livekit/plugins/assemblyai/stt.py Outdated Show resolved Hide resolved

tests/test_stt.py

@@ @@ -9,7 +9,7 @@ @@
               import pytest
               from livekit import agents, rtc
-              from livekit.plugins import azure, deepgram, google, openai, silero
+              from livekit.plugins import assemblyai, azure, deepgram, google, openai, silero

ploeber Sep 4, 2024

since we add this new dependency, we need to add ./livekit-plugins/livekit-plugins-assemblyai \ to the tests.yml file

Contributor

keepingitneil Sep 5, 2024

Nice work so far, care to join a shared slack channel to help get this over the finish line? Just need your emails

livekit-plugins/livekit-plugins-assemblyai/livekit/plugins/assemblyai/stt.py Outdated Show resolved Hide resolved

ryan-assemblyai added 10 commits

September 5, 2024 10:58


          update assemblyai.stt.STT to use new STTCapabilities

6958e9c


          remove token in favor of api key

4fc4e3c

this code will run on server so no need to generate temporary token


          add catch-all for unexpected data types in SpeechStream

fcd08ca


          update AAI plugin README

93d130b


          add assemblyai to agents README

158b431


          remove mentions of token

669e9a8


          Assembly AI -> AssemblyAI

7e861b9


          remove arguments/attributes redundant with STTOptions

55f4bb4

The following parameters of `SpeechStream`'s init are redundant with the parameter `opts: STTOptions`

- buffer_size_seconds
- sample_rate
- end_utterance_silence_threshold

they have been removed and their corresponding instance attributes have been removed


          remove unused function

5ba8be0

`prerecorded_transcription_to_speech_event` appears to have been brought over from a copy-paste


          fix definition of STT.session

65ad305

Author

oconnoob commented Sep 5, 2024

@ploeber the STT interface requires that the abstract async method _main_task is implemented. Currently _main_task exists as an instance attribute of SpeechStream - I'm not sure how to reconcile this

add

dd8ec5c

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet