useLocalAudioInputActivity/useLocalAudioInputActivityPreview does not work in a started meeting if the microphone toggle is off

[mmarekbb]

Describe the bug
The useLocalAudioInputActivity and useLocalAudioInputActivityPreview hooks only work when the meeting either has not started yet (meetingManager.start() was not called) or the microphone has been turned on in a started meeting using useToggleLocalMute.

We'd like our customers to be able to change their device setup during a live session as well - everything else works (camera preview, changing devices, etc.)

To Reproduce
Steps to reproduce the behavior:

1. Join a meeting but do not call meetingManager.start() - see the meeting demo for an example
2. Notice that the audio activity preview works independently of the audio toggle state - it works both when muted or not
3. Enter the meeting by calling meetingManager.start()
4. Notice that the audio activity preview now ONLY works if the microphone is not muted

Desktop (please complete the following information):

• OS: macOS 10.15.7
• Browser: Chrome
• Version: 91.0.4472.77

[anuranduttaroy]

Hey @mmarekbb

We tried to reproduce the issue on our side by using the meeting demo in the component library. We also tried the same configuration as yours [email protected] | [email protected]

When we tried to log the value of decimal in useLocalAudioInputActivityPreview, we were not able to reproduce it. Here is the output we saw in multiple rounds of testing.

Decimal in useLocalAudioInputActivityPreview -0
Decimal in useLocalAudioInputActivityPreview 0.184955642535897
Decimal in useLocalAudioInputActivityPreview 0.5616194758652027
Decimal in useLocalAudioInputActivityPreview 0.8780564452362938
Decimal in useLocalAudioInputActivityPreview 0.954911684825456
Decimal in useLocalAudioInputActivityPreview 0.9506951010570246
Decimal in useLocalAudioInputActivityPreview 0.9210900184985222
Decimal in useLocalAudioInputActivityPreview 0.820489028679166
Decimal in useLocalAudioInputActivityPreview 0.4463950151760658
Decimal in useLocalAudioInputActivityPreview 0.24742501084004695
Decimal in useLocalAudioInputActivityPreview 0.0969100130080564
Decimal in useLocalAudioInputActivityPreview -0
Decimal in useLocalAudioInputActivityPreview 0.9059340613641027
Decimal in useLocalAudioInputActivityPreview 0.9329589419759152
Decimal in useLocalAudioInputActivityPreview 0.9506951010570246
Decimal in useLocalAudioInputActivityPreview 0.9329589419759152
Decimal in useLocalAudioInputActivityPreview 0.6620768972555594
Decimal in useLocalAudioInputActivityPreview 0.33547064036788754
Decimal in useLocalAudioInputActivityPreview 0.6365006360318688
Decimal in useLocalAudioInputActivityPreview 0.9352568178204897
Decimal in useLocalAudioInputActivityPreview 0.9506951010570246
Decimal in useLocalAudioInputActivityPreview 0.9569896647110349
Decimal in useLocalAudioInputActivityPreview 0.9610869040186706
Decimal in useLocalAudioInputActivityPreview 0.9442126124748408
Decimal in useLocalAudioInputActivityPreview 0.948555702067387
Decimal in useLocalAudioInputActivityPreview 0.8979400086720376
Decimal in useLocalAudioInputActivityPreview 0.747425010840047
Decimal in useLocalAudioInputActivityPreview 0.5194590330151848
Decimal in useLocalAudioInputActivityPreview 0.29588001734407515
Decimal in useLocalAudioInputActivityPreview -0
Decimal in useLocalAudioInputActivityPreview 0.33547064036788754
Decimal in useLocalAudioInputActivityPreview 0.954911684825456
Decimal in useLocalAudioInputActivityPreview 0.9631068929195405
Decimal in useLocalAudioInputActivityPreview 0.9442126124748408
Decimal in useLocalAudioInputActivityPreview 0.81259189508755
Decimal in useLocalAudioInputActivityPreview 0.4670913577551783
Decimal in useLocalAudioInputActivityPreview 0.3689440351831942
Decimal in useLocalAudioInputActivityPreview 0.9161546414853752
Decimal in useLocalAudioInputActivityPreview 0.8390599326814493
Decimal in useLocalAudioInputActivityPreview 0.684955642535897
Decimal in useLocalAudioInputActivityPreview 0.6453650195120846
Decimal in useLocalAudioInputActivityPreview 0.6365006360318688
Decimal in useLocalAudioInputActivityPreview 0.5616194758652027
Decimal in useLocalAudioInputActivityPreview 0.39794000867203755
Decimal in useLocalAudioInputActivityPreview 0.24742501084004695
Decimal in useLocalAudioInputActivityPreview 0.0969100130080564
Decimal in useLocalAudioInputActivityPreview -0
Decimal in useLocalAudioInputActivityPreview 0.4670913577551783
Decimal in useLocalAudioInputActivityPreview 0.892442316521306
Decimal in useLocalAudioInputActivityPreview 0.9136442407978497
Decimal in useLocalAudioInputActivityPreview 0.9059340613641027
Decimal in useLocalAudioInputActivityPreview 0.6272589331848623
Decimal in useLocalAudioInputActivityPreview 0.8594324165264791
Decimal in useLocalAudioInputActivityPreview 0.8460652899028567
Decimal in useLocalAudioInputActivityPreview 0.4235162698957282
Decimal in useLocalAudioInputActivityPreview 0.6075046625430254
Decimal in useLocalAudioInputActivityPreview 0.7914930551903226
Decimal in useLocalAudioInputActivityPreview 0.5857718156524803
Decimal in useLocalAudioInputActivityPreview 0.33547064036788754
Decimal in useLocalAudioInputActivityPreview -0

Could you try the meeting demo app once to see if you see the same issue? If you still see the issue, could you share some logs for us to investigate the it further?

[mmarekbb]

I have an update - I was wrong with the assumption that the version upgrade has broken the functionality.

However, the mic preview only works when the microphone is enabled while the meeting has already started. If I don't run meetingManager.start(), then the preview works even with audio muted. However, once the meeting is started that is no longer the case.

Is it possible to change this behavior so the audio activity preview works independently on the audio toggle state?

Edit: I've updated the issue description to reflect this.

[devalevenkatesh]

Hi @mmarekbb ,

Could you please clarify a little more on your use case (looks like a feature request for now)? Do you mean that the useLocalAudioActivityPreview should work irrespective of audio mute / unmute?

[mmarekbb]

Yes, when the meeting isn't joined then you can see the preview even if the audio is muted. But once the meeting starts, that is no longer the case. We want to give users the ability to change their audio and video preferences during the session, but to do this, we cannot require them to have their actual in-session microphone turned on.

[mmarekbb]

I'm not sure if this should be considered a feature request, because the Camera Preview works even with the local camera toggle disabled. You can preview your own camera just fine without having to broadcast the stream to all other participants, but this is not the case with the audio preview.

[mmarekbb]

Hello, is there any update to this issue? @anuranduttaroy @devalevenkatesh

[devalevenkatesh]

Yes, you are right with the video camera. The Amazon Chime SDK for JavaScript (JS SDK) has separate APIs like the startVideoPreviewForVideoInput and stopVideoPreviewForVideoInput which help in doing it separately irrespective of the local video enabled/disabled.

Note: There is limitation on using PreviewVideo and LocalVideo together as at a time only one video is seen as JS SDK connects video stream to a single video tile.

For the useLocalAudioInputActivity and useLocalAudioInputActivityPreview hooks in the current react meeting demo, the device selection page already has audio input unmuted. I added useToggleMute and checked muted state which is false. Hence, you are seeing the working preview. These both hooks are internally depending on the JS SDK's AudioVideoFacade's createAnalyserNodeForAudioInput API which is dependent on the current audio input's stream. The stream will be disabled when audio input is muted and hence, the createAnalyserNodeForAudioInput won't work. This would be an enhancement to make in the JS SDK first and then in the hooks in the React SDK.

There is a workaround explained in a similar issue on the JS SDK. Could you please check this issue's comment and see if the provided workaround works for you?

[devalevenkatesh]

Update on the workaround:

I tried to implement the workaround in the useLocalAudioInputActivity hook and it worked fine irrespective of mute / unmute local audio input state.

You can use the react meeting demo to test the workaround.

1. In the MeetingView component, add MicrophoneActivityPreview after <UserActivityProvider> in return function. This will get the hook in the in-meeting screen page as well.
2. Now, replace the useLocalAudioInputActivity code with below code. In the code below, I added useToggleLocalMute to get the muted state and instead of creating and getting the analyserNode from JS SDK's AudioVideo, create it from the stream received from getUserMedia from the selected audio input device. Finally, add dependency of muted so that the useLocalAudioInputActivity updates on mute/unmute.

// Copyright 2020-2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.
// SPDX-License-Identifier: Apache-2.0

import { useEffect } from 'react';

import { useAudioVideo } from '../../providers/AudioVideoProvider';
import { useAudioInputs } from '../../providers/DevicesProvider';
import useToggleLocalMute from './useToggleLocalMute';

export const useLocalAudioInputActivity = (cb: (decimal: number) => void) => {
  const audioVideo = useAudioVideo();
  const { muted } = useToggleLocalMute();
  const { selectedDevice } = useAudioInputs();

  useEffect(() => {
    if (!audioVideo) {
      return;
    }

    let analyserNode: AnalyserNode | null;
    let restart = false;
    let data: Uint8Array;
    let frameIndex: number;
    let isMounted = true;
    let lastDecimal: number;

    audioVideo.addDeviceChangeObserver({
      audioInputsChanged: () => {
        restart = true;
      },
    });

    function analyserNodeCallback() {
      if (!analyserNode) {
        return;
      }
      if (frameIndex === 0) {
        analyserNode.getByteTimeDomainData(data);
        const lowest = 0.01;
        let max = lowest;
        for (const f of data as any) {
          max = Math.max(max, (f - 128) / 128);
        }
        const decimal = (Math.log(lowest) - Math.log(max)) / Math.log(lowest);

        if (lastDecimal !== decimal) {
          lastDecimal = decimal;

          if (cb) {
            cb(decimal);
          }
        }
      }

      frameIndex = (frameIndex + 1) % 2;

      if (restart) {
        setTimeout(initializePreview, 500);
      } else if (isMounted) {
        requestAnimationFrame(analyserNodeCallback);
      }
    }

    function initializePreview() {
      if (!audioVideo || !isMounted) return;
      // Instead of using the audioVideo, create a new AudioContext and connect the stream
      // received from getUserMedia.
      // analyserNode = audioVideo.createAnalyserNodeForAudioInput(); <- Existing retrieval of analyserNode which we do not use.
      navigator.mediaDevices.getUserMedia({
        audio: { deviceId: selectedDevice as string }
      }).then((stream: MediaStream) => {
        // Can destroy this in un-mounting phase for this hook.
        const audioContext = new window.AudioContext();
        analyserNode = audioContext.createAnalyser();
        audioContext.createMediaStreamSource(stream).connect(analyserNode);
        if (!analyserNode?.getByteTimeDomainData) {
          return;
        }
        data = new Uint8Array(analyserNode.fftSize);
        frameIndex = 0;
        restart = false;
        requestAnimationFrame(analyserNodeCallback);
      });
    }

    initializePreview();

    return () => {
      isMounted = false;
    };
  }, [audioVideo, selectedDevice, muted, cb]);
};

export default useLocalAudioInputActivity;

3. Run the React SDK meeting demo, you will see the microphone activity bar on top of the in-meeting screen once joining the meeting.
4. Use, mute/unmute button in the meeting controls and see the activity bar still works.
5. I tested the mute / unmute state toggling between true and false, but still the audio activity worked.

[mmarekbb]

Thanks for providing the workaround @devalevenkatesh! It works as expected for our use case. Are there any plans to bring this into the main codebase in the future?

[vidya-mohan]

thanks for your interest. We are happy to note you have a workaround on the issue @mmarekbb - we will keep this issue active till it is resolved

[ltrung]

@mmarekbb For the Firefox use case since it does not allow you to get a second stream, you can use meetingManager.selectAudioInputDevice(null) to reset the current stream before you select the new stream from getUserMedia.

[mmarekbb]

Posting an update here:

The workaround above works great in Chrome but it fails in Firefox (and looks like Android as well) as it does not allow two concurrent media streams to exist.

@ltrung We need to test the device selected using useSelectAudioInputDevice() (the preview is in the same dialog where we also display the select box). Unsetting the device would also remove it from the select box.

I'm assuming there's a way to solve this problem by not using the SDK hooks and instead enumerate the devices, let the user try the preview out and only set it using useSelectAudioInputDevice() after they have confirmed their selection.

Please let me know if there are plans to improve this behavior in the SDK itself which we would happily wait for - otherwise we can try the workaround I'm mentioning above.

FYI @giridharknamz

[vidya-mohan]

@mmarekbb allowing the DeviceController to support multiple active streams would require rework of the SDK - we will consider this usecase in future major version upgrade (this is right now not in scope for 3.0) and will keep this issue until then. Please try the workaround and do share if it worked across all the browsers