skip to Main Content

I would like to integrate an HTML5 microphone in my web application, record audio and send it to a (Node.js) back-end, use the Dialogflow API for audio, and return audio result to a client to play this in a browser.

(I use Windows 10, Windows Subsystems for Linux, Debian 10.3 and Google Chrome browser.
)

I found a github project which is exactly what I want to do. https://github.com/dialogflow/selfservicekiosk-audio-streaming

This is Ms. Lee Boonstra’s Medium blog. (https://medium.com/google-cloud/building-your-own-conversational-voice-ai-with-dialogflow-speech-to-text-in-web-apps-part-i-b92770bd8b47) She has developed this project. (Thank you very much, Ms. Boonstra!) She explains this project very precisely.

This project contains selfservicekiosk application and 6 simple examples.
I tried all of them.
selfservicekiosk application and simple example 1,2, 4,5,6 worked perfectly, but example3 didn’t work.
Unfortunately, example3 is what I want to do.
https://github.com/dialogflow/selfservicekiosk-audio-streaming/tree/master/examples

These are results when I tried example3.

This is Terminal.
terminal
This is Chrome’s console.
enter image description here

I focus on this message.

(index):59
ArrayBuffer(0)
[[Int8Array]]: Int8Array []
[[Int16Array]]: Int16Array []
[[Int32Array]]: Int32Array []
[[Uint8Array]]: Uint8Array []

I think that browser can get audio result, but can’t play it.

First, I checked my computer’s mic setting and browser’s Web App Activity & Voice/Audio(https://myaccount.google.com/activitycontrols).

Both were enabled.

Next, I check example3.html file and find the code which seems not working in my environment. However, I don’t know how to change it.

     /*
   * When working with Dialogflow and Dialogflow matched an intent,
   * and returned an audio buffer. Play this output.
   */
   function playOutput(arrayBuffer){
        let audioContext = new AudioContext();
        let outputSource;
        try {
            if(arrayBuffer.byteLength > 0){
                audioContext.decodeAudioData(arrayBuffer,
                function(buffer){
                    audioContext.resume();
                    outputSource = audioContext.createBufferSource();
                    outputSource.connect(audioContext.destination);
                    outputSource.buffer = buffer;
                    outputSource.start(0);
                },
                function(){
                    console.log(arguments);
                });
            }
        } catch(e) {
            console.log(e);
        }
    }

Could you give me any advice? Thank you in advance.


I would like to check the audio result, so I opened simpleserver.js file and changed

async function detectIntent(audio){}

https://github.com/dialogflow/selfservicekiosk-audio-streaming/blob/master/examples/simpleserver.js

async function detectIntent(audio){
    request.inputAudio = audio;
    console.log(request);
    const responses = await sessionClient.detectIntent(request);
    const audioFile = responses[0].outputAudio;
    util.promisify(fs.writeFile)('test.wav', audioFile, 'binary');
    console.log('completed');
 }

I opened test.wav file and made sure that dialogflow gave me audio result.

2

Answers


  1. Chosen as BEST ANSWER

    I added this line in simpleserver.js file's setupDialogflow()function.

    outputAudioConfig: {
            audioEncoding: 'OUTPUT_AUDIO_ENCODING_LINEAR_16'
          },
    

    Now I can get voice from my browser.


  2. Hmm that is strange because I did clone a fresh repo, on my Windows 10 machine (without changing the code), and tested it with Chrome (79.0.3945.130) and it just worked. The problem for you is indeed the playing part, because your browser did receive an audio buffer.

    Since you mentioned that the SelfServiceKiosk app worked, and example 3 not; maybe you could replace the playOutput function with the function that is been used by the SelfServiceKiosk app? You can find it here, but be aware that the code is written in TypeScript.
    https://github.com/dialogflow/selfservicekiosk-audio-streaming/blob/master/client/src/app/dialogflow/dialogflow.component.ts

    I know that this code is a little different, and i think I have wrote it that way that it resumes and starts, because otherwise IOS seems to block the auto play. Hope that helps?

    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search