Logging Action Items from a Conversation in Node.js with the Symbl.ai SDK
#conversations, #conversationintelligence, #CI, #nodejs, #javascript, #RTC, #RTE, #transcription, #Real-Time Artificial Intelligence, #action-items
Hello World! There are a lot of Software Development Kits within the realm of Real-Time Communication or Real-Time Engagement but there are is only one SDK yet available for generating Artificial Intelligence from conversations in real-time: Symbl.ai's Node.js SDK. Symbl.ai's Node.js SDK is logs Artificial Intelligence from RTC or RTE conversations in real-time with little to no code at all.
Introduction
In the following blog you download the SDK, set up the environment for the SDK, authenticate securely with an environment, initialize the SDK, start a conversation. Within the conversations you set up event-driven listeners to handle conversation data in real-time. Within those listeners you loop through the Artificial Intelligence. After looping through the event-driven listener's conversation data in real-time, you select questions, logging these to the console.
What are Action-Items?
Symbl.ai's proprietary algorithms for detecting aspects of language naturally processed provides conversational insights such as questions, follow-ups or action-items. Conversational insights arise from Symbl.ai's platform's ability to mine conversation data in real-time to provide developers with access not just to a raw audio stream but to the stream's contents. If you previously developed access to voice, video, messages or live streaming, Symbl.ai now provides you as a developer to conversational Artificial Intelligence that is far beyond merely transcribing or automatically recognizing speech.
In short, action-items as a conversational entity recognized by Symbl.ai's platform are a reflection of a speaker's call to action during his or her conversation at any time. In terms of Symbl.ai's platform an action-item arises through automatic speech recognition so that a speaker does not have to return to his or her conversation at all to receive his or her action items. These conversation entities as real-time conversation insights come automatically with Symbl.ai.
Step One
The first step is to sign up for an account at Symbl.ai. Register for an account at Symbl (i.e., platform.symbl.ai). Grab both your appId and your appSecret. With both of those you should authenticate either with a cURL command or with Postman so that you receive your x-api-key. Here is an example with cURL:
curl -k -X POST "https://api.symbl.ai/oauth2/token:generate" \
-H "accept: application/json" \
-H "Content-Type: application/json" \
-d "{ \"type\": \"application\", \"appId\": \"<appId>\", \"appSecret\": \"<appSecret>\"}"
Ideally a token server would handle authentication (with code that makes RESTful API call for generating token) so that neither the appSecret nor the appId were ever exposed. However, cURL sets you up immediately anyway.
Step Two
Create a file called .env. In the .env file create constants in the following way for your credentials:
APPID=
APPSECRET
Save the .env file in your project root directory.
Step Three
The third step is to create a file called index.js. In your file called index.js your first step is to set up the variables for 1) installing Symbl.ai's Node.js package symbl-node, 2) handling authentication with environment variables, 3) enabling the microphone.
require('dotenv').config()
const {sdk} = require('symbl-node')
const uuid = require('uuid').v4
// For demo purposes, we're using mic to simply get audio from microphone and pass it on to websocket connection
const mic = require('mic')
const appId = "";
const appSecret = "";
const emailAddress = "user@example.com";
const myName = "Jane Doe";
const sampleRateHertz = 16000
const micInstance = mic({
rate: sampleRateHertz,
channels: '1',
debug: false,
exitOnSilence: 6,
});
Fourth Step
The fourth step is to: 1) initialize the SDK, 2) create a unique ID, 3) start the real-time request
(async () => {
try {
// Initialize the SDK
await sdk.init({
appId,
appSecret,
basePath: 'https://api.symbl.ai',
})
// Need unique Id
const id = uuid()
// Start Real-time Request (Uses Realtime WebSocket API behind the scenes)
const connection = await sdk.startRealtimeRequest({
id,
insightTypes: ['action_item', 'question'],
config: {
meetingTitle: 'My Test Meeting',
confidenceThreshold: 0.7,
timezoneOffset: 480, // Offset in minutes from UTC
languageCode: 'en-US',
sampleRateHertz
},
speaker: {
// Optional, if not specified, will simply not send an email in the end.
userId: emailAddress, // Update with valid email
name: myName
},
handlers: { }
});
Fifth Step
After creating a request to start the SDK's real-time WebSocket, the next step is configure the event driven listeners to handle conversation data in real-time. Symbl.ai's SDK's methods for onSpeechDetected, onMessageResponse, onInsightResponse, or onTopicResponse empower you to log questions. To configure those methods, add the following code to your index.js file:
onSpeechDetected: (data) => {
if (data) {
const {punctuated} = data
console.log('Live: ', punctuated && punctuated.transcript)
console.log('');
}
console.log('onSpeechDetected ', JSON.stringify(data, null, 2));
},
/**
* When processed messages are available, this callback will be called.
*/
onMessageResponse: (data) => {
console.log('onMessageResponse', JSON.stringify(data, null, 2))
},
/**
* When Symbl detects an insight, this callback will be called.
*/
onInsightResponse: (data) => {
console.log('onInsightResponse', JSON.stringify(data, null, 2))
},
/**
* When Symbl detects a topic, this callback will be called.
*/
onTopicResponse: (data) => {
console.log('onTopicResponse', JSON.stringify(data, null, 2))
}
Sixth Step
The sixth step is to log questions in real-time. To do so, configure onInsightResponse to loop through the return data. Symbl.ai's SDK provides an array of data as a response. In the array of the returned data there is a key for questions. After accessing that key through a for-of loop in JavaScript, you log the questions to the console in real-time.
onInsightResponse: (data) => {
console.log("The onInsightResponse is logging.")
console.log('onInsightResponse', JSON.stringify(data, null, 2))
for (const insight of data) {
console.log("The loop is looping.");
console.log("Insight type is: ", insight.type);
if (insight.type === "action_item") {
console.log("Insight content is:", insight.payload.content)
console.log("Insight userId is:", insight.assignee.userId)
console.log("Insight name is:", insight.assignee.name)
}
}
}
Seventh Step
The seventh step is to try it out. To run your application, you have to add sox to your project's root directory. If you run brew, adding sox is as easy as brew install sox. After adding sox, navigate back to the project's root directory. In the project's root directory, run node index.js.
If you added all of the code to your project's root directory correctly, then you would be able to log properly. The full code snippet is here:
require('dotenv').config()
const {sdk} = require('symbl-node')
const uuid = require('uuid').v4
// For demo purposes, we're using mic to simply get audio from microphone and pass it on to websocket connection
const mic = require('mic')
const appId = process.env.APP_ID;
const appSecret = process.env.APP_SECRET;
const emailAddress = "user@example.com";
const myName = "Jane Doe";
//if (APP_ID === undefined || APP_SECRET === undefined) {
// throw "You must define APP_ID and APP_SECRET in your .env file";
//}
const sampleRateHertz = 16000
const micInstance = mic({
rate: sampleRateHertz,
channels: '1',
debug: false,
exitOnSilence: 6,
});
(async () => {
try {
// Initialize the SDK
await sdk.init({
appId,
appSecret,
basePath: 'https://api.symbl.ai',
})
// Need unique Id
const id = uuid()
// Start Real-time Request (Uses Realtime WebSocket API behind the scenes)
const connection = await sdk.startRealtimeRequest({
id,
insightTypes: ['action_item', 'question'],
config: {
meetingTitle: 'My Test Meeting',
confidenceThreshold: 0.7,
timezoneOffset: 480, // Offset in minutes from UTC
languageCode: 'en-US',
sampleRateHertz
},
speaker: {
// Optional, if not specified, will simply not send an email in the end.
userId: emailAddress, // Update with valid email
name: myName
},
handlers: {
/**
* This will return live speech-to-text transcription of the call.
*/
// onSpeechDetected: (data) => {
// if (data) {
// const {punctuated} = data
// console.log('Live: ', punctuated && punctuated.transcript)
// console.log('');
// }
// console.log('onSpeechDetected ', JSON.stringify(data, null, 2));
// },
/**
* When processed messages are available, this callback will be called.
*/
// onMessageResponse: (data) => {
// console.log('onMessageResponse', JSON.stringify(data, null, 2))
// },
// /**
// * When Symbl detects an insight, this callback will be called.
// */
onInsightResponse: (data) => {
console.log("The onInsightResponse is logging.")
console.log('onInsightResponse', JSON.stringify(data, null, 2))
for (const insight of data) {
console.log("The loop is looping.");
console.log("Insight type is: ", insight.type);
if (insight.type === "action_item") {
console.log("Insight content is:", insight.payload.content)
console.log("Insight userId is:", insight.assignee.userId)
console.log("Insight name is:", insight.assignee.name)
}
}
},
/**
* When Symbl detects a topic, this callback will be called.
*/
// onTopicResponse: (data) => {
// console.log('onTopicResponse', JSON.stringify(data, null, 2))
// }
}
});
console.log('Successfully connected. Conversation ID: ', connection.conversationId);
const micInputStream = micInstance.getAudioStream()
/** Raw audio stream */
micInputStream.on('data', (data) => {
// Push audio from Microphone to websocket connection
connection.sendAudio(data)
})
micInputStream.on('error', function (err) {
cosole.log('Error in Input Stream: ' + err)
})
micInputStream.on('startComplete', function () {
console.log('Started listening to Microphone.')
})
micInputStream.on('silence', function () {
console.log('Got SIGNAL silence')
})
micInstance.start()
setTimeout(async () => {
// Stop listening to microphone
micInstance.stop()
console.log('Stopped listening to Microphone.')
try {
// Stop connection
await connection.stop()
console.log('Connection Stopped.')
} catch (e) {
console.error('Error while stopping the connection.', e)
}
}, 60 * 1000) // Stop connection after 1 minute i.e. 60 secs
} catch (e) {
console.error('Error: ', e)
}
})()
Conclusion
After having logged these questions to the console, you are now in a position to determine what extension or integration you add to your conversation to augment its experience in real-time. With third party SaaS such as those designed to enable the transformation of the conversation into a form other than logged questions through connections, transformations or visualizations.
Community
Sign up for an account at Hashnode to start blogging today. If you would like to become more involved with Symbl.ai, our software is open sourced. If you would like to add a new feature to Symbl.ai's node.js SDK, then feel free to clone, fork, or push pull requests to our repository's base repository: github.com/symblai/https://github.com/symbl..
If you have feedback or would like to speak one of our team members about your integration, join our Slack channel or reach out to us directly via email. In addition, Symbl.ai is actively seeking developers to join its development efforts. If you would like to sign up to create content, test our APIs, or create community engagement around our APIs, sign up for our developer programs.