Automatic relevance and coherence assessment for spoken languages

One of the toughest problems in assessment of spoken language proficiency using speech recognition is automatically scoring the relevance and coherence in a candidate’s spoken response. No matter how good the candidate’s pronunciation, fluency, grammar and vocabulary are but if the candidate is not cogent then they must not get a high score.

Today, we are excited to announce two new breakthrough capabilities in our premium API offering:

A. Relevance assessment: This capability assesses whether a candidate’s spoken response is related, sensible and specific to an arbitrary assessment prompt.

B. Coherence assessment: This capability assesses the level of connectedness and logical flow of different parts in a candidate’s response.

Note that our relevance capability accepts any arbitrary prompt and does not require pre-training on a specific prompt. Further note that both capabilities automatically account for the imperfections in automatic speech recognition. This is a gigantic leap in language assessment technology and puts enormous power in the hands of language learning providers who can now build extremely credible spoken language assessment solutions.

Examples

Let us review a few examples that illustrate our new capabilities. Consider an assessment prompt: “Why do you want to work for British Airways?”

The first candidate provides an irrelevant response as can be heard in the below audio:

In this case, our API correctly returns the relevance result as False.

Now consider a relevant albeit casually spoken response from a second candidate:

In this case, our API correctly returns the relevance result as True.

Notice that the relevance API also takes care of cases wherein the candidate just repeats the assessment prompt text as in the below audio:

In this case, the relevance API will correctly return False. Note that although the speaker is repeating the assessment prompt verbatim but the relevance API also rejects answers that are inexact or synonymous representations of the assessment prompt. In future we plan to provide additional granular relevance scores to indicate which parts of the response are relevant vs which parts are not relevant.

Let us now review a few examples for coherence API. Note that unlike the relevance API, the coherence API provides an IELTS style continuous decimal score between 1-9. Consider the following assessment prompt: “Do you think parents should monitor children’s internet use?”

Our first candidate provides a not so coherent answer:

Our coherence API will score the above audio as 5.8, which is a relatively low coherence score.

Now listen to our second candidate who provides a much more coherent answer:

In this case the coherence API will provide a score of 8.0, which is a near perfect score.

We hope you found the above examples intriguing. If you’d like to try out any of the above capabilities then please reach out to us for an API Key on the Speechace API plan page. You may also e-mail us directly at contact@speechace.com. We look forward to hearing from you!