21 new feedback metrics for Grammar, Vocab, and Coherence

by Speechace 10 months ago

Assessment and eLearning application developers today rely on the Speechace Spontaneous Speech API to assess language use in open-ended speech. The API evaluates spoken language and returns Grammar, Vocabulary, and Coherence scores aligned with standard rubrics such as CEFR and IELTS.

Each language skill is scored with a dedicated model trained specifically for evaluating speech. These models are often incorrectly assumed to do something similar to Grammarly or the Microsoft Grammar Checker.

In reality Speechace’s Grammar, Vocab, and Coherence models are fundamentally different in several aspects:

  1. The models are specific to spoken language
  2. The models look beyond just accuracy (i.e. errors) and evaluate language on other dimensions such as: range, variety, precision, sophistication, academic language use
  3. The models are optimized for scoring and feedback rather than just correction

Chose your words!

In simple terms, a speaker who uses basic grammar and limited vocabulary but makes no errors should still receive a lower score than a speaker who makes some errors but consistently demonstrates richer, more advanced language.

The words you choose when you speak matter. Each individual is essentially a language model of the grammatical structures, word combinations, connectives, and lexical range that they use. To assess their proficiency we assess all of these attributes through hundreds of features based on the language use.

Today we are announcing 21 new feedback metrics which provide greater insight into Grammar, Vocabulary, and Coherence scores. These metrics provide score explainability and identify specific areas of weakness for practice and improvement.

Anatomy of a feedback metric

Each feedback metric was chosen to measure a dimension of the speaker’s language and reflect how they compare to other non-native speakers in that dimension alone. It is important to note that is it not necessary, and nearly impossible, to score high on every metric all the time.

Each metrics has 3 components:

  • Score: On a scale of 0 to 10
  • Level: Low, Mid, or High
  • Message: A feedback message to the learner

Learners should typically work on areas where their level is low, which coincides with scores below 4. Let’s look at a few real examples.

1. Here is an example result from the vocab.overall_metrics.idiomaticity metric:

				
					"idiomaticity": 
{
    "score": 3,
    "level": "low",
    "message": "Your use of word combinations is unusual and typical of adaptation from another language. You should study and use combinations such as adjective-noun and adverb-verb which are associated with each other and found in idiomatic language. "
}

2. Here is an example result from the coherence.overall_metrics.verb_diversity metric:

In metrics which assess the overuse of certain connectives, verbs, or adverbs, the result includes a list of examples from the user’s response.

				
					"verb_diversity":
{
    "score": 1,
    "level": "low",
    "message": "Your response is low on verb diversity. See the list of most overused verbs and practice avoiding repeating verbs especially in consecutive sentences.",
    "examples":
    [
      "drive",
      "buy",
      "have"
    ]
}

3. And finally here is an example from the grammar.overall_metrics.grammatical_accuracy metric:

The metric indicates the extent to which grammatical accuracy affected the users score, and the errors list provides a list of categorized errors with suggested replacements.

				
					"overall_metrics": 
{
    "grammatical_accuracy": 
    {
      "score": 6,
      "level": "mid"
    },
    ...
}
"errors" :
[
{
    "category": "STYLE",
    "message": "Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.",
    "span": [257, 259],
    "matched_text": "It",
    "replacements":
    [
        "Furthermore, it",
        "Likewise, it",
        "Not only that, but it"
    ]
},
...
]

How to use the feedback metrics

To use the new feedback metrics you need to pass the include_ielts_feedback parameter on your call to the Speechace API. You can see a complete listing of all 21 metrics and their descriptions in the Speechace API Documentation.

You can use these metrics to provide specific guidance to users such as:

  • Enhancing a particular aspect of grammatical range (e.g. learning new more advanced noun phrases)
  • Fixing grammatical inaccuracies
  • Increasing use of sophisticated, academic, or idiomatic language
  • Reducing overuse of certain connectives, verbs, and adverbs

We hope these feedback metrics provide an additional level of visibility into each speaker’s proficiency and allow applications to provide more insights into each score, and help learners understand how to improve.

All the best

The Speechace Team