21 new feedback metrics for Grammar, Vocab, and Coherence

by Speechace 2 years ago

Assessment and eLearning application developers today rely on the Speechace Spontaneous Speech API to assess language use in open-ended speech. The API evaluates spoken language and returns Grammar, Vocabulary, and Coherence scores aligned with standard rubrics such as CEFR and IELTS.

Each language skill is scored with a dedicated model trained specifically for evaluating speech. These models are often incorrectly assumed to do something similar to Grammarly or the Microsoft Grammar Checker.

In reality Speechace’s Grammar, Vocab, and Coherence models are fundamentally different in several aspects:

  1. The models are specific to spoken language
  2. The models look beyond just accuracy (i.e. errors) and evaluate language on other dimensions such as: range, variety, precision, sophistication, academic language use
  3. The models are optimized for scoring and feedback rather than just correction

Chose your words!

In simple terms, a speaker who uses basic grammar and limited vocabulary but makes no errors should still receive a lower score than a speaker who makes some errors but consistently demonstrates richer, more advanced language.

The words you choose when you speak matter. Each individual is essentially a language model of the grammatical structures, word combinations, connectives, and lexical range that they use. To assess their proficiency we assess all of these attributes through hundreds of features based on the language use.

Today we are announcing 21 new feedback metrics which provide greater insight into Grammar, Vocabulary, and Coherence scores. These metrics provide score explainability and identify specific areas of weakness for practice and improvement.

Anatomy of a feedback metric

Each feedback metric was chosen to measure a dimension of the speaker’s language and reflect how they compare to other non-native speakers in that dimension alone. It is important to note that is it not necessary, and nearly impossible, to score high on every metric all the time.

Each metrics has 3 components:

  • Score: On a scale of 0 to 10
  • Level: Low, Mid, or High
  • Message: A feedback message to the learner

Learners should typically work on areas where their level is low, which coincides with scores below 4. Let’s look at a few real examples.

1. Here is an example result from the vocab.overall_metrics.idiomaticity metric:

2. Here is an example result from the coherence.overall_metrics.verb_diversity metric:

In metrics which assess the overuse of certain connectives, verbs, or adverbs, the result includes a list of examples from the user’s response.

3. And finally here is an example from the grammar.overall_metrics.grammatical_accuracy metric:

The metric indicates the extent to which grammatical accuracy affected the users score, and the errors list provides a list of categorized errors with suggested replacements.

How to use the feedback metrics

To use the new feedback metrics you need to pass the include_ielts_feedback parameter on your call to the Speechace API. You can see a complete listing of all 21 metrics and their descriptions in the Speechace API Documentation.

You can use these metrics to provide specific guidance to users such as:

  • Enhancing a particular aspect of grammatical range (e.g. learning new more advanced noun phrases)
  • Fixing grammatical inaccuracies
  • Increasing use of sophisticated, academic, or idiomatic language
  • Reducing overuse of certain connectives, verbs, and adverbs

We hope these feedback metrics provide an additional level of visibility into each speaker’s proficiency and allow applications to provide more insights into each score, and help learners understand how to improve.

All the best

The Speechace Team