Announcing the Score Task API
One of the most widely used features of the Speechace Spontaneous Speech API is Relevance assessment. Relevance assessment allowed API callers to identify whether a response was relevant to the question prompt presented to the user.
As we studied millions of uses of relevance assessment, we identified scenarios where in addition to relevance scoring the degree of effectiveness of a response was needed. If the speaker was asked to analyze or present a conclusion from a particular business chart, we wanted to know how well they have accomplished that task and not just if their response was on topic.
Introducing Task Achievement
Task Achievement simply is how well a response accomplished the task posed by the question. The Task score is independent of language use, fluency, or pronunciation. To receive a high or the maximum task score, the speaker must comprehend the question fully and provide a response that addresses every aspect of it. The Task score is continuous and therefore a partial response receives a partial score.
To enable scoring of Task Achievement we introduce the Score Task API with initial support for the following task types:
-
describe-image: The speaker is presented with an image and asked to describe the details, relationships, and conclusion to be drawn from elements of the image.
-
retell-lecture: The speaker listens to a 1-2 minute lecture and is asked to summarize the lecture focusing on key elements, concepts and conclusions from the lecture.
-
answer-question: The speaker is presented with a short question which typically requires a one or two word answer.
We believe these initial task types allow for a variety of uses to assess comprehension, knowledge, logic, and problem solving. And we are eager to hear about additional task types you would like to see supported. Please email us at contact@speechace.com with your ideas or requests.
The Score Task API also supports different modes to offer a flexibility in combining task scores and language scores in assessment:
-
include_speech_score: You can include Speech scoring or not along with the task score. Speech scoring will also assess: Pronunciation, Fluency, Grammar, Vocabulary, and Coherence of the response.
-
user_audio_file or user_audio_text: You can pass the speaker’s response as either audio or text. This means you can use task scoring with a written response as well.
The above means that you can use the Score Task API and get the task score alone or in addition to all the Speechace Spontaneous API scores. It also means you don’t need to pass audio to get a task score.
Here are some examples of use. The full request/response examples are available at the API Docs:
Answer Question
task_score: 0 (incorrect) or 1 (correct).
task_question: What do you call a system of government in which people vote for the people who will represent them?
Example response 1:
{
"status": "success",
"task_score": {
"type": "answer-question",
"version": "0.1",
"score": 0,
"transcript": "elections"
},
"quota_remaining": -1,
"version": "9.7"
}
Example response 2:
{
"status": "success",
"task_score": {
"type": "answer-question",
"version": "0.1",
"score": 1,
"transcript": "A democracy?"
},
"quota_remaining": -1,
"version": "9.7"
}
Describe Image
task_score: 0 to 5.
task_context: This bar chart illustrates the declining trend related to the percent of U.S. workforce engaged in farm labor in the 19th century. In 1840, for example, around 69% of the U.S. workforce was engaged in farm labor; in 1860, almost 60% of the U.S. workforce was engaged in farm labor; in 1880, only 50% of the U.S. workforce was engaged in farm labor; and in 1900, less than 40% of the U.S. workforce was engaged in farm labor.
Example response 1:
{
"status": "success",
"task_score": {
"type": "describe-image",
"version": "0.1",
"score": 0,
"transcript": "This is a beautiful image infront of me with a chart depicting many colors and numbers. I can see 1995, 1996, 1997, 1998, 1999 and France's national debt."
},
"quota_remaining": -1,
"version": "9.7"
}
Example response 2:
{
"status": "success",
"task_score": {
"type": "describe-image",
"version": "0.1",
"score": 4,
"transcript": "This bar graph shows the percent of US workforce engaged in farm labor, and that's data from 1840 to 1900. Ear now starting with 1840, the percentage was 70 percentage. After that there is a gradual decrease in the number of workforce engaged in farm labor to 60 percentage in 1860 and further down to 18 around 50% in 1880. And then in 1900 it decreased to 40 percentage. Overall, there is a continuous decrease in the engagement in the farm sector."
},
"quota_remaining": -1,
"version": "9.7"
}
The Score Task API is currently available in all regions by invitation only. For access please email us at support@speechace.com and we will be glad to provide your Speechace Premium API subscription with access.
All the best
The Speechace Team
One Reply to “Announcing the Score Task API”
Comments are closed.
I wish Speechace worked in Spanish and Spanish learners!!!