HLT Evaluation Portal

On statistical machine translation

Chiao — 2006-10-09T08:46:25Z

For further information on research, campaigns, conferences, software and data regarding statistical machine translation and its evaluation, please refer to www.statmt.org

60- Speech Recognition

Chiao — 2006-04-21T14:45:50Z

Speech recognition, also known as automated speech recognition (ASR) or speech-to-text (STT) is a process by which a program or a system transcribes an acoustic speech signal to text.

Systems generally perform two different types of recognition: single-word and continuous speech recognition. Continuous speech is more difficult to handle because of a variety of effects such as speech rate, coarticulation, etc. Today's state-of-the-art systems are able to transcribe unrestricted continuous speech from broadcast data with acceptable performance [1].

Evaluation measures

For ASR evaluation, the criterion is recognition accuracy, one commonly used measure is word error rate or the related metric word accuracy rate (WER [2]), also used in machine translation evaluation.

The method used in the current DARPA speech recognition evaluation involves comparing system transcription of the input speech to the reference (i.e., transcription by a human expert), using algorithms to score agreement at the word level. More higher-level metrics such as setence error rate asn concept error rate can be applied regarding different applications.

Communication style (i.e., speaker independent, spontaneous speech, etc), vocabulary size, language model and usage conditions are also important features which can affect the performance of a speech recognizer for a particular task.

Related projects

- AURORA distributed speech recognition project
- CORETEX Improving Core Speech Recognition Technology
- EARS Effective Affordable Reusable Speech-To-Text, DARPA's research program
- NESPOLE-NEgotiating through SPOken Language in E-commerce.

Voice User Interface

Chiao — 2006-04-21T10:31:52Z

Voice user interface (VUI) is considered as an alternative to the traditional graphical user interface (GUI). Instead of using keyboard and mouse to input data, VUI enables human computer interaction by using speech signal.

Evaluation of Voice User Interfaces

In general, evaluation methods and metrics for speech recognition and speech synthesis can be applied to VUI evaluations. There are two main approaches: technology and usibility evaluation.

Both subjective and objective tests can be used for VUI evaluations. Subjective evaluation relates to human factors. It consists in judging parameters by reference to users' opinions (satisfaction). Objective evaluation produces subject independant parameter assessment. It consists in measuring task completion rates and times.

According to different purposes of evaluation, one can differentiate diagnostic evaluation, performance evaluation and adequacy evaluation. For more information, see >What is HLT evaluation?.

Evaluation Metrics

Technology evaluation measures the performance of technical components, such as speech recognizers and speech synthesizers. The most common speech recognition evaluation metric is word error rate (WER). Intelligibility and pleasantness are two factors which can be performance of speech synthesizers.

Usibility evaluation uses both subjective and objective metrics. James A. Larson, et al. proposed ten criteria for measuring effective voice user interfaces:

Caller satisfaction
Ease of use
Quality of audio output
Perceived first-call resolution rate
Time-to-task
Task rate
Task completion time
Correct transfer
Abandonment rate
Containment rate

The 4 first criteria are concerned with subjective metrics and the others with objective metrics. Detailed description of these metrics is presented in Ten Criteria for Measuring Effective Voice User Interfaces.

Specifications

Chiao — 2006-01-13T15:45:54Z

What to evaluate?

ELRA carries out evaluation activities for the following technologies :

Text processing	Speech processing	Mutimodal interface
Information Retrieval	Speech Synthesis	Multimodal Person Tracking
Question Answering	Speech Translation	Audiovisual Speech Recognition
Machine Translation	Automatic Speech Recognition	Multimodal Person Identification
Automatic Summarization	Broadcast News Transcription
Parsing	Acoustic Person Tracking
Multilingual Text Alignment	Acoustic Speaker Identification
Terminology Extraction	Speech Activity Detection

How to evaluate?

According to the different technologies, ELRA produces detailed evaluation requirements for competitive campaign and individual system evaluation.

Evaluation

On the basis of the defined requirements, ELRA develops test materials to perform the evaluation.

Evaluation report

ELRA provides an evaluation report.

For further information, please contact us.

NIST HLT Evaluations

Chiao — 2005-11-29T18:07:00Z

NIST Human Language Technology evaluations

The Information Access Division (IAD), part of NIST Information Technology Laboratory, provides measurements and standards to advance technologies dealing with access to multimedia information.

The NIST HLT programs supports the development of language technologies by conducting open community evaluations: speech recognition, speaker recognition, spoken language understanding, information search, retrieval, and filtering, document understanding, summarization, interactive speech interfaces, etc.

Text REtrieval Conference (TREC)

Document Understanding Conferences (DUC)

Large Vocabulary Conversational Speech Recognition (LVCSR)

Speaker Recognition

Topic Detection and Tracking (TDT)

Automatic Content Extraction (ACE)

Rich Transcription

MT evaluation for DARPA's GALE program.

DARPA's programs

Chiao — 2005-11-29T18:06:59Z

The DARPA's [3] Information Processing Technology Office (IPTO) supports research and development programs on human language technology and National Institute of Standards and Technology (NIST) conduct annual performance evaluations of these technologies.

The DARPA/IPTO HLT programs and the relative evaluations conducted by NIST are:

- CAST Compact Aids for Speech Translation (formerly Babylon, multilingual Speech to Speech Translation interfaces for users in combat and other field environments)

- EARS, Effective Affordable Reusable Speech-To-Text. NIST's Rich Transcription evaluations

- GALE, Global Autonomous Language Exploitation. See related evaluations conducted by NIST

- TIDES, Translingual Information Detection, Extraction and Summarization. NIST's related evaluations: MT (Machine Translation), DUC (Document Understanding Conferences), ACE (Automatic Content Extraction)

France

Chiao — 2005-11-29T18:06:16Z

- EVALDA (in French only), HLT Evaluation projects in Technolangue (in French only) programs supported by the French Ministry of Research. 8 evaluation campaigns were launched involving French language technologies such as speech synthesis (EVASY), machine translation (CESTA), human-machine interaction (MEDIA), terminology extraction (CESART), automatic broadcast news transcriptions (ESTER), multilingual text alignment (ARCADE2), question/answering (EQUER), syntactic parsing (EASY).

- GRACE evaluation campaign-Grammaire et Ressources pour les Analyseurs de Corpus et leur Evaluation (completed project), sponsored by CNRS aims at evaluating morphosyntactic taggers for French.

- Aupelf-Uref's FRANCIL ARCs program-Research actions of FRANCIL network on Language Engineering cover human language technology evaluation.

- AMARYLLIS-a national evaluation campaign for information retrieval systems for the French language.

Northern Europe

Chiao — 2005-11-29T18:06:14Z

Nordic Treebank Network

In order to promote the Nordic language technologies, a Nordic language technology research program was initiated by NordForsk, an independent institution operating under the Nordic Council of Ministers.

The goal of Nordic Treebank Network project under the Nordic Language Technology Program is to promote research related to creation, usage and evaluation of treebanks in the Nordic countries (Denmark, Estonia, Finland, Iceland, Norway and Sweden).

Projects related to evaluation within the network

- TREPIL, Norwegian treebank project, started from 2004, aims to build semi-automatically a Norwegian treebank including linguistic principles, database design, resources, tools, evaluation, etc.

ELRA's Validation Centre

- ELRA's Validation Centre for Written Language Resources (WLR) was established in November 2002 at Center for Sprogteknologi (CST) in Denmark.

The main task of the Validation Centre is to develop a methodology for the validation of Written Language Resources (WLR) and provide validation manuals for lexical resources and for corpora.

Germany

Chiao — 2005-11-29T18:06:12Z

The DFKI Evaluation Center for Language Technology, part of COLLATE project, was funded as a public service to the HLT R&D community, industry and works on user-centred assessment of Language Technology methods, technologies, applications and specific products.

Verbmobil is a project of the Germany's Federal Ministry of Education, Science, Research and Technology. It was focused on the development of speech-to-speech translation technology for spontaneous dialogs in mobile situations.

China HTRDP Evaluation (863 Evaluation)

Chiao — 2005-11-29T18:05:28Z

The National Key Technologies R&D Program (NKTRDP), the National Natural Science Foundation (NSFC), 863 Program and the National Basic Research Program (also called 973 Program) have formed the main body of the state programs for science and technology.

The HTRDP evaluation plan of Chinese language processing and intelligent human-machine interface, started in 1995, also known as 863 evaluation is sponsored by China's National High Technology Research and Development Program (HTRDP, 863 Program).

The goal of the 863 evaluation is to provide infrastructural support for research and development on Chinese information processing and intelligent human-machine interface technology in order to speed up the transfer of technology from research area into commercial products.

A series of evaluations of Chinese language technology such as machine translation, information retrieval or speech recognition is conducted each year. The evaluation is performed by using human assessments or automatic metrics in following steps:

Registration
Test
Assessments
Evaluation results generation and notification
Evaluation workshop