“OLR Challenge 2021”版本间的差异
来自cslt Wiki
（→Oriental Language Recognition (OLR) 2021 Challenge） |
|||
| （相同用户的67个中间修订版本未显示） | |||
| 第1行： | 第1行： | ||
=Oriental Language Recognition (OLR) 2021 Challenge= | =Oriental Language Recognition (OLR) 2021 Challenge= | ||
| − | Oriental languages involve interesting specialties. The OLR challenge series aim at improve the performance of language recognition systems and speech recognition systems within multilingual scenarios. Following the success of [[OLR Challenge 2016]], [[OLR Challenge 2017]], [[OLR Challenge 2018]] | + | Oriental languages involve interesting specialties. The OLR challenge series aim at improve the performance of language recognition systems and speech recognition systems within multilingual scenarios. Following the success of [[OLR Challenge 2016]], [[OLR Challenge 2017]], [[OLR Challenge 2018]], [[OLR Challenge 2019]], and [[OLR Challenge 2020]] the new challenge in 2021 focuses on more practical and challenging problems, with four tasks: |
* Task 1: constrained LID is a close-set identification task, which means the language of each utterance is among the known 13 target languages, but utterances were recorded in different environments. And only the data provided by the organizer can be used to build the LID system. | * Task 1: constrained LID is a close-set identification task, which means the language of each utterance is among the known 13 target languages, but utterances were recorded in different environments. And only the data provided by the organizer can be used to build the LID system. | ||
| − | * Task 2: unconstrained LID is also a close-set identification task, but test data from wild, which means utterances are obtained from real-life environments. It is therefore more challenging than the constrained task. In this task, any data (except the evaluation data) you can access is allowed to build the system | + | * Task 2: unconstrained LID is also a close-set identification task, but test data from wild, which means utterances are obtained from real-life environments. It is therefore more challenging than the constrained task. In this task, any data (except the evaluation data) you can access is allowed to build the system. |
* Task 3: constrained multilingual ASR is a data resources constrained task, only the data provided by the organizer can be used. | * Task 3: constrained multilingual ASR is a data resources constrained task, only the data provided by the organizer can be used. | ||
| 第14行： | 第14行： | ||
* Jul. 23, challenge registration open. | * Jul. 23, challenge registration open. | ||
| − | |||
==Data== | ==Data== | ||
| 第21行： | 第20行： | ||
For OLR 2021 Challenge, a standard test set OLR21-test is also provided. | For OLR 2021 Challenge, a standard test set OLR21-test is also provided. | ||
| − | OLR16-OL7 is provided by Speechocean (www.speechocean.com), and | + | OLR16-OL7 is provided by Speechocean (www.speechocean.com), and OLR17-OL3 is provided by Tsinghua University, Northwest Minzu University and Xinjiang University, under the [http://cslt.riit.tsinghua.edu.cn/mediawiki/index.php/Asr-project-nsfc M2ASR project] supported by NSFC. |
The features for OLR16-OL7 involve: | The features for OLR16-OL7 involve: | ||
| 第32行： | 第31行： | ||
* The License for the data is [[AP16-OL7-licence-AP17|here]] | * The License for the data is [[AP16-OL7-licence-AP17|here]] | ||
| − | The features for | + | The features for OLR17-OL3 involve: |
* Mobile channel | * Mobile channel | ||
| 第43行： | 第42行： | ||
* The License for the data is [[AP17-OL3-licence|here]] | * The License for the data is [[AP17-OL3-licence|here]] | ||
| − | + | For the OLR 2021 Challenge, the trials of the four tasks will be divided into two subsets: a progress subset, and a test subset. The progress subset will comprise 30% of the trials and will be used to monitor progress in the leaderboard. The remaining 70% of the trials will form the test subset, and will be used to generate the official results which are the base of the final ranking. The OLR21-test database is the standard test set for the OLR 2021 challenge, which contains two parts: OLR21-cross-domain-test and OLR21-wild-test. | |
| − | * | + | |
| − | + | *OLR21-cross-domain-test: This subset is designed for three tasks: the constrained LID task, the constrained multilingual ASR task, and the unconstrained multilingual ASR task. It contains 13 languages, and was recorded by different recording equipments in different environments. The 13 languages are Indonesian, Japanese, Russian, Korean, Vietnamese, Mandarin, Cantonese (China), Sichuanese (China), Shanghainese (China), Hokkien (China), Tibetan (China), Kazakh (China), and Uyghur (China). | |
| − | * | + | |
| + | * OLR21-wild-test: This subset is designed for the unconstrained LID task, which contains 17 languages: Indonesian, Japanese, Russian, Korean, Vietnamese, Thai, Malay, Telugu, Hindi, English (British and American), Kazakh (China), Tibetan (China), Uyghur (China), Mandarin, Sichuan (China), Shanghainese (China), Hokkien (China). Utterances in this subset are obtained from reallife environments. | ||
==Evaluation plan== | ==Evaluation plan== | ||
Refer to the following paper: | Refer to the following paper: | ||
| + | Binling Wang, Wenxuan Hu, Jing Li, Yiming Zhi, Zheng Li, Qingyang Hong, Lin Li, Dong Wang, Liming Song and Cheng Yang: OLR 2021 Challenge: Datasets, Rules and Baselines, submitted to APSIPA ASC 2021. | ||
| + | [[媒体文件:OLR 2021 Plan.pdf|pdf]] | ||
| − | + | ==Baseline Systems== | |
| − | + | * The recipes for LID baselines. [https://github.com/Snowdar/asv-subtools#3-olr-challenge-2021-baseline-recipe-language-identification] | |
| − | * The | + | * The recipes for ASR baselines.[[媒体文件:OLR2021 ASR BASELINE.tar.gz| [2]]] |
==Participation rules== | ==Participation rules== | ||
| − | * Participants from both academy and industry are welcome | + | * Participants from both academy and industry are welcome. |
* Publications based on the data provided by the challenge should cite the following paper: | * Publications based on the data provided by the challenge should cite the following paper: | ||
| 第72行： | 第74行： | ||
'''Zheng Li, Miao Zhao, Qingyang Hong, Lin Li, Zhiyuan Tang, Dong Wang, Liming Song and Cheng Yang: AP20-OLR Challenge: Three Tasks and Their Baselines, submitted to APSIPA ASC 2020.''' [https://arxiv.org/pdf/2006.03473.pdf pdf] | '''Zheng Li, Miao Zhao, Qingyang Hong, Lin Li, Zhiyuan Tang, Dong Wang, Liming Song and Cheng Yang: AP20-OLR Challenge: Three Tasks and Their Baselines, submitted to APSIPA ASC 2020.''' [https://arxiv.org/pdf/2006.03473.pdf pdf] | ||
| + | |||
| + | '''Binling Wang, Wenxuan Hu, Jing Li, Yiming Zhi, Zheng Li, Qingyang Hong, Lin Li, Dong Wang, Liming Song and Cheng Yang: OLR 2021 Challenge: Datasets, Rules and Baselines, submitted to APSIPA ASC 2021.'''[[媒体文件:OLR 2021 Plan.pdf|pdf]] | ||
==Important dates== | ==Important dates== | ||
| − | * | + | * Jul. 23, challenge registration open. |
| + | * Jul. 30, training set release. | ||
* Oct. 1, register deadline. | * Oct. 1, register deadline. | ||
| − | * | + | * Nov. 1, progress subset release. |
| − | * Nov. | + | * Nov. 15, test subset release. |
| − | + | * Nov. 31, 24:00, Beijing time, submission deadline. | |
* Dec. 10, results announcement. | * Dec. 10, results announcement. | ||
| + | |||
(Due to the COVID-19, the seminar and award ceremony will be adjusted according to the actual situation.) | (Due to the COVID-19, the seminar and award ceremony will be adjusted according to the actual situation.) | ||
| 第86行： | 第92行： | ||
If you intend to participate the challenge, or if you have any questions, comments or suggestions about the challenge, please send email to the organizers ( | If you intend to participate the challenge, or if you have any questions, comments or suggestions about the challenge, please send email to the organizers ( | ||
| − | '''<span style="color:red"> | + | '''<span style="color:red">olr_challenge@163.com</span>''' ). For participants, the following information is required, also please sign the '''[[媒体文件:olr2021_license.pdf | <span style="color:red"> Data License Agreement</span>]]''' on behalf of an organization/company of speech research/technology, |
and send back the scanned copy by email. | and send back the scanned copy by email. | ||
- Team Name: | - Team Name: | ||
| − | - Institute: | + | - Institute & Nationality: |
- Participants: | - Participants: | ||
- Duty person: | - Duty person: | ||
| 第98行： | 第104行： | ||
* Qingyang Hong, Xiamen University [[https://speech.xmu.edu.cn/2015/1010/c5243a82406/page.htm home]] | * Qingyang Hong, Xiamen University [[https://speech.xmu.edu.cn/2015/1010/c5243a82406/page.htm home]] | ||
| − | * Lin Li, Xiamen University [[https://speech.xmu.edu.cn/ | + | * Lin Li, Xiamen University [[https://speech.xmu.edu.cn/lilin/list.htm home]] |
| − | * | + | * Binling Wang, Xiamen University |
| + | * Wenxuan Hu, Xiamen University | ||
* Dong Wang, Tsinghua University [[http://wangd.cslt.org home]] | * Dong Wang, Tsinghua University [[http://wangd.cslt.org home]] | ||
| − | |||
* Ming Li, Duke-Kunshan University | * Ming Li, Duke-Kunshan University | ||
* Xiaolei Zhang, NWPU | * Xiaolei Zhang, NWPU | ||
| 第109行： | 第115行： | ||
=Ranking list= | =Ranking list= | ||
| − | The Oriental Language Recognition (OLR) Challenge | + | The Oriental Language Recognition (OLR) Challenge 2021, co-organized by Xiamen University, CSLT@Tsinghua University, Duke-Kunshan University, Northwestern Polytechnical University and Speechocean, was completed with a great success. |
| + | |||
| + | ==Overview== | ||
| + | There are totally <span style="color:red"> '''47'''</span> teams that registered this challenge. | ||
| + | Until the deadline of submission, <span style="color:red">'''27'''</span> teams submitted their results. | ||
| + | The submissions have been ranked in terms of the 4 language recognition tasks respectively, | ||
| + | one is constrained LID, the second one is unconstrained LID, the third one is constrained ASR, and the fourth one is unconstrained ASR. | ||
| + | |||
| + | More details and history about the challenge, see [[媒体文件:OLR2021 Challenge Summary-final.pdf | OLR2021 Challenge Summary]] . | ||
| + | |||
| + | ==Task1== | ||
| + | |||
| + | [[文件:olr21-constrained-lid-task.png]] | ||
| + | |||
| + | [[文件:olr21-constrained-lid-task_.png]] | ||
| + | |||
| + | ==Task2== | ||
| + | |||
| + | [[文件:olr21-unconstrained-lid-task.png]] | ||
| + | |||
| + | [[文件:olr21-unconstrained-lid-task_.png]] | ||
| + | |||
| + | ==Task3== | ||
| + | |||
| + | [[文件:olr21-constrained-asr-task.png]] | ||
| + | |||
| + | [[文件:olr21-constrained-asr-task_.png]] | ||
| + | |||
| + | ==Task4== | ||
| + | |||
| + | [[文件:olr21-unconstrained-asr-task.png]] | ||
| + | |||
| + | [[文件:olr21-unconstrained-asr-task_.png]] | ||
| + | |||
| + | ==Top system description== | ||
| + | *[[媒体文件:X-Voice.pdf | Descriptions]] from X-Voice. | ||
| + | *[[媒体文件:TalTech.pdf | Descriptions]] from TalTech. | ||
| + | *[[媒体文件:funspeech.pdf | Descriptions]] from funspeech. | ||
| + | *[[媒体文件:nisp_speech.pdf | Descriptions]] from nisp_speech. | ||
| + | *[[媒体文件:CCDL.pdf | Descriptions]] from CCDL. | ||
| + | *[[媒体文件:OLR_BIT.pdf | Descriptions]] from OLR_BIT. | ||
| + | *[[媒体文件:RoyalFlush.pdf | Descriptions]] from RoyalFlush. | ||
| + | *[[媒体文件:Huawei AMS.pdf | Descriptions]] from Huawei AMS. | ||
| + | |||
| + | |||
| + | [[媒体文件:OLR2021 Challenge Summary-final.pdf | OLR2021 Challenge Summary]] | ||
2022年1月15日 (六) 09:31的最后版本
Oriental Language Recognition (OLR) 2021 Challenge
Oriental languages involve interesting specialties. The OLR challenge series aim at improve the performance of language recognition systems and speech recognition systems within multilingual scenarios. Following the success of OLR Challenge 2016, OLR Challenge 2017, OLR Challenge 2018, OLR Challenge 2019, and OLR Challenge 2020 the new challenge in 2021 focuses on more practical and challenging problems, with four tasks:
- Task 1: constrained LID is a close-set identification task, which means the language of each utterance is among the known 13 target languages, but utterances were recorded in different environments. And only the data provided by the organizer can be used to build the LID system.
- Task 2: unconstrained LID is also a close-set identification task, but test data from wild, which means utterances are obtained from real-life environments. It is therefore more challenging than the constrained task. In this task, any data (except the evaluation data) you can access is allowed to build the system.
- Task 3: constrained multilingual ASR is a data resources constrained task, only the data provided by the organizer can be used.
- Task 4: unconstrained multilingual ASR is a task with unconstrained data resources and any data can be used for training and optimization.
News
- Jul. 23, challenge registration open.
Data
The challenge is based on two multilingual databases, OLR16-OL7 that was designed for the OLR challenge 2016, and OLR17-OL3 database that was designed for the OLR challenge 2017. For OLR 2021 Challenge, a standard test set OLR21-test is also provided.
OLR16-OL7 is provided by Speechocean (www.speechocean.com), and OLR17-OL3 is provided by Tsinghua University, Northwest Minzu University and Xinjiang University, under the M2ASR project supported by NSFC.
The features for OLR16-OL7 involve:
- Mobile channel
- 7 languages in total
- 71 hours of speech signals in total
- Transcriptions and lexica are provided
- The data profile is here
- The License for the data is here
The features for OLR17-OL3 involve:
- Mobile channel
- 3 languages in total
- Tibetan provided by Prof. Guanyu Li@Northwest Minzu Univ.
- Uyghur and Kazak provided by Prof. Askar Hamdulla@Xinjiang University.
- 35 hours of speech signals in total
- Transcriptions and lexica are provided
- The data profile is here
- The License for the data is here
For the OLR 2021 Challenge, the trials of the four tasks will be divided into two subsets: a progress subset, and a test subset. The progress subset will comprise 30% of the trials and will be used to monitor progress in the leaderboard. The remaining 70% of the trials will form the test subset, and will be used to generate the official results which are the base of the final ranking. The OLR21-test database is the standard test set for the OLR 2021 challenge, which contains two parts: OLR21-cross-domain-test and OLR21-wild-test.
- OLR21-cross-domain-test: This subset is designed for three tasks: the constrained LID task, the constrained multilingual ASR task, and the unconstrained multilingual ASR task. It contains 13 languages, and was recorded by different recording equipments in different environments. The 13 languages are Indonesian, Japanese, Russian, Korean, Vietnamese, Mandarin, Cantonese (China), Sichuanese (China), Shanghainese (China), Hokkien (China), Tibetan (China), Kazakh (China), and Uyghur (China).
- OLR21-wild-test: This subset is designed for the unconstrained LID task, which contains 17 languages: Indonesian, Japanese, Russian, Korean, Vietnamese, Thai, Malay, Telugu, Hindi, English (British and American), Kazakh (China), Tibetan (China), Uyghur (China), Mandarin, Sichuan (China), Shanghainese (China), Hokkien (China). Utterances in this subset are obtained from reallife environments.
Evaluation plan
Refer to the following paper: Binling Wang, Wenxuan Hu, Jing Li, Yiming Zhi, Zheng Li, Qingyang Hong, Lin Li, Dong Wang, Liming Song and Cheng Yang: OLR 2021 Challenge: Datasets, Rules and Baselines, submitted to APSIPA ASC 2021. pdf
Baseline Systems
- The recipes for LID baselines. [1]
- The recipes for ASR baselines. [2]
Participation rules
- Participants from both academy and industry are welcome.
- Publications based on the data provided by the challenge should cite the following paper:
Dong Wang, Lantian Li, Difei Tang, Qing Chen, AP16-OL7: a multilingual database for oriental languages and a language recognition baseline, APSIPA ASC 2016. pdf
Zhiyuan Tang, Dong Wang, Yixiang Chen, Qing Chen: AP17-OLR Challenge: Data, Plan, and Baseline, APSIPA ASC 2017. pdf
Zhiyuan Tang, Dong Wang, Qing Chen: AP18-OLR Challenge: Three Tasks and Their Baselines, submitted to APSIPA ASC 2018. pdf
Zhiyuan Tang, Dong Wang, Liming Song: AP19-OLR Challenge: Three Tasks and Their Baselines, submitted to APSIPA ASC 2019. pdf
Zheng Li, Miao Zhao, Qingyang Hong, Lin Li, Zhiyuan Tang, Dong Wang, Liming Song and Cheng Yang: AP20-OLR Challenge: Three Tasks and Their Baselines, submitted to APSIPA ASC 2020. pdf
Binling Wang, Wenxuan Hu, Jing Li, Yiming Zhi, Zheng Li, Qingyang Hong, Lin Li, Dong Wang, Liming Song and Cheng Yang: OLR 2021 Challenge: Datasets, Rules and Baselines, submitted to APSIPA ASC 2021.pdf
Important dates
- Jul. 23, challenge registration open.
- Jul. 30, training set release.
- Oct. 1, register deadline.
- Nov. 1, progress subset release.
- Nov. 15, test subset release.
- Nov. 31, 24:00, Beijing time, submission deadline.
- Dec. 10, results announcement.
(Due to the COVID-19, the seminar and award ceremony will be adjusted according to the actual situation.)
Registration procedure
If you intend to participate the challenge, or if you have any questions, comments or suggestions about the challenge, please send email to the organizers ( olr_challenge@163.com ). For participants, the following information is required, also please sign the Data License Agreement on behalf of an organization/company of speech research/technology, and send back the scanned copy by email.
- Team Name: - Institute & Nationality: - Participants: - Duty person: - Hompage or published papers in speech field of person/organization/company:
Organization Committee
- Qingyang Hong, Xiamen University [home]
- Lin Li, Xiamen University [home]
- Binling Wang, Xiamen University
- Wenxuan Hu, Xiamen University
- Dong Wang, Tsinghua University [home]
- Ming Li, Duke-Kunshan University
- Xiaolei Zhang, NWPU
- Liming Song, Speechocean
- Cheng Yang, Speechocean
Ranking list
The Oriental Language Recognition (OLR) Challenge 2021, co-organized by Xiamen University, CSLT@Tsinghua University, Duke-Kunshan University, Northwestern Polytechnical University and Speechocean, was completed with a great success.
Overview
There are totally 47 teams that registered this challenge. Until the deadline of submission, 27 teams submitted their results. The submissions have been ranked in terms of the 4 language recognition tasks respectively, one is constrained LID, the second one is unconstrained LID, the third one is constrained ASR, and the fourth one is unconstrained ASR.
More details and history about the challenge, see OLR2021 Challenge Summary .
Task1
Task2
Task3
Task4
Top system description
- Descriptions from X-Voice.
- Descriptions from TalTech.
- Descriptions from funspeech.
- Descriptions from nisp_speech.
- Descriptions from CCDL.
- Descriptions from OLR_BIT.
- Descriptions from RoyalFlush.
- Descriptions from Huawei AMS.
OLR2021 Challenge Summary
