TY - GEN
T1 - Pre-training and Evaluating Transformer-based Language Models for Icelandic
AU - Daðason, Jón Friðrik
AU - Loftsson, Hrafn
N1 - Funding Information: This project was funded by the Language Technology Programme for Icelandic 2019-2023 (Nikulásdóttir et Funding Information: This project was funded by the Language Technology Programme for Icelandic 2019-2023 (Nikulásdóttir et al., 2020). The programme, which is managed and coordinated by Almannarómur10, is funded by the Icelandic Ministry of Education, Science and Culture. This research was also supported with Cloud TPUs from Google's TPU Research Cloud (TRC). Publisher Copyright: © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0.
PY - 2022
Y1 - 2022
N2 - In this paper, we evaluate several Transformer-based language models for Icelandic on four downstream tasks: Part-of-Speech tagging, Named Entity Recognition. Dependency Parsing, and Automatic Text Summarization. We pre-train four types of monolingual ELECTRA and ConvBERT models and compare our results to a previously trained monolingual RoBERTa model and the multilingual mBERT model. We find that the Transformer models obtain better results, often by a large margin, compared to previous state-of-the-art models. Furthermore, our results indicate that pre-training larger language models results in a significant reduction in error rates in comparison to smaller models. Finally, our results show that the monolingual models for Icelandic outperform a comparably sized multilingual model.
AB - In this paper, we evaluate several Transformer-based language models for Icelandic on four downstream tasks: Part-of-Speech tagging, Named Entity Recognition. Dependency Parsing, and Automatic Text Summarization. We pre-train four types of monolingual ELECTRA and ConvBERT models and compare our results to a previously trained monolingual RoBERTa model and the multilingual mBERT model. We find that the Transformer models obtain better results, often by a large margin, compared to previous state-of-the-art models. Furthermore, our results indicate that pre-training larger language models results in a significant reduction in error rates in comparison to smaller models. Finally, our results show that the monolingual models for Icelandic outperform a comparably sized multilingual model.
KW - Evaluation
KW - Icelandic
KW - Language Models
KW - Transformer
UR - https://www.scopus.com/pages/publications/85144359038
M3 - Conference contribution
T3 - 2022 Language Resources and Evaluation Conference, LREC 2022
SP - 7386
EP - 7391
BT - 2022 Language Resources and Evaluation Conference, LREC 2022
A2 - Calzolari, Nicoletta
A2 - Bechet, Frederic
A2 - Blache, Philippe
A2 - Choukri, Khalid
A2 - Cieri, Christopher
A2 - Declerck, Thierry
A2 - Goggi, Sara
A2 - Isahara, Hitoshi
A2 - Maegaard, Bente
A2 - Mariani, Joseph
A2 - Mazo, Helene
A2 - Odijk, Jan
A2 - Piperidis, Stelios
PB - European Language Resources Association (ELRA)
T2 - 13th International Conference on Language Resources and Evaluation Conference, LREC 2022
Y2 - 20 June 2022 through 25 June 2022
ER -