Knowledge Distillation: A Method for Making Neural Machine Translation More Efficient

Jooste, Wandri, Haque, Rejwanul and Way, Andy (2022) Knowledge Distillation: A Method for Making Neural Machine Translation More Efficient. Information, 13 (2). 1-14 (88). ISSN 2078-2489

Full text not available from this repository.

Abstract

Neural machine translation (NMT) systems have greatly improved the quality available from machine translation (MT) compared to statistical machine translation (SMT) systems. However, these state-of-the-art NMT models need much more computing power and data than SMT models, a requirement that is unsustainable in the long run and of very limited benefit in low-resource scenarios. To some extent, model compression—more specifically state-of-the-art knowledge distillation techniques—can remedy this. In this work, we investigate knowledge distillation on a simulated low-resource German-to-English translation task. We show that sequence-level knowledge distillation can be used to train small student models on knowledge distilled from large teacher models. Part of this work examines the influence of hyperparameter tuning on model performance when lowering the number of Transformer heads or limiting the vocabulary size. Interestingly, the accuracy of these student models is higher than that of the teachers in some cases even though the student model training times are shorter in some cases. In a novel contribution, we demonstrate for a specific MT service provider that in the post-deployment phase, distilled student models can reduce emissions, as well as cost purely in monetary terms, by almost 50%.

Item Type:	Article
Uncontrolled Keywords:	NMT; Green AI; knowledge distillation; CO2 savings
Subjects:	Q Science > QA Mathematics > Electronic computers. Computer science T Technology > T Technology (General) > Information Technology > Electronic computers. Computer science P Language and Literature > P Philology. Linguistics > Language Services
Divisions:	School of Computing > Staff Research and Publications
Depositing User:	Clara Chan
Date Deposited:	21 Feb 2022 15:39
Last Modified:	21 Feb 2022 15:39
URI:	https://norma.ncirl.ie/id/eprint/5471

Actions (login required)

View Item