Post-editing quality: Analysing the correctness and necessity of post-editor corrections
DOI:
https://doi.org/10.52034/lanstts.v16i0.439Keywords:
machine translation, post-editing, post-editing quality, evaluation, translation quality assessmentAbstract
Post-editing (PE) machine translations (MT) has become an increasingly common practice in the translation field in recent years. Research has investigated, among other issues, the types of error corrected by post-editors, but less emphasis has been placed on the corrections themselves and how they reflect MT errors. This article presents a pilot study analysing the edits made by five student post-editors in an English–Finnish post-editing task. We analyse the correctness and necessity of the edits. Our results show that, whereas most edits performed in the task are correct, a significant number of them (34%) are unnecessary. The findings suggest that specific types of edit, such as word-order changes and deletions of personal pronouns, are generally unnecessary for this language pair, which may have implications for post-editing practice and training.
References
Blain, F., Senellart, J., Schwenk, H., Plitt, M., & Roturier J. (2011). Qualitative analysis of post-editing for high quality machine translation. In Proceedings of the 13th Machine Translation Summit (pp. 164–171). Asia-Pacific Association for Machine Translation.
Bojar, O., Chatterjee, R., Federmann, C., Graham, Y. Haddow, B., Huck, M., Jimeno Yepes, A., Koehn, P., Logacheva, V., Monz, C., Negri, M., Neveol, A., Neves, M., Popel, M., Post, M., Rubino, R., Scarton, C., Specia, L., Turchi, M., Verspoor, K., & Zampieri, M. (2016). Findings of the 2016 Conference on machine translation. In Proceedings of the First Conference on Machine Translation (WMT) (pp. 131–198). Stroudsburg, PA: The Association for Computational Linguistics.
Campbell, S. (2000). Critical structures in the evaluation of translations from Arabic into English as a second language. The Translator, 6, 37–58.
Carl, M. (2012). Translog – II: A program for recording user activity data for empirical reading and writing research. In Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC'12), (pp. 4108–4112). European Language Resources Association.
Daems, J., Vandepitte, S., Hartsuiker, R., & Macken, L. (2015). The impact of machine translation error types on post-editing effort indicators. In S. O’Brien & M. Simard (Eds.), Proceedings of Fourth Workshop on Post-editing Technology and Practice (WPTP4) (pp. 31–45). Association for Machine Translation in the Americas.
de Almeida, G. (2013). Translating the post-editor: An investigation of post-editing changes and correlations with professional experience across two romance languages (Doctoral thesis, Dublin City University, Dublin, Ireland). Retrieved from http://doras.dcu.ie/17732/
Englund Dimitrova, B. (2005). Expertise and explicitation in the translation process. Amsterdam: John Benjamins.
Flanagan, M., & Christensen, T.P. (2014). Testing post-editing guidelines: How translation trainees interpret them and how to tailor them for translator training purposes. The Interpreter and Translator Trainer, 8(2), 276–294.
ISO/DIS 18587. (2016). Translation Services – Post-editing of machine translation output: Requirements. Geneva: International Organization for Standardization.
Koby, G.S., & Champe, G.G. (2013). Welcome to the real world: Professional-level translator certification. Translation & Interpreting, 5(1), 156–173.
Koponen, M. (2012). Comparing human perceptions of post-editing effort with post-editing operations. In 7th Workshop on Statistical Machine Translation. Proceedings of the Workshop (pp. 181–190). Stroudsburg, PA: Association for Computational Linguistics.
Koponen, M. (2013). This translation is not too bad: An analysis of post-editor choices in a machine translation post-editing task. In S. O’Brien, M. Simard & L. Specia (Eds.), Workshop Proceeding: Workshop on Post-editing Technology and Practice (WPTP-2) (pp. 1–9). Allschwil: The European Association for Machine Translation.
Koponen, M. (2016). Machine translation post-editing and effort: Empirical studies on the post-editing process (Doctoral thesis, University of Helsinki, Helsinki, Finland). Retrieved from http://hdl.handle.net/10138/160256
Koponen, M., Aziz, W., Ramos, L., & Specia, L. (2012). Post-editing time as a measure of cognitive effort. In S. O'Brien, M. Simard & L. Specia (Eds.), Proceedings of WPTP 2012. AMTA 2012 Workshop on Post-editing Technology and Practice (pp. 11–20).
Koponen, M., & Salmi, L. (2015). On the correctness of machine translation: A machine translation post-editing task. Journal of Specialised Translation, 23, 118–136.
Koskenniemi, K., Lindén, K., Carlson, L., Vainio, M., Arppe, A., Lennes, M., Westerlund, H., Hyvärinen, M., Bartis, I., Nuolijärvi, P., & Piehl, A. (2012). Suomen kieli digitaalisella aikakaudella – The Finnish Language in the Digital Age. META-NET White Paper Series. Heidelberg: Springer.
Krings, H.P. (2001). Repairing texts: Empirical investigations of machine translation post-editing process. G. S. Koby (Translated into English. Original Texte Reparieren, 1994). Kent, OH: The Kent State University Press.
Lacruz, I., Denkowski, M., & Lavie, A. (2014). Cognitive demand and cognitive effort in post-editing. In S. O'Brien, M. Simard & L. Specia (Eds.), Proceedings of the Third Workshop on Post-Editing Technology and Practice (WPTP-3) (pp. 73–84). Association for Machine Translation in the Americas.
Leal Fontes, H. (2013). Evaluating machine translation: Preliminary findings from the first DGT-wide translators’ survey. Languages and Translation, 6, 10–11.
Mikhailov, M. (2015). Minor language, major challenges: The results of a survey into the IT competences of Finnish translators. The Journal of Specialised Translation, 24, 952–975.
Nordman, M. (2015). “Aina irrota vedenkeitin jahka ei kotona apu”: käyttöohjeiden konekäännösten virheanalyysi. [“Always unplug kettle until no help at home”: error analysis of machine translations of kettle user instructions.] (Unpublished master’s thesis). University of Turku, Turku, Finland.
O’Brien, S. (2005). Methodologies for measuring the correlations between post-editing effort and machine translatability. Machine Translation, 19(1), 37–58.
Pirinen. T. (2008). Automatic finite state morphological analysis of Finnish language using open source resources (in Finnish) (Unpublished master’s thesis). University of Helsinki, Helsinki, Finland.
Pirinen, T., Toral, A., & Rubino, R. (2016). Rule-based and statistical morph segments in English-Finnish SMT. In T.A. Pirinen, E. Simon, F.M. Tyers & V. Vincze (Eds.), Proceedings of the Second International Workshop on Computational Linguistics for Uralic Languages (pp. 56–69). Retrieved from http://rgai.inf.u-szeged.hu/project/iwclul/proceedings.pdf
Plitt, M., & Masselot, F. (2010). A productivity test of statistical machine translation post-editing in a typical localisation context. The Prague Bulletin of Mathematical Linguistics, 93, 7–16.
Popović, M., Lommel, A., Burchardt, A., Avramidis, E.& Uszkoreit, H. (2014). Relations between different types of post-editing operations, cognitive effort and temporal effort. In EAMT 2014: 17th Annual Conference of the European Association for Machine Translation (pp. 191–198). Allschwil: The European Association for Machine Translation.
Silfverberg, M., Ruokolainen, T., Lindén, K. and Kurimo, M. (2016). FinnPos: An open-source morphological tagging and lemmatization toolkit for Finnish. Language Resources and Evaluation, 50(4), 863–878.
Snover, M., Dorr, B., Schwartz, R., Micciulla, L., & Makhoul, J. (2006). A study of translation edit rate with targeted human annotation. In Proceedings of the 7th Conference of the Association for Machine Translation of the Americas: Visions for the Future of Machine Translation (pp. 223–231). Association for Machine Translation of the Americas.
TAUS. (2010). Machine translation post-editing guidelines. Retrieved from https://evaluation.taus.net/resources/guidelines/post-editing/machine-translation-post-editing-guidelines.
Temizöz, Ö. (2016). Postediting machine translation output: Subject-matter experts versus professional translators. Perspectives, 24(4), 646–665.
Temnikova, I. (2010). A cognitive evaluation approach for a controlled language post-editing experiment. In N. Calzolari, K. Choukri, B. Maegaard, J. Mariani, J. Odijk, S. Piperidis, M. Rosner & D. Tapias (Eds.), Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10). European Language Resources Association (ELRA).
Temnikova, I., Zaghouani, W., Vogel, S., & Habash. N. (2016). Applying the cognitive machine translation evaluation approach to Arabic. In N. Calzolari, K. Choukri, T. Declerck, S. Goggi, M. Grobelnik, B. Maegaard, J. Mariani, H. Mazo, A. Moreno, J. Odijk & S. Piperidis (Eds.), Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016) (pp. 3644–3651). European Language Resources Association (ELRA).
Tiedemann, J., Ginter, F., & Kanerva, J. (2015). Morphological segmentation and OPUS for Finnish–English machine translation. In Tenth Workshop on Statistical Machine Translation. Proceedings of the Workshop (pp. 177–183). Red Hook, NY: Association for Computational Linguistics.
Downloads
Published
How to Cite
Issue
Section
License
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under the CC BY-NC 4.0 Deed that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal. The material cannot be used for commercial purposes.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
