Machine translation output for Arabic-to-English translation of legal texts : A comparative study between AI Tools /

Abstract

The effectiveness of machine translation (MT) in the legal domain requires close evaluation using appropriate quality assessment models. This study assesses the translation quality of two advanced MT systems, Gemini and ChatGPT by analyzing their English translations of three Arabic Memorandums of Association. The TAUS Dynamic Quality Framework (DQF) was adopted as the primary evaluation metric, with a focus on error typology and frequency to measure translation performance. The research adopts a quantitative approach, examining the outputs in terms of accuracy and fluency, while identifying and categorizing errors. A total of 1,022 errors were recorded and analyzed: 425 in ChatGPT translations and 597 in Gemini. The findings indicate distinct tendencies in each system: ChatGPT often omits source text content, while Gemini exhibits a tendency toward over-translation. ChatGPT’s output showed a higher percentage of accuracy-related errors, particularly mistranslations, whereas Gemini was more prone to over translation errors. The study underscores the ongoing necessity of human post-editing in legal translation workflows. It also emphasizes the importance of incorporating domain-specific training data and tailored quality assurance (QA) modules to improve MT output in legal contexts. Ultimately, this research contributes to the growing body of literature on MT evaluation by offering insight into the strengths, limitations, and error patterns of emerging AI-powered translation tools. The researcher recommends further exploration of additional TAUS error categories, such as style and locale, and calls for broader experimentation with other MT systems to reflect the rapid development of AI in legal translation. Keywords: Machine Translation, Legal Translation, Neural Machine Translation, TAUS Error Typology, ChatGPT, Gemini.