Improving RAG Performance with Translation-Based Embeddings for Non-English Documents

Although multilingual embedding models are advancing rapidly, they still perform more weakly on non-English text than on English. This gap limits how successfully Retrieval-Augmented Generation (RAG) based question-answering systems built in languages other than English find the right context. The paper by Can İşcan and Muhammet Furkan Özara of the Next4biz R&D Center, and Akhan Akbulut of the Next4biz R&D Center and Istanbul Kültür University, focuses on exactly this problem.

In the proposed method, documents are first translated into English and the embedding vectors are produced from that translation, while the text in the original language is stored as metadata. This makes it possible to benefit from strong English embedding models while the answer is still generated from the original-language text of the documents. The team built a question-answering pipeline that uses a large language model for answer generation; using 88 Turkish documents from the Next4biz CSM user manual and 15 human-written question-answer pairs, it compared the conventional approach with the translation-based approach across different embedding models within the RAGAS framework.

With text-embedding-3-large, the best-performing model, the translation-based approach raised context recall from 0.76 to 0.81 and context precision from 0.65 to 0.72. The gain is not the same for every model: with multilingual-e5-large, context precision fell from 0.64 to 0.58. Testing the approach on other languages and with other translation models is left as future work. As stated in our R&D team's project article, this approach is used in Next4biz chatbot systems that work with different large language models.

The paper was presented at ASYU 2024 – the Innovations in Intelligent Systems and Applications Conference – held in Ankara on October 16–18, 2024 and hosted by Gazi University, and appeared in the conference proceedings published by IEEE (pp. 1–6).