Turkish BERT-Based Language Models for Interpreting Customer Feedback in CRM

Customer relationship management (CRM) solutions are among the primary tools for strengthening customer bonds and increasing satisfaction. Prepared within the scope of the Next4biz R&D Center's "Customer Satisfaction Score Detection" project, this paper looks for an answer to the question of whether adding deep learning and natural language processing (NLP) methods to CRM platforms can produce more accurate satisfaction scores from customer feedback.

In the study, BERT-based language models trained for Turkish are compared on the task of classifying customer feedback as "complaint / not a complaint". The experiments used 360 labeled records in total: about 200 selected from the notification records of a Turkish fitness chain, plus synthetic examples added to balance the classes. The records were split evenly between the two classes, and the models were evaluated with 10-fold cross-validation. The authors note that entity recognition can also help identify recurring patterns in feedback, and that these patterns can be used for product development and new customer acquisition. Among the five models (BERTurk, TSA, ConvBert, ElecTRa, XLM-RoBERTa), ElecTRa achieved the highest precision (0.85) and matched BERTurk on accuracy, recall and F1 score (0.84); it shared the highest ROC AUC value (0.93) with ConvBert.

The practical benefit highlighted by the paper is the ability to monitor the performance of customer representatives through feedback flagged as complaints: by identifying the topics where errors concentrate, targeted training can be planned for representatives, and service quality can be improved through this feedback loop.

The study was presented at UBMK 2024 (9th International Conference on Computer Science and Engineering), held in Antalya on 26–28 October 2024, was published by IEEE in the conference proceedings (pp. 227–232), and received the "Best Paper" award at the conference.

Award: UBMK 2024 Best Paper.