C017

Predicting Human Choices in The Wild: A Study Using The Chatbot Arena Dataset

Anik Sen, Riadul Islam Rabbi

AFFILIATION
Faculty of Information Science & Technology, Multimedia University

Description of Invention

This study explores the use of machine learning to predict human choices in chatbot conversations using the Chatbot Arena Dataset. An LSTM model was trained to identify patterns between conversational features and user preferences. The model achieved 37% accuracy, surpassing random guessing (25% in a four-choice scenario), but still faced challenges in predicting complex, real-world conversations. A log loss of 1.16 indicates room for improvement in model confidence and accuracy. These results highlight the need for better feature extraction and advanced models to enhance chatbot performance and align responses more effectively with user preferences.