Comparative study of pre-trained RoBERTa sentiment models and zero-shot LLM on indonesian and english texts

Abstract

The growth of user-generated content on social media has increased the need for effective sentiment analysis methods. Although fine-tuned transformer-based models and zero-shot large language models (LLMs) have both been applied to sentiment classification, comparisons across languages under unified evaluation settings remain limited. This study examines the trade-offs between task-specific fine-tuning and instruction-based zero-shot inference for multilingual sentiment classification. Experiments were conducted using two publicly available Twitter sentiment datasets in Indonesian and English, each annotated into three sentiment classes. Fine-tuned RoBERTa-based models were evaluated on full test sets, while all models, including a zero-shot LLM, were compared using an identical controlled subset. Performance was assessed using accuracy and macro-averaged precision, recall, and F1-score, with macro F1-score as the primary metric. The results show that fine-tuned RoBERTa-based models achieve stable and balanced performance across sentiment classes, with monolingual models consistently outperforming multilingual variants. Under controlled evaluation, zero-shot LLMs demonstrate competitive performance in English but remain less effective in Indonesian, indicating that their effectiveness is influenced by language resource availability. Overall, this study provides a controlled comparison of the strengths and limitations of fine-tuned and zero-shot approaches for multilingual sentiment classification.