Validation of a large language models-based socio-political sentiment analysis system with a retrieval-augmented generation approach

Abstract

Socio-political discourse on digital platforms generates large volumes of unstructured textual data, posing challenges for reliable sentiment analysis and contextual interpretation. Conventional sentiment analysis approaches often lack contextual grounding, while direct use of Large Language Models (LLMs) may introduce hallucinations, bias, and excessive generalization. To address these issues, this study proposes an LLM-based sentiment analysis system that integrates multi-source data acquisition, zero-shot sentiment classification, Retrieval-Augmented Generation (RAG), and generative reasoning for contextual interpretation. Unlike prior studies that mainly focus on improving model accuracy, this research emphasizes system-level validation of the entire analytical pipeline. The evaluation includes sentiment calibration using a Golden Dataset, data ingestion performance analysis, retrieval quality assessment based on semantic distance, and generative evaluation using an LLM-as-a-Judge framework. Experimental results indicate that the system provides stable data acquisition and produces coherent analytical reports. However, retrieval noise and neutrality bias introduced by LLM alignment mechanisms still affect the groundedness of generated outputs. These findings highlight the importance of retrieval quality and demonstrate the potential of LLMs as reasoning and evaluation components for analyzing complex socio-political discourse.