EVALUATING A LARGE LANGUAGE MODEL CHATBOT FOR ENHANCING HISTORICAL LITERACY ON PANGLIMA MAYU OF THE SUMBAWA SULTANATE

DOI: https://doi.org/10.33650/codex.v1i2.15485
Authors

(1) * Gusti Arsyad   (Universitas Binawan)  
        Indonesia
(*) Corresponding Author

Abstract


The digital preservation of regional history in the modern era requires innovative approaches to bridge the gap between traditional archives and user accessibility. This study presents the development and evaluation of the Panglima Mayu ChatBot, a specialized Large Language Model (LLM) designed to enhance historical literacy regarding the 18th-century Sumbawa Sultanate. The chatbot was developed using a prompt engineering approach based on curated historical materials and a culturally grounded system prompt to ensure factual consistency and alignment with Adat Samawa values. Deployed on an autoscale server environment (2 vCPU / 4 GiB RAM), the system was evaluated through a quantitative survey involving 45 respondents, including general users, students, and educators. The results indicate positive technical and educational perceptions. the system achieved a 97.8% satisfaction rate for accessibility and speed, while 84.4% of users reported increased insight into Islamic historical narratives. Crucially, 77.8% of respondents perceived the chatbot's responses as accurate, indicating a high level of user confidence in the information provided. However, these findings reflect perceived accuracy rather than expert-verified historical accuracy, which requires further validation by subject-matter experts. Furthermore, the system showed strong practical utility, with 73.3% of respondents endorsing the ChatBot as a viable quick reference for tourism and educational purposes. These findings suggest that grounded AI personas can successfully preserve cultural integrity—specifically Adat Samawa values—while providing an interactive alternative to traditional textbooks. This research provides a scalable framework for digitizing regional heroic narratives across Indonesia

Keywords

Digital Heritage, Historical Literacy, Panglima Mayu, Sumbawa Sultanate.



Full Text: PDF



References


Aji, A. F., Winata, G. I., Koto, F., Awaluddin, M. A., Romadhony, M., Anwari, F., Sipahutar, H., Strobl, S., & Purwarianti, A. (2022). One country, many languages: A comprehensive study of natural language processing for Indonesian languages. *Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics*. https://aclanthology.org/2022.acl-long.500

Arsyad, G., & Rifai, A. (2025). Jejak dakwah Islam Panglima Mayu era Sultan Muhammad Jalaluddin Syah II Kesultanan Sumbawa abad ke-18. *Jurnal Al-Islah, 17*(1). https://al-islah.staisumbawa.ac.id/index.php/al-islah/article/view/10

Arsyad, G., Rifai, A., Wijaya, I., & Haris, A. (2025). *Napak tilas dakwah Islam Panglima Ma-yu dan keturunannya di Kesultanan Sumbawa abad 18–20*. Nur Hidayah Press.

Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., Brynjolfsson, E., Buch, S., Card, D., Castellon, R., Chatterji, N., Chen, A., Creel, K., Davis, J. Q., Demszky, D., ... Liang, P. (2021). *On the opportunities and risks of foundation models* (arXiv Preprint No. 2108.07258). arXiv. https://doi.org/10.48550/arXiv.2108.07258

Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Edunov, S., Knobloch, M., Grave, E., & Zettlemoyer, L. (2020). Unsupervised cross-lingual representa-tion learning at scale. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics* (pp. 8440–8451). Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.acl-main.747

Giaccardi, E. (Ed.). (2012). *Heritage and social media: Understanding heritage in a participa-tory culture*. Routledge. https://doi.org/10.4324/9780203151448

Ji, Z., Lee, N., Frieske, R., Yu, T., Su, J., Xu, B., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023). Survey of hallucination in natural language generation. *ACM Computing Sur-veys, 55*(12), 1–38. https://doi.org/10.1145/3571730

Romein, C. A., Kemman, M., Birkholz, J. M., Baker, J., de Gruijter, M., Meroño-Peñuela, A., Ries, T., Ros, R., & Scagliola, S. (2020). State of the field: Digital history. History, 105(365), 291–312. https://doi.org/10.1111/1468-229X.12969

Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., Riedel, S., & Kiela, D. (2020). Retrieval-augmented genera-tion for knowledge-intensive NLP tasks. *Advances in Neural Information Processing Sys-tems, 33*, 9459–9475. https://proceedings.neurips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html

Michel-Villarreal, R., Vilalta-Perdomo, E., Canavari, M., & Matopoulos, A. (2023). Large lan-guage models in education: A review of opportunities and challenges. *International Journal of Educational Technology in Higher Education, 20*(1). https://doi.org/10.1186/s41239-023-00426-1

Ricklefs, M. C. (2008). *A history of modern Indonesia since c. 1200* (4th ed.). Palgrave Mac-millan.

Ridge, M. (Ed.). (2014). *Crowdsourcing our cultural heritage*. Ashgate Publishing.

Rifai, A. (2024). *Panglima Mayu ChatBot interface*. Retrieved February 19, 2026, from https://panglima-chat-bot--akhmadrifaiusah.replit.app/

Suryomintardjo, B. (2015). *Sejarah kerajaan-kerajaan lokal di Nusa Tenggara Barat: Menelusu-ri jejak Kesultanan Sumbawa*. Pustaka Daerah.

UNESCO. (2023). *Digital transformation and access to cultural heritage* [Policy report]. UNESCO. https://www.unesco.org/

Weller, T. (2013). History in the digital age. Routledge. https://doi.org/10.4324/9780203093953


Dimensions, PlumX, and Google Scholar Metrics

10.33650/codex.v1i2.15485


Refbacks

  • There are currently no refbacks.


Copyright (c) 2026 Gusti Arsyad

Creative Commons License
 

CODEX: Journal of Software Engineering
Published by Lembaga Penerbitan, Penelitian dan Pengabdian kepada Masyarakat (LP3M) of Nurul Jadid University, Probolinggo, East Java, Indonesia.