CultureTalk-ID: A Multi-Task Dialogue Benchmark for Cultural Commonsense in Indonesian Local Languages

Source: arXiv:2607.21016 · Published 2026-07-23 · By Muhammad Dehan Al Kautsar, Salsabila Pranida, Bilal Elbouardi, Fajri Koto

TL;DR

This paper addresses the mismatch between existing Indonesian cultural commonsense benchmarks, which rely on short, isolated prompts, and the inherently dialogic nature of cultural knowledge expression. The authors introduce CULTURETALK-ID, the first large-scale, dialogue-based benchmark focusing on Indonesian and its local languages, covering 4,496 human-annotated dialogues spanning 11 languages and 13 culturally salient topics. The dataset was created through a rigorous multi-stage human-in-the-loop pipeline ensuring cultural authenticity and linguistic accuracy. The benchmark includes three complementary tasks: dialogue-based multiple-choice cultural commonsense reasoning, culturally faithful bidirectional machine translation between Indonesian and local languages, and language steering to generate contextually grounded outputs in requested languages.

Evaluation across a variety of large language models (LLMs)—including proprietary state-of-the-art, multilingual open models, and Southeast Asian-centric models—demonstrates that while proprietary models achieve relatively strong performance on these tasks, open-source and multilingual models lag significantly, especially on local language dialogues and generation tasks. Additional culturally grounded pretraining improves performance, but challenges remain in capturing nuanced cultural context and generating fluent, accurate local language utterances. These results underline a substantive gap in current models’ ability to understand and generate culturally rooted language in conversational Indonesian settings, calling for future work on richer pretraining and broader cultural and linguistic coverage.

Key findings

• CULTURETALK-ID contains 4,496 multi-turn dialogues across 11 Indonesian languages, covering 13 culturally salient topics and 10 provinces.
• Proprietary models like Gemini-2.5-flash achieve the highest dialogue-based cultural commonsense MCQ accuracy (~87.7%), outperforming multilingual and SEA-centric open-source models by 10–30 percentage points.
• Performance drops by 6.7 points on average for proprietary models and over 10 points for open-source models when moving from Indonesian to local language dialogues, showing increased difficulty with low-resource local languages.
• Supervised fine-tuning on the CULTURETALK-ID translation task improves BLEU scores from around 20 to over 40 for Indonesian→local language.
• Open-source models achieve low success rates in language steering, with the best (SEA-LION-v3.5-8B) generating only 27.5% of utterances in the correct local language, whereas proprietary models reach over 90%.
• Additional Indonesian-centric pretraining (e.g., Sahabat-AI-v1) consistently improves performance on downstream cultural commonsense tasks across provinces.
• LLM-as-a-judge evaluation strongly correlates with human assessment of machine translation quality, with mean absolute distance below 1 and accuracy-at-one around 0.83.
• Fluency and register scores lag behind adequacy and terminology in machine translation, highlighting challenges in generating natural, culturally appropriate style.

Threat model

n/a — The focus is on evaluating LLMs' cultural commonsense understanding and generation rather than security threats. The study assumes non-adversarial use cases and does not consider manipulative or deceptive adversaries.

Methodology — deep read

The study begins from the observation that cultural knowledge in Indonesia is deeply embedded in conversational context across many local languages, yet prior benchmarks rely mostly on isolated, short prompts. To build a more authentic and challenging benchmark, the authors construct CULTURETALK-ID with these steps:

1. Threat Model & Assumptions: While not a security adversary scenario, the benchmark tests LLMs’ ability to understand and generate culturally nuanced dialogue. The assumption is that adversarial manipulation is not modeled; rather, the focus is on faithful language and cultural comprehension across Indonesian and low-resource local languages.

2. Data Collection & Provenance: The dataset starts by transforming two existing Indonesian cultural commonsense multiple-choice datasets: IndoCulture and COPAL-ID. The question-answer pairs from these serve as topics and candidate endings.

3. Dialogue Generation: GPT-5 was prompted to generate Indonesian multi-turn dialogues for each cultural scenario, with the final utterance omitted. This turn becomes the multiple-choice target. These dialogues serve as a scaffold for the human annotation.

4. Human Annotation & Translation: Teams of fluent annotators from 10 Indonesian provinces who are native speakers of Indonesian and their local language (22 annotators total) reviewed GPT-5 dialogues. They corrected content for cultural appropriateness and answer correctness, translated dialogues into local languages allowing non-literal yet fluent mappings, and revised answer choices to remove shortcut cues and unnatural options.

5. Quality Control: The dataset underwent multi-phase QC, including cross-checking between annotators, insertion of deliberate errors (5% of samples) to test attention, and querying annotators on MCQ answers to verify clarity. Dialogues failing quality thresholds were revised.

6. Dataset Statistics: The final dataset has 4,496 dialogues (2,980 region-specific), 29,420 turns, and covers 11 languages including Indonesian. Average dialogue length is 60 words with ~6.5 utterances.

7. Experimental Setup: Evaluations use three tasks: (a) dialogue-based MCQ (likely-based scoring for open models), (b) bidirectional machine translation (Indonesian ↔ local languages) evaluated by BLEU, BERTScore, and LLM-as-judge, and (c) language steering where models generate responses in Indonesian or indicated local language.

8. Models Evaluated: Proprietary (GPT-5.1, Gemini-2.5-flash, Cohere-Command-A), open multilingual (Qwen3-8B, Llama-3.1-8B, Gemma-2-9B), and SEA-centric LLMs (Sailor2-8B, SEA-LION-v3.5, Sahabat-AI-v1) of comparable ~8-9B parameters.

9. Training & Fine-tuning: Selected models (Gemma-2-9B, Sahabat-AI-v1) fine-tuned supervisedly on CULTURETALK-ID for MT task.

10. Evaluation Protocol: MCQ accuracy measured with and without province/language context to study context effects; MT evaluated zero-shot and fine-tuned; language steering assessed for correctness of target language (Glot-LID) and quality (BLEU, LLM judge). Results include statistical variances over multiple runs where applicable.

11. Human Evaluation & Correlation: Fluent native speakers validated select MT outputs to confirm reliability of LLM-as-judge scoring.

All code and full annotation protocols are internal or anonymized at this time, although detailed appendix descriptions facilitate reproducibility. However, the local language data is sensitive and may require special access. An example end-to-end is the adaptation of IndoCulture multiple-choice across 13 cultural topics, converted to Indonesian dialog by GPT-5, then human-annotated and translated into regional languages like Wamesa, with multiple-choice endings verified for appropriateness.

Technical innovations

• Creation of the first human-curated multi-turn dialogue benchmark embedding cultural commonsense across Indonesian and 10 local languages with parallel data.
• Introduction of a three-task evaluation suite combining dialogue-based cultural commonsense multiple-choice reasoning, culturally grounded bidirectional machine translation, and language steering for controlled linguistic output.
• Multi-stage annotation pipeline with native speakers including cross-checking, error-injection for quality control, and flexible non-literal translation strategies to preserve cultural and linguistic authenticity.
• Demonstration that additional culturally and linguistically focused pretraining can improve LLM cultural commonsense understanding and cross-lingual transfer.

Datasets

• CULTURETALK-ID — 4,496 dialogues covering 11 Indonesian languages and 13 cultural topics — constructed from IndoCulture and COPAL-ID with human revisions and parallel translations

Baselines vs proposed

• Gemini-2.5-flash MCQ accuracy: 87.77% vs multilingual average ~56.6%
• GPT-5.1 MCQ accuracy: 82.31% vs open source SEA-centric average ~69.6%
• Multilingual MT zero-shot BLEU Indonesian→local languages: 17.62–20.54 vs proprietary Gemini-2.5-flash 28.36
• SFT fine-tuned GEMMA-2-9B BLEU improves from 20.54 to 43.14
• Language steering correct language generation: Gemini-3-flash 92.7% vs SEA-LION-v3.5 27.5%
• Open-source model Qwen3-8B language steering: 1.5% correct generation rate

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.21016.

Fig 1: Overview of the CULTURETALK-ID benchmark. CULTURETALK-ID supports three tasks: (1) dialogue-

Fig 2: The CULTURETALK-ID dataset creation pipeline. It consists of four phases: dialogue generation;

Fig 3 (page 4).

Fig 4: An example of the dataset transformation process from an original IndoCulture instance into

Fig 9: Task 3 - Language steering’s user prompt.

Limitations

• Dataset covers only 10 provinces out of 38 in Indonesia, limiting cultural and linguistic representativeness.
• Topic distribution is unbalanced, with some domains like agriculture underrepresented.
• Dialogue difficulty and cultural reasoning complexity vary unevenly across provinces.
• Human evaluation correlation with LLM-as-judge was performed in detail only for Javanese and Minangkabau languages.
• Recruiting fluent annotators for many local languages is challenging due to language standardization and resource availability.
• Open-source models struggle significantly on local languages and generation tasks, limiting current applicability for low-resource languages.

Open questions / follow-ons

• How to extend coverage to more Indonesian provinces and local languages with balanced cultural topics?
• What are effective pretraining or adaptation strategies to improve LLM proficiency in extremely low-resource local languages with diverse dialects?
• How to better model dialogue-level cultural context and pragmatic knowledge to reduce performance gaps in generation tasks?
• Can multimodal signals (e.g., visual or audio cues) enhance culturally grounded dialogue understanding and generation?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, CULTURETALK-ID highlights that culturally grounded understanding in conversational AI remains challenging, especially for multilingual and low-resource contexts. CAPTCHA systems aiming to leverage language or cultural reasoning as a bot-detection mechanism can benefit from this benchmark by incorporating dialogue-based cultural commonsense tasks that require localized cultural knowledge, making it harder for automated bots lacking such grounding to succeed. Furthermore, the findings about poor open-source model performance on local languages and language steering suggest that bot detection methods relying on culturally nuanced multilingual dialogue understanding must carefully consider model limitations and likely focus on proprietary or fine-tuned models with better cultural competence. Finally, the multi-task nature of CULTURETALK-ID—involving reasoning, translation, and language control—provides a useful evaluation framework for assessing bot capabilities in diverse linguistic and cultural environments, relevant for CAPTCHA design choices targeting Indonesia's rich linguistic landscape.

Cite

@article{arxiv2607_21016,
  title={ CultureTalk-ID: A Multi-Task Dialogue Benchmark for Cultural Commonsense in Indonesian Local Languages },
  author={ Muhammad Dehan Al Kautsar and Salsabila Pranida and Bilal Elbouardi and Fajri Koto },
  journal={arXiv preprint arXiv:2607.21016},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.21016}
}

Read the full paper

• arXiv:2607.21016 (abstract)
• arXiv:2607.21016 (PDF)