Comparing BERT Sentence-Pair Classification and Few-Shot LLM Prompting for Detecting Threat and Solution Framing in German Climate News

Source: arXiv:2606.26489 · Published 2026-06-25 · By Raven Adam, David Maier, Marie Kogler

TL;DR

This paper addresses the automated detection of threat versus solution framing at the sentence level in German-language climate change news articles, a key content distinction that shapes public perception and policy support. It systematically compares two approaches: (1) fine-tuning a monolingual German BERT model (deepset/gbert-large) for sentence-pair classification where the preceding sentence contextualizes the target sentence, and (2) few-shot prompting of an open-weights large language model (meta-llama/llama-4-maverick) using chain-of-thought reasoning and structured JSON outputs including confidence scores. Both approaches employ two independent binary classifiers for threat and solution framing, evaluated on a corpus of 440 Austrian newspaper articles manually annotated with domain-specific detailed coding guidelines. The BERT models outperform the LLM approach, achieving F1 scores of 0.83 versus 0.78 respectively for both framing detection tasks. An ablation study confirms the critical value of providing preceding context for BERT, boosting F1 by roughly 10 points compared to single-sentence classification. Disagreement analysis shows BERT better resolves ambiguous cases where the two models diverge. The study highlights nuanced trade-offs between supervised fine-tuning needing labeled data and fast, deterministic inference versus LLM prompting with no training but higher computational cost and occasional inconsistent reasoning. The work advances computational social science by benchmarking fine-tuned encoders against few-shot prompting for nuanced, domain-specific frame detection in German climate discourse.

Key findings

• Fine-tuned BERT sentence-pair classification achieves F1 scores of 0.83 for both threat and solution framing on the test set.
• Few-shot LLM prompting with Llama 4 Maverick attains F1 scores of 0.78 for both tasks, about 5 points lower than BERT.
• Providing the immediately preceding sentence as context improves BERT’s F1 by approximately 10 points compared to single-sentence input (low 0.70s to 0.83).
• LLM receives full article context but still performs worse than BERT with only one-sentence context, indicating supervised fine-tuning produces stronger task-specific decision boundaries.
• On approximately 20-24% of sentences where BERT and LLM disagree, BERT’s predictions align with ground truth about twice as often (F1 ≈ 0.65 vs. 0.34 for threat and 0.60 vs. 0.35 for solution).
• LLM’s self-reported confidence scores are poorly calibrated: mean confidence > 0.93 despite error rates over 20%, limiting usefulness for threshold filtering.
• LLM’s chain-of-thought reasoning outputs often contain contradictory or unfaithful justifications not reflecting true model decision process.
• Balanced sampling creates test sets of 1,041 threat sentences and 1,655 solution sentences, ensuring equal positive and neutral examples.

Threat model

N/A - This paper focuses on automated classification of news text for framing detection and does not consider adversarial manipulation or attacks on classification models.

Methodology — deep read

1. Threat Model & Assumptions: The adversary in this study is irrelevant since the task is automated content classification. The paper assumes correct manual coding as ground truth. The task is to classify sentences as threat-oriented, solution-oriented, both, or neither. Threat and solution framing are orthogonal binary labels per sentence.

2. Data: The corpus consists of 440 German-language climate news articles from two major Austrian newspapers (Kronen Zeitung: 304 articles; Der Standard: 136 articles) published Jan-Jun 2024. Articles were retrieved using search keywords klima* OR CO2 from the WISO database and manually filtered for thematic relevance. Manual coding followed detailed interdisciplinary guidelines specifying threat and solution categories independently, with iterative coder training and adjudication. Approximately 8% of data was double-coded for reliability (not quantified as Krippendorff's alpha). Sentence segmentation yielded 10,981 sentences: 2,575 threat-labeled and 4,108 solution-labeled. Balanced datasets were created by randomly downsampling neutral sentences to equal positive count per task, producing: threat (5,180 sentences, 4,139 train, 1,041 test) and solution (8,242 sentences, 6,587 train, 1,655 test). Train-test splits were random at sentence level but kept consistent across methods.

3. Architectures:

• BERT-based approach uses deepset/gbert-large (335M parameters) pretrained German BERT. Input is sentence-pair: preceding sentence + target sentence, formatted with [CLS], [SEP] tokens. Two independent binary classifiers are fine-tuned separately for threat and solution detection with AdamW optimizer, lr=5e-5, 6 epochs. When no preceding sentence exists (first sentence), first segment is empty.
• LLM approach uses meta-llama/llama-4-maverick, an open-weight model accessed via API. Classification is done via few-shot prompting with 8 distinct XML-structured prompt sections: role def, speech-act taxonomy, framing definitions, classification rules, positive and negative examples, full article text, and target sentence. Chain-of-thought reasoning requires the model to generate a short rationale before classification output. Outputs are structured JSON with three fields: reason (text), classification (bool), confidence (numeric). Two separate prompts are used for each binary task. Deterministic decoding is applied (temperature=0, top_p=1). Each sentence is processed with separate API calls with retries and parallelization.

4. Training Regime: Only BERT undergoes parameter updates with 6 epochs, batch size unspecified. LLM is inference-only with no fine-tuning, relying on prompt design and in-context examples (4 positives, 4 negatives).

5. Evaluation Protocol: Both methods evaluated on identical held-out test sets (20%) reporting precision, recall, F1 on positive class and accuracy. An ablation study compares BERT sentence-pair versus single-sentence input, quantifying contextual impact. Confidence threshold sweeps were performed for LLM outputs to optimize metrics, but raw threshold 0.0 was optimal. Disagreement analysis isolates test sentences where BERT and LLM disagree to compare accuracy specifically on difficult cases.

6. Reproducibility: Code and data availability not specified in detail; the paper mentions use of public pretrained models and APIs for LLM inference. Manual annotation process detailed. Dataset is closed but based on publicly available newspapers through subscription database. Prompt templates are described in detail. Exact random seeds and split IDs are provided for consistency.

Example: For classifying a single sentence, the BERT model receives the concatenation of the immediately preceding sentence and the target sentence. It outputs a binary label indicating threat or solution framing. The LLM receives a full XML prompt including role, coding rules, examples, and the whole article text with the target sentence highlighted. It outputs a JSON object with chain-of-thought reasoning preceding the classification and confidence score. These outputs are then evaluated against the manual ground truth labels for that sentence to compute precision, recall, and F1.

Technical innovations

• Systematic direct comparison of fine-tuned German BERT sentence-pair classification versus few-shot prompting of an open-weights LLM (Llama 4 Maverick) for climate-specific framing detection.
• Use of sentence-pair input format to incorporate contextual information from the immediately preceding sentence for sentence-level framing classification, showing substantial improvement over single-sentence input.
• Design of detailed multi-section XML prompt for LLM few-shot classification including speech-act taxonomy, coding rules, positive/negative examples, full article context, and chain-of-thought structured output with confidence scoring.
• Confidence-based threshold filtering applied to LLM outputs to explore calibration, finding poor confidence-quality correlation limiting threshold utility.

Datasets

• Austrian Climate News Corpus — 440 articles (304 Kronen Zeitung, 136 Der Standard) from Jan-Jun 2024 — manually annotated for threat and solution framing by domain experts

Baselines vs proposed

• BERT sentence-pair: Threat F1 = 0.83 vs LLM few-shot: Threat F1 = 0.78
• BERT sentence-pair: Solution F1 = 0.83 vs LLM few-shot: Solution F1 = 0.78
• BERT single-sentence: Threat and Solution F1 ≈ low 0.70s vs BERT sentence-pair F1 = 0.83 (approx. 10 point drop)
• On disagreement subsets: BERT Threat F1 = 0.65 vs LLM Threat F1 = 0.34; BERT Solution F1 = 0.60 vs LLM Solution F1 = 0.35

Limitations

• Train-test split performed at sentence level, so sentences from the same article may appear in both splits, possibly inflating performance due to topical overlap.
• Balanced sampling by randomly undersampling neutral sentences discards potentially informative negative data.
• No cross-validation or multiple splits used; only a single random split reported, so sampling variance not quantified.
• Formal intercoder reliability statistics (e.g., Krippendorff's alpha) for manual annotations are missing.
• Only one monolingual German BERT model and one LLM (Llama 4 Maverick) evaluated; generalizability across other models unknown.
• LLM confidence scores poorly calibrated, limiting their utility for filtering or automated quality control.

Open questions / follow-ons

• How does expansion of context window beyond the immediately preceding sentence affect classification performance for framing detection?
• Can supervised fine-tuning of LLMs further close the performance gap to fine-tuned BERT encoders on domain-specific framing tasks?
• What strategies improve calibration of confidence scores from LLM-generated structured outputs to enable reliable post-hoc filtering?
• How do additional BERT variants or multilingual/multitask models affect results in German climate framing detection compared to the monolingual GBERT studied?

Why it matters for bot defense

For practitioners developing automated bot-defense content filters or CAPTCHA challenges that rely on semantic understanding of nuanced language (such as detecting threat or solution framing in user-generated or news content), this study highlights the strengths and weaknesses of fine-tuned transformer encoders versus few-shot LLM prompting. The fine-tuned models produce more reliable and context-sensitive predictions but require labeled training data and computational resources upfront. Few-shot prompting offers immediate deployment without training but at a slight cost to accuracy, and with challenges around confidence calibration and contextual misclassification due to overly broad input windows. Incorporating contextual information via sentence-pair input benefits sentence-level classification substantially, an insight that can inform designing context-aware layers or input representations for automated content analysis in bot-detector pipelines. The demonstration that LLM chain-of-thought outputs may be inconsistent suggests reliance on LLM-generated explanations for security-critical decisions should be cautious. Overall, the findings encourage prioritizing fine-tuned encoder models for precise classification tasks where labeled data exists, reserving LLM prompting as a practical fallback for zero-shot or low-data settings relevant in dynamic CAPTCHA text generation or filtering scenarios.

Cite

@article{arxiv2606_26489,
  title={ Comparing BERT Sentence-Pair Classification and Few-Shot LLM Prompting for Detecting Threat and Solution Framing in German Climate News },
  author={ Raven Adam and David Maier and Marie Kogler },
  journal={arXiv preprint arXiv:2606.26489},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.26489}
}

Read the full paper

• arXiv:2606.26489 (abstract)
• arXiv:2606.26489 (PDF)