BnBERT-iPET: Sparse Few-Shot Language Modeling for Bengali via Lottery Ticket Pruning

Source: arXiv:2608.05104 · Published 2026-08-05 · By Sajib Hossain, Md Kamrus Samad, Anan Ghosh, Labib Imam Chowdhury, Nabeel Mohammed

TL;DR

This paper addresses the challenge of developing efficient language models for Bengali, a resource-constrained language with limited computing capacity and data availability. Large pretrained models like BERT deliver state-of-the-art NLP performance but are computationally expensive and environmentally costly, making them impractical for many users and languages like Bengali. To solve this, the authors propose BnBERT-iPET, which combines iterative pattern exploiting training (iPET) to enable few-shot learning with Lottery Ticket Hypothesis-based pruning to achieve extreme model sparsity (90%) without sacrificing performance. They build a diverse Bengali text corpus (BanglaDDS) and fine-tune a lightweight, ALBERT-based BERT variant on this corpus, then apply pruning and iPET. Extensive evaluation on six downstream Bengali tasks shows that their pruned model, at just 10% of the original size, performs competitively with larger models such as Bangla Electra, Indic-BERT, and XLM-RoBERTa, while significantly reducing model size and computational demands.

Key findings

• BnBERT-iPET Pruned model achieves 90% sparsity via Lottery Ticket pruning while maintaining comparable performance on six Bengali NLP tasks.
• On emotion classification, BnBERT-iPET Pruned attains 50.3% accuracy and 48.1% F1, outperforming Bangla Electra and matching Indic-BERT.
• In news categorization, the pruned model reaches 88.48% accuracy, improving over Bangla Electra's 80.4% but slightly lower than Indic-DistilBERT (89.0%).
• Authorship classification accuracy is 97.67% for the pruned model, exceeding XLM-RoBERTa baseline (93.8%) and Indic-BERT (95.2%).
• Perplexity for BnBERT-iPET is substantially lower (1.046) than BnBERT baseline (3.87), indicating better language modeling.
• The pruned model has higher perplexity (46.85), reflecting compression trade-offs but still effective downstream performance.
• On punctuation restoration, the pruned model achieves an F1 of 96.36 on news test set, surpassing XLM-RoBERTa (87.0%) and Bangla Electra (56.3%).
• Pruned BnBERT-iPET achieves 76.93% F1 on POS tagging (combined corpora), outperforming Bangla Electra (74.7%) and DistilBERT-m (78.2%).

Threat model

The adversary is implicitly assumed to be resource constraints in terms of computation, memory, and dataset availability, rather than active adversarial attackers. The threat is inability to deploy or efficiently fine-tune large pretrained language models for Bengali due to hardware and data limitations. The model aims to mitigate this by pruning and few-shot learning to reduce resource requirements. No security or poisoning threat vectors are considered.

Methodology — deep read

The authors started with the goal to develop lightweight Bengali language models suitable for resource-constrained environments. Their threat model implicitly assumes computational resource limits preventing training or running large BERT models, rather than adversarial attacks on model security. The main challenge is model size and data scarcity.

They collected and constructed a diverse Bengali corpus (BanglaDDS) of about 80MB combining social media, newspaper, and classical 'Shadhu' text. Data was cleaned via OCR etc., and split 80% train / 20% test.

For modeling, they used ALBERT (a lightweight BERT variant) as a base with 4 transformer layers, hidden size 312, and 12 attention heads, with vocab size 30522. This base model, called BnBERT, was fine-tuned unsupervised on their dataset.

To enable few-shot learning, they applied iterative Pattern Exploiting Training (iPET), which reformulates inputs as cloze-style questions with masked tokens and task-descriptive patterns. Multiple pattern-verbalizer pairs (PVPs) were ensembles and iteratively refined on expanding labeled data with pseudo-labeling, per Schick and Schütze 2020. This trained BnBERT-iPET.

To reduce model size, they applied Lottery Ticket Hypothesis pruning: iteratively removing 10% of smallest-magnitude weights, rewinding remaining weights to their original initialization, and retraining for multiple iterations until 90% weights pruned, yielding BnBERT-iPET Pruned.

They trained the models on their corpus with standard MLM objectives, measured perplexity to evaluate language modeling quality, then fine-tuned on six downstream NLP tasks for Bengali: sentiment classification, emotion classification, authorship classification, news categorization, POS tagging, and punctuation restoration. These tasks involved multiple publicly available Bengali labeled datasets with train/dev/test splits, detailed in the paper.

Evaluation metrics included accuracy, precision, recall, and F1 for classification and token tasks. They benchmarked against several baselines including BanglaBERT, Bangla Electra, Indic-BERT, XLM-RoBERTa, and DistilBERT variants. They performed end-to-end fine-tuning of models on each task with held-out dev/test sets and reported standard metrics. Perplexity was computed on language modeling evaluation data.

The authors do not report adversarial or robustness testing, nor cross-lingual transfer. The pruning method requires rewinding to initial weights and retraining multiple times, which can be computationally intensive despite resulting in smaller inference models. Code or pretrained weights appear not publicly released. The proprietary BanglaDDS corpus is custom-collected and OCR-processed.

End-to-end example: Starting from ALBERT-base, fine-tune on BanglaDDS with MLM loss for BnBERT. Then apply iPET to convert task examples into cloze-style queries, iteratively train ensemble models, generate pseudo-labels, and refine model (BnBERT-iPET). Finally apply iterative lottery ticket pruning removing 10% weights at each iteration and rewinding, until 90% pruning, producing BnBERT-iPET Pruned which is fine-tuned on downstream tasks and evaluated.

Technical innovations

• Applying iterative Pattern Exploiting Training (iPET) in Bengali NLP few-shot learning to refine small models with cloze-style task reformulations.
• Combining iPET with Lottery Ticket Hypothesis pruning to obtain extremely sparse (90%) Bengali language models without significant performance loss.
• Development of a diversified unsupervised corpus (BanglaDDS) mixing social media, newspapers, and classical text to improve Bengali language modeling robustness.
• Demonstrating that a lightweight ALBERT-based model with only 4 layers and hidden size 312 can match or exceed larger models like Bangla Electra and Indic-BERT on Bengali downstream tasks.

Datasets

• BanglaDDS — 80 MB raw Bengali text corpus — custom collected from newspapers, social media, and classical text
• Emotion Classification — 2890 YouTube comments — public [37,38]
• Authorship Classification — ~18,000 writings from 14 authors — public[38,39]
• News Categorization — ~14,000 news articles — public [38,40]
• Sentiment Classification — combined datasets totaling ~6800 samples from YouTube, newspapers, Twitter, Facebook [37,41-44]
• POS Tagging — LDC Corpus (7,393 sentences), IITKGP POS Corpus, CRBLP POS Corpus — public [45-48]
• Punctuation Restoration — ~1.3M tokens train, test/dev sets from Bangla articles and ASR transcripts [49,50]

Baselines vs proposed

• Bangla Electra: emotion classification F1=36.3% vs BnBERT-iPET Pruned 48.1%
• Indic-BERT: news categorization accuracy=89.0% vs BnBERT-iPET Pruned 88.48%
• XLM-RoBERTa: authorship classification accuracy=93.8% vs BnBERT-iPET Pruned 97.67%
• BanglaBERT: emotion classification accuracy=71.05% vs BnBERT-iPET Pruned 50.3% (BanglaBERT outperforms in this task)
• XLM-RoBERTa: punctuation restoration F1=87.0% vs BnBERT-iPET Pruned 96.36%
• Bangla Electra: POS tagging F1=74.7% (LDC+IITKGP+CRBLP) vs BnBERT-iPET Pruned 76.93%
• DistilBERT-m: sentiment classification YouTube comments accuracy=70.0% vs BnBERT-iPET Pruned 64.52%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2608.05104.

Fig 1: Illustration of iterative pattern exploiting training and pruning of a

Fig 2: An illustration of the work process. The performance of each model

Fig 3: The detailed process of data collection and construction of a diverse

Fig 4: Train, test, and development data distributions for the emotion dataset.

Fig 5: Train, test, and development data distributions for the authorship

Fig 6: Train, test, and development data distributions for the news catego-

Fig 7: Train, test, and development data distributions for the YouTube

Fig 8: Train, test, and development data distributions for the BengFastText

Limitations

• The pruned model demonstrates higher perplexity (46.85) indicating some loss in language modeling quality after pruning.
• Performance on emotion classification is below BanglaBERT baseline, suggesting pruning and model size reduction may hinder some tasks.
• Dataset size is modest (~80 MB) and downstream labeled datasets are relatively small which may limit generalization.
• No evaluation reported on adversarial robustness, domain shift, or multilingual transfer scenarios.
• Lack of details on hyperparameter tuning and training compute resource requirements for pruning and iPET iterations.
• Code and pretrained weights are not publicly released limiting reproducibility and broader adoption.

Open questions / follow-ons

• Can the pruning strategy be combined with quantization or other compression methods to further reduce model size for Bengali NLP?
• How does BnBERT-iPET Pruned perform under domain shift or on other Bengali dialects and genres not covered in BanglaDDS?
• Would multilingual transfer learning or zero-shot cross-lingual adaptation improve performance for resource-constrained Bengali tasks?
• Can PET/iPET approaches be adapted to other sparse pruning techniques or structured pruning for hardware efficiency?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners interested in languages with limited NLP resources like Bengali, this work demonstrates the feasibility of creating competitive lightweight language models through iterative few-shot learning and lottery ticket pruning. These sparse, small-footprint models can significantly reduce computational overhead while maintaining solid performance on downstream NLP tasks relevant to text analysis and user interaction (e.g., sentiment detection, punctuation restoration).

Such models might be integrated into CAPTCHA systems or bot detection pipelines to process user input in Bengali with lower latency and resource demand on edge devices. Furthermore, the approach highlights the potential of leveraging task-specific cloze-style pattern training and massive pruning to deploy language technology in constrained environments—a key consideration for global bot-defense applications involving diverse low-resource languages. Practitioners should note, however, that the pruning method involves complex retraining cycles not trivial for continuous updates, and evaluation on robustness or adversarial inputs is still an open question.

Cite

@article{arxiv2608_05104,
  title={ BnBERT-iPET: Sparse Few-Shot Language Modeling for Bengali via Lottery Ticket Pruning },
  author={ Sajib Hossain and Md Kamrus Samad and Anan Ghosh and Labib Imam Chowdhury and Nabeel Mohammed },
  journal={arXiv preprint arXiv:2608.05104},
  year={ 2026 },
  url={https://arxiv.org/abs/2608.05104}
}

Read the full paper

• arXiv:2608.05104 (abstract)
• arXiv:2608.05104 (PDF)