L3Cube-MahaPOS: A Marathi Part-of-Speech Tagging Dataset and BERT Models

Source: arXiv:2606.24825 · Published 2026-06-23 · By Hariom Ingle, Ronit Ghode, Ishwari Gondkar, Jidnyasa Harad, Raviraj Joshi

TL;DR

This paper addresses the lack of large-scale, manually annotated resources for Marathi part-of-speech (POS) tagging by introducing L3Cube-MahaPOS, a corpus of 32,354 sentences from Marathi news text. The corpus is annotated with a 16-tag scheme aligned with Universal Dependencies, manually labeled by Marathi-proficient annotators following a detailed guideline with quality assurance. This dataset is an order of magnitude larger than prior Marathi POS resources and reflects contemporary formal Marathi usage with preprocessing tailored to Devanagari script and Marathi linguistic phenomena, including code-mixed English tokens.

The authors benchmark six diverse model families across classical (HMM, CRF), neural recurrent (BiLSTM, BiLSTM+CharCNN), multilingual transformer (MuRIL), and Marathi-specific transformer (MahaBERT-v2) approaches, finding the MahaBERT-v2 fine-tuned model achieves the best performance: 88.67% token-level accuracy and 81.67% macro-F1 across 15 evaluated tags. Detailed analysis of errors highlights challenges due to morphological richness, ambiguity between adjectives and adverbs, lack of capitalization for proper nouns, sparse minority classes like interjection and postposition. The paper releases dataset, annotation guidelines, and trained models to enable further Marathi NLP research.

Key findings

• L3Cube-MahaPOS dataset contains 32,354 manually annotated Marathi sentences from news domains with 472,459 total tokens.
• Annotation uses a 16-tag scheme aligned with Universal Dependencies, with POSTP tag excluded from final evaluation due to extremely low test frequency (3 tokens).
• MahaBERT-v2 fine-tuned on the dataset achieves 88.67% token-level accuracy and 81.67% macro-F1 over 15 tags on the test set.
• Classical methods: HMM achieves 72.1% accuracy and 63.1% macro-F1; CRF gets 80.6% accuracy and 72.8% macro-F1.
• Neural recurrent methods: BiLSTM-CRF achieves 84.3% accuracy and 76.6% macro-F1; BiLSTM+CharCNN improves to 85.7% accuracy and 78.3% macro-F1.
• MuRIL multilingual transformer baseline gets 86.9% accuracy and 79.7% macro-F1, while Marathi-specific MahaBERT-v2 leads.
• Per-class F1 analysis shows high-frequency tags like NOUN, VERB, PUNCT have F1 > 91%, while PROPN and INTJ lag at 59.52% and 44.5% respectively due to lack of orthographic cues and data scarcity.
• Confusion matrix reveals common systematic errors: PROPN→NOUN confusion (24%), ADJ→NOUN (12%), ADJ-ADV interchange (~18% of ADJ errors), and complete POSTP misclassification as ADP.

Methodology — deep read

1. Threat Model and Assumptions: The paper does not explicitly state an adversarial threat model since this is a dataset/model benchmark for POS tagging. The implicit assumption is the dataset reflects formal Marathi news text; the environment excludes severe domain shift or adversarial noise. The models are trained and evaluated on manually annotated gold-standard data without injected adversarial samples.

2. Data: The corpus is sourced from Marathi news portals covering politics, sports, culture, technology, and local news. HTML content is scraped, cleaned of boilerplate, duplicates removed via MinHash locality-sensitive hashing. Sentence boundary detection uses a rule-based system recognizing Marathi punctuation including the Devanagari danda. The final dataset contains 32,354 sentences split into training (22,652), validation (4,848), and test (4,854) sets stratified to maintain tag class and domain balance.

3. Annotation: Done manually by a team of Marathi-proficient annotators following a 16-tag Universal Dependencies-aligned scheme, extended with a Marathi-specific POSTP (postposition) tag. Annotators were trained on written guidelines inclusive of decision trees for complex Marathi structures and code-mixed tokens. Inter-annotator agreement was assessed on a subset; disagreements were discussed and guidelines refined iteratively. Sentences containing tokens labeled as 'X' (foreign/unclassifiable) were excluded entirely for label consistency.

4. Preprocessing Pipeline: Includes Unicode NFC normalization with Devanagari-specific canonicalization, a Devanagari-aware tokenizer that respects Marathi clitics (using an exception lexicon), noise filtering to remove/replace numerals, URLs, emoticons, and filtering out very short/long sentences. English words embedded in Marathi are treated as single tokens with appropriate tags.

5. Models: Six model configurations benchmarked:

   • Classical: Hidden Markov Model (HMM) with add-one smoothing; Conditional Random Field (CRF) with unigram, bigram surface forms and character suffix/prefix features, optimized by L-BFGS.
   • Neural recurrent: BiLSTM-CRF (two-layer BiLSTM over 100-dim learned embeddings with CRF decoding), and BiLSTM+CharCNN (adding character convolutional embeddings of multiple filter sizes concatenated to word embeddings).
   • Transformer-based: MuRIL (multilingual Indian languages model by Google) as cross-lingual baseline; MahaBERT-v2, a Marathi-specific BERT variant pre-trained on large Marathi corpus, fine-tuned on MahaPOS for token classification.

6. Training: Neural models run on NVIDIA Tesla T4 GPUs. MahaBERT-v2 fine-tuning hyperparameters were grid searched; best model used learning rate 5e-5, batch size 16, max sequence length 512, cosine decay with 10% warmup, weight decay 0.01, early stopping with patience 3, converging around epoch 3. Labels aligned to first subword tokens; subsequent subword tokens ignored in loss and metrics.

7. Evaluation: Metrics reported are token-level accuracy, weighted macro precision, recall, and unweighted macro-F1 averaged over 15 tag classes (POSTP excluded due to test scarcity). Evaluation splits stratified by domain and tag distribution. No cross-validation; held-out test is final evaluation. Baseline statistical, recurrent, and transformer methods compared. Confusion matrices and per-tag precision/recall/F1 computed for detailed error analysis.

8. Reproducibility: Dataset, annotation guidelines, and trained MahaBERT-v2 checkpoints are released openly on GitHub, enabling replication and further research. The Marathi BERT source is also available. No frozen weights or closed datasets are reported.

Example: A raw Marathi sentence from news is normalized, tokenized with special handling for clitics, then passed through MahaBERT-v2's WordPiece tokenizer. The 12-layer transformer encodes subword sequences; the first subword token representation of each word is fed to a linear classification head outputting logits over the 15 evaluated POS tags. Predicted tags are aligned back to original tokens for evaluation and error analysis.

Technical innovations

• Creation of a large-scale, manually annotated Marathi POS tagging dataset (32,354 sentences) with a UD-aligned 16-tag scheme specially extended for Marathi with POSTP tag.
• A Devanagari-aware preprocessing pipeline including Unicode normalization, tokenization sensitive to Marathi clitics and embedded English tokens, and noise filtering tailored for Marathi news text.
• Benchmarking a Marathi-specific transformer (MahaBERT-v2) pre-trained on Marathi text against classical and multilingual baselines, demonstrating clear gains in token-level accuracy and macro-F1.
• Introduction of an annotation protocol with iterative adjudication and amendment logs to improve consistency, including treatment of code-mixed tokens and low-frequency tags.

Datasets

• L3Cube-MahaPOS — 32,354 sentences, 472,459 tokens — manually annotated Marathi news corpus, publicly released via GitHub
• UD_Marathi-UFAL — ~466 sentences — publicly available Universal Dependencies Marathi treebank (used for comparison only)

Baselines vs proposed

• HMM: accuracy = 72.1%, macro-F1 = 63.1% vs MahaBERT-v2: accuracy = 88.67%, macro-F1 = 81.67%
• CRF: accuracy = 80.6%, macro-F1 = 72.8% vs MahaBERT-v2: accuracy = 88.67%, macro-F1 = 81.67%
• BiLSTM-CRF: accuracy = 84.3%, macro-F1 = 76.6% vs MahaBERT-v2: accuracy = 88.67%, macro-F1 = 81.67%
• BiLSTM+CharCNN: accuracy = 85.7%, macro-F1 = 78.3% vs MahaBERT-v2: accuracy = 88.67%, macro-F1 = 81.67%
• MuRIL: accuracy = 86.9%, macro-F1 = 79.7% vs MahaBERT-v2: accuracy = 88.67%, macro-F1 = 81.67%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.24825.

Fig 3: Row-normalised confusion matrix for L3Cube-MahaPOS-BERT on the test set (16 tags). Darker diagonal

Limitations

• Dataset is drawn solely from formal news text, limiting generalizability to informal or code-mixed domains like social media or conversational Marathi.
• The POSTP (postposition) tag is extremely underrepresented (23 total tokens), preventing reliable modeling or evaluation and excluded from core metrics.
• Class imbalance affects performance on low-frequency tags like INTJ (interjections) with limited training data and low recall.
• The models do not incorporate explicit morphological analyzers to address Marathi's rich morphology, potentially limiting disambiguation.
• Code-mixing is only partially addressed (English words annotated as single tokens), but mixed Hindi-Marathi or other borrowings are not explicitly modeled.
• Fine-tuning MahaBERT-v2 requires GPUs with >12GB memory and several minutes of training, challenging for resource-constrained deployment.

Open questions / follow-ons

• How can explicit morphological features or analyzers be integrated to reduce adjective-adverb ambiguity for Marathi POS tagging?
• Can targeted data collection or augmentation improve performance on underrepresented tags like POSTP and INTJ?
• What techniques can address the lack of capitalization and other orthographic signals in Marathi to improve proper noun recognition?
• How does the dataset and models generalize to informal or code-mixed domains such as social media, and what adaptations are needed?

Why it matters for bot defense

While this paper is not directly about bot-defense or CAPTCHAs, its contributions are relevant for designing NLP components that analyze Marathi text in human-computer interaction systems. Practitioners developing CAPTCHAs requiring natural language understanding in Marathi could leverage the L3Cube-MahaPOS dataset and fine-tuned MahaBERT-v2 model as reliable POS taggers, improving linguistic feature extraction for text-based challenges or assessing user generated content authenticity. The work highlights the importance of language-specific resources and preprocessing pipelines adapted to morphological richness and script details, which can inform language-aware CAPTCHAs or bot-detection systems handling Marathi input. The error analyses on ambiguous tags and proper nouns also reveal linguistic nuances important to modeling genuine human usage versus automated text.

Cite

@article{arxiv2606_24825,
  title={ L3Cube-MahaPOS: A Marathi Part-of-Speech Tagging Dataset and BERT Models },
  author={ Hariom Ingle and Ronit Ghode and Ishwari Gondkar and Jidnyasa Harad and Raviraj Joshi },
  journal={arXiv preprint arXiv:2606.24825},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.24825}
}

Read the full paper

• arXiv:2606.24825 (abstract)
• arXiv:2606.24825 (PDF)