Analyzing and Encoding the Al-Mawrid Arabic-English Dictionary with the ISO Language Markup Framework and TEI Lex-0

Source: arXiv:2606.18205 · Published 2026-06-16 · By Diaa Fayed, Laurent Romary

TL;DR

The paper addresses the significant challenge of converting the legacy print Al-Mawrid Arabic-English dictionary, a major bilingual lexicon with about 35,000 headwords, into a standardized, machine-readable resource. This transformation is crucial because Al-Mawrid had previously existed only in print or non-standard digital forms, limiting its usability for NLP and digital humanities. The authors propose a robust, reproducible methodology that aligns two complementary standards: the ISO Lexical Markup Framework (LMF) for conceptual lexical modeling and the TEI Lex-0 guidelines for XML serialization. Their approach adopts an editorial view on the dictionary structure, emphasizing a linear token stream rather than just layout, and systematically resolves print-induced structural ambiguities and implicit cues using punctuation and typographic signals. Using a representative 4.6% pilot sample (the letter Ayn), the study demonstrates an overall structural parsing accuracy of 91% and high precision/recall in automatically extracting lexical relations like synonyms (85% precision, 98% recall) and other semantic features. Beyond the encoding itself, the work critically compares Al-Mawrid with existing Arabic lexical resources, highlights the limitations of TEI Lex-0 in capturing complex Arabic phenomena, and proposes a scalable URI prefix system to support integration into Linguistic Linked Open Data (LLOD) frameworks. The result is a FAIR-compliant, interoperable, richly structured digital lexicon suitable for computational use and semantic web inclusion.

Key findings

• Structural parsing accuracy on a 4.6% representative sample (letter Ayn) is 91%.
• Synonym extraction achieves 85% precision and 98% recall using heuristic rules mapped to punctuation cues.
• Domain labels and derivations each reach 88% precision with 73% recall.
• Hyponym extraction shows lower recall (33%) due to implicit and 'open set' semantic relations not formally tagged in print.
• Approximately 13% of lexical relations involve cross-references, demanding explicit modeling.
• Less than 1% of entries have explicit part-of-speech tags, necessitating heuristic POS inference.
• Semantic and syntactic delimiters like colons, commas, semicolons, and brackets are systematically encoded as XML punctuation (<pc>) elements to resolve ambiguities.
• The dual-standard framework (LMF conceptual layer + TEI Lex-0 serialization) supports standardized, machine-actionable lexical entries with unique IDs and language tags.

Threat model

Not a security-focused paper; the primary challenge is the adversarial ambiguity and inconsistency inherent in the legacy printed dictionary's typographic cues and implicit linguistic metadata. The 'adversary' is the unstructured, print-based source format that obscures clear digital parsing and semantic extraction. The authors assume no intentional tampering or hostile interference in the data. The threat is limited to noise, inconsistency, and complexity in legacy typography and lexicography.

Methodology — deep read

1. Threat Model & Assumptions: The adversary in this context is not a malicious agent but the structural ambiguity and lexical inconsistency inherent in the legacy print resource. The study assumes the dictionary is available only as scanned print or non-standard digital forms devoid of explicit markup, with implicit linguistic cues conveyed via typographic and punctuation patterns. The objective is to produce a FAIR-compliant digital lexicon that resolves these ambiguities for computational NLP applications. No adversarial attacks or corruption are considered.

2. Data: The primary data source is the Al-Mawrid Arabic-English dictionary, which contains approximately 35,000 headwords and 60,000 subentries arranged alphabetically. The study works primarily on a representative lexicographic sample covering the letter Ayn (ع), accounting for 4.6% of the volume, to empirically estimate parsing accuracies and extraction performance. Data includes raw text and typographic features such as boldface, colons, commas, semicolons, and brackets indicative of semantic and syntactic boundaries. Preprocessing involves digitizing the print text, tokenizing into lexical units, and normalizing punctuation.

3. Architecture/Algorithm: The core methodology uses a dual-layer encoding framework. At the conceptual level, ISO Lexical Markup Framework (LMF, ISO 24613) provides an abstract model of lexical entries, senses, variants, morphological forms, translation equivalents, and semantic relations. Complementing this is TEI Lex-0, a community-driven XML serialization standard tailored for retro-digitized dictionaries. The mapping explicitly aligns LMF conceptual classes to TEI XML elements—for example, LMF LexicalEntry maps to <entry>, LMF Sense to <sense>, and translation equivalents to <cit type="translationEquivalent"> etc. Crucially, the authors adopt an editorial view treating the dictionary as a linear stream of tokens rather than a purely typographic layout, allowing them to parse structural elements systematically as sequences of punctuation-delimited fields. Heuristic rules based on punctuation roles (colons for definitions, commas for synonyms, semicolons for subsense boundaries, brackets for domain labels) guide the transformation. Cross-references and multiword expressions (MWEs) are modeled as nested or related <entry> elements with explicit reference URIs using <ref>. Given the scarcity of explicit POS tags (<1%), they infer morphosyntactic properties heuristically and represent these via <gramGrp> and <lbl> elements.

4. Training Regime: Since this is a rule-based transformation and encoding pipeline rather than a statistical model, there is no training in the ML sense. Instead, linguistic and typographic heuristics were developed based on empirical analysis of the Ayn sample, refined iteratively. Precision and recall metrics on lexical relations were computed on this sample to validate extraction rules.

5. Evaluation Protocol: The authors evaluate structural parsing accuracy (91% achieved) by manually inspecting whether the heuristic rules correctly segment and structure entries and senses. Information extraction of lexical relations—synonyms, domain labels, derivations, hyponyms, antonyms—is quantitatively assessed for precision and recall using manual gold standards. Error analyses identify major failure modes such as missing colons or inconsistent brackets. No adversarial or cross-lingual validation was discussed. The 4.6% Ayn sample serves as a statistically representative pilot, providing scientific weight for extrapolation to the full dictionary.

6. Reproducibility: The paper does not state explicit public code or dataset release, but extensive appendices containing XML encoding examples and detailed mapping tables are provided. The methodology is described to be reproducible for retro-digitization of similar legacy Arabic bilingual lexica. Given the proprietary status of Al-Mawrid, the full encoded dataset is likely closed-source, but the workflow is transparent.

An End-to-End Example: For the entry "ʿayn (عَيْن)", the print dictionary's bold face headword is identified as <entry> with xml:id and xml:lang="ar". Colons separate the headword from definitions. Synonyms within Arabic and English sections are delimited by commas and semicolons, each mapped to <sense> or <cit type="translationEquivalent"> elements. Punctuation marks are explicitly encoded as <pc>. Domain labels in brackets (e.g., [طب]) become <usg type="domain">. Cross-references "راجع" are encoded in <xr> blocks with <ref> links. Morphosyntactic cues, typically implicit, are manually or heuristically annotated within <gramGrp> elements. This transforms a visually complex print entry into a machine-actionable, standardized XML structure compatible with NLP pipelines and linked data frameworks.

Technical innovations

• Introduction of a dual-standard methodology aligning ISO Lexical Markup Framework (LMF) conceptual models with TEI Lex-0 XML serialization for Arabic bilingual lexica.
• Application of the editorial view treating legacy print dictionaries as linear token streams rather than typographic layouts, enabling systematic disambiguation of structural and semantic cues.
• Heuristic mapping of complex Arabic lexical and semantic phenomena, including implicit 'open set' hyponymy and sparse morphosyntactic cues, into explicit, queryable XML elements.
• Development of a scalable prefix-based URI referencing system to facilitate integration of digitized legacy lexica into Linguistic Linked Open Data (LLOD) environments.

Datasets

• Al-Mawrid Arabic-English Dictionary — ~35,000 headwords and ~60,000 subentries — legacy print; partial pilot sample (letter Ayn) representing 4.6% volume used for empirical analysis

Baselines vs proposed

• Structural parsing accuracy baseline: 91% on letter Ayn sample
• Synonym extraction: precision = 85%, recall = 98%
• Domain label extraction: precision = 88%, recall = 73%
• Derivation extraction: precision = 88%, recall = 73%
• Hyponym extraction: precision = 88%, recall = 33% (low recall due to implicit open-set relations)
• Antonym markers: precision = 88%, recall = 73%

Limitations

• Low recall for hierarchical relations like hyponyms due to implicit open set semantic patterns not formally tagged in source.
• Less than 1% of entries have explicit POS tags, requiring heuristic or manual annotation, limiting linguistic completeness.
• Parsing errors predominantly arise from misprints or inconsistencies in the original dictionary (e.g., missing colons, bracket mismatches), which require manual correction or cannot be fully resolved automatically.
• No adversarial testing or evaluation on distributional shift (e.g., other Arabic bilingual dictionaries) reported.
• Exact datasets and code availability not confirmed, limiting immediate reproducibility.
• TEI Lex-0 limits modeling of certain Arabic-specific linguistic phenomena such as scattered morphological cues or implicit semantic relations.

Open questions / follow-ons

• How can the methodology be extended to fully capture implicit 'open set' semantic relations, such as hyponyms, using automated NLP techniques?
• What is the impact of integrating formal Arabic morphological analyzers or POS taggers to reduce heuristic dependence on implicit cues?
• How generalizable is the dual-standard LMF/TEI Lex-0 pipeline across other Arabic legacy dictionaries or other bilingual lexicons with different typographic conventions?
• What are optimal strategies to link the encoded lexicon data into broader linked open data ecosystems, enabling richer semantic querying?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, the paper's methodology provides a valuable approach to transforming complex, legacy bilingual lexical data into standardized, interoperable digital lexical resources. Detailed lexical encoding that captures morpho-semantic relations, synonyms, antonyms, and translation equivalents can enhance natural language understanding modules used in bot detection systems, such as challenging CAPTCHA solvers or linguistic profiling of requests. Additionally, integration into linked data frameworks may support semantic consistency checks and vocabulary standardization used in language-based bot defenses. While the paper focuses on Arabic-English bilingual resources, the editorial view and dual-standard approach offer a replicable model for digitizing other legacy lexicons, improving NLP components integral to CAPTCHA and bot-detection pipelines.

Cite

@article{arxiv2606_18205,
  title={ Analyzing and Encoding the Al-Mawrid Arabic-English Dictionary with the ISO Language Markup Framework and TEI Lex-0 },
  author={ Diaa Fayed and Laurent Romary },
  journal={arXiv preprint arXiv:2606.18205},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.18205}
}

Read the full paper

• arXiv:2606.18205 (abstract)
• arXiv:2606.18205 (PDF)