MarkushGlyph and OCSRGlyph: Improved Chemical Structure Recognition

Source: arXiv:2607.28532 · Published 2026-07-30 · By Alex Andonian, Samuel G Rodriques, Andrew D White, Siddharth M Narayanan

TL;DR

This work addresses two related tasks of chemical structure recognition from images: optical chemical structure recognition (OCSR) of single molecules and Markush structure recognition, which involves parsing complex patent diagrams describing families of related molecules with variable substitution sites. While OCSR has mature prior art, Markush structure recognition remains challenging due to the need to jointly interpret molecular drawings with adjacent textual annotations and handle variable groups and constraints. The authors frame both tasks as image-to-text translation problems and propose two models: OCSRGlyph, a Transformer-based image-to-SMILES model that improves stereochemistry recognition through data oversampling; and MarkushGlyph, a vision-language model built on the pre-trained Qwen3.5-2B Transformer that reads patent images as single input and outputs a unified structured CXSMILES representation covering both ordinary molecules and Markush structures. They introduce a new stricter evaluation metric based on parsed-graph equality to better capture errors missed by prior Markush scoring approaches. Evaluation on standard benchmarks shows that OCSRGlyph achieves state-of-the-art exact-match accuracy of 93.8% on the USPTO OCSR dataset, outperforming strong baselines. MarkushGlyph sets new state-of-the-art results under both existing and proposed scores, outperforming prior multi-stage pipeline systems on IP5-M, M2S, and USPTO-Markush benchmarks. The combined training of MarkushGlyph on both ordinary and Markush structures allows robust, unified processing without task-type knowledge ahead of time.

Key findings

• OCSRGlyph achieves 93.8% canonical exact-match accuracy on the USPTO benchmark (5,719 images), surpassing MolSight (92.0%) by 1.8 points under strict stereochemistry consideration.
• Oversampling stereo-rich subsets in training boosts stereochemistry recognition substantially without degrading overall accuracy, raising exact-match accuracy from ~90% to 93.8%.
• MarkushGlyph outperforms prior Markush recognition methods (MarkushGrapher-2) on IP5-M (878 examples), M2S (103), and USPTO-Markush (74) benchmarks by 4-7 percentage points (e.g., 60.6% vs 53.7% on IP5-M under MarkushGrapher-2 score).
• MarkushGlyph benefits from majority voting over 8 samples to improve accuracy by 0.9 to 4.0 percentage points across benchmarks.
• The introduced strict parsed-graph equality metric more rigorously evaluates Markush predictions by requiring consistent atom mappings and penalizing extra or misplaced annotations, catching errors that prior metrics missed.
• Adding an ordinary molecule training subset (5% to 15%) to MarkushGlyph training boosts OCSR accuracy substantially with minimal impact on Markush performance (OCSR up to ~90% exact-match).
• MarkushGlyph uses early fusion of image and text via a vision-language model avoiding multi-stage pipelines, simplifying processing and improving accuracy.
• Standard large vision-language models like Claude Opus 4.8 and GPT-5.6-sol perform far worse (≤26% on IP5-M) than specialized MarkushGlyph.

Threat model

Not applicable. This work focuses on chemical structure image recognition and translation models, not on adversarial security or threat mitigation scenarios.

Methodology — deep read

The authors address two related tasks: OCSR (single molecule image to SMILES) and Markush structure recognition (image + text to structured CXSMILES capturing variable sites).

Threat Model & Assumptions: The adversary is not explicitly defined as this is a machine learning recognition task; the model must robustly parse variable drawing styles and printed text in patents without corruptions. No explicit adversarial attack evaluation is presented.

Data: OCSRGlyph trains on ~2.2 million molecule images combining PubChem-1M, USPTO-680K, and stereo-enriched subsets including 64,752 adjacent-ring stereo molecules oversampled twice, plus Stereo-200K. Targets are canonical SMILES via RDKit. MarkushGlyph trains on 263,158 Markush and molecule structure records from three sources: synthetic Markush structures (61% share), real structure-only patents (21%), and molecule-to-Markush real examples (18%). Ordinary molecules form 5% of MarkushGlyph data for multi-task learning.

Architecture:

• OCSRGlyph: Swin-B Transformer encoder (ImageNet-pretrained) encodes 384x384 image, feeding a 6-layer Transformer decoder trained from scratch to generate SMILES strings character by character (101-token vocabulary). Total ~94 million parameters.
• MarkushGlyph: Built on Qwen3.5-2B-Base (ViT vision encoder + Transformer decoder), jointly encodes image and short prompt text via early fusion, producing cxsmiles_opt structured records token by token. Fine-tuning uses Low-Rank Adaptation (LoRA) at rank 128 and scale 128 to specialize model efficiently.

Training regime: OCSRGlyph trained over 100,000 steps with fixed architecture and optimizer; stereochemistry enriched data oversampled. MarkushGlyph fine-tuned with LoRA on 263K records with mixed Markush and molecule data.

Evaluation:

• OCSRGlyph evaluated on USPTO OCSR benchmark (5,719 images) with canonical exact-match, chirality-kept, and graph-only metrics; molecule validity also assessed.
• MarkushGlyph evaluated on three Markush benchmarks: IP5-M, M2S, USPTO-Markush. Two metrics: MarkushGrapher-2 score (a recall-like partial match) and strict parsed-graph equality introduced by authors, requiring atom-to-atom consistent mapping, full annotation agreement, and penalizing hallucinations or misplaced labels.
• Ablations include varying share of ordinary molecules during MarkushGlyph training and testing majority voting decoding.

Reproducibility: Code and reproduction commands are publicly released. MarkushGlyph uses publicly available Qwen3.5-2B base. Benchmarks used are standard public datasets (IP5-M, M2S, USPTO-Markush, USPTO OCSR).

Concrete Example End-to-End: MarkushGlyph ingests a cropped patent chemical image containing Markush graphical elements plus printed substituent text embedded in image pixels; the Qwen vision encoder jointly processes this with a short prompt instructing extraction; the Transformer decoder outputs a token sequence encoding cxsmiles_opt with inline R-group labels, repeat unit annotations, and attachment points. This output is canonicalized and compared against ground truth using strict parsed-graph equality scoring to confirm exact structural equivalence including stereochemistry.

The single-model, early-fusion approach contrasts sharply with prior staged pipelines that separately process image, OCR text, and layout.

Technical innovations

• Formulating Markush structure recognition as a single-stage image-to-text translation using an early-fusion vision-language model, avoiding multi-stage pipelines.
• Introducing OCSRGlyph, a specialized Transformer encoder-decoder for OCSR with stereochemistry-focused training data oversampling to improve chiral center recognition.
• Developing MarkushGlyph, a unified model that jointly handles ordinary molecules and Markush structures via a common cxsmiles_opt output format.
• Designing a novel strict parsed-graph equality metric that rigorously evaluates Markush predictions by requiring consistent atom mappings and penalizing extraneous or misplaced annotations.

Datasets

• PubChem-1M — 1,000,000 molecules — synthetic, rendered with Indigo
• USPTO-680K — 680,220 molecules — pre-rendered patent molecules
• Stereo-200K — 200,000 molecules — stereo-enriched subset
• Adjacent-ring stereo subset — 64,752 molecules — extracted from PubChem-1M
• MarkushMixed — 263,158 records total — mix of synthetic and real Markush, structure-only subsets with 250,000 Markush plus 13,158 ordinary molecule records
• Benchmarks: USPTO OCSR (5,719 images), IP5-M (878), M2S (103), USPTO-Markush (74) — public standard evaluation sets

Baselines vs proposed

• DECIMER 2.7: canonical exact-match = 58.4% vs OCSRGlyph: 93.8%
• MolGrapher: canonical exact-match = 65.7% vs OCSRGlyph: 93.8%
• MolScribe: canonical exact-match = 88.4% vs OCSRGlyph: 93.8%
• MolSight: canonical exact-match = 92.0% vs OCSRGlyph: 93.8%
• MarkushGrapher-2: MarkushGrapher-2 score = 53.7% (IP5-M) vs MarkushGlyph greedy = 58.2%, MV@8 = 60.6%
• MarkushGrapher-2: strict parsed-graph equality = 51.1% (IP5-M) vs MarkushGlyph greedy = 52.1%, MV@8 = 54.7%
• MarkushGrapher-2: MarkushGrapher-2 score = 56.0% (M2S) vs MarkushGlyph greedy = 61.2%, MV@8 = 62.1%
• MarkushGrapher-2: strict parsed-graph equality = 51.5% (M2S) vs MarkushGlyph greedy = 61.2%, MV@8 = 64.1%
• MarkushGrapher-2: MarkushGrapher-2 score = 55.0% (USPTO-Markush) vs MarkushGlyph greedy = 59.5%, MV@8 = 63.5%
• MarkushGrapher-2: strict parsed-graph equality = 41.9% (USPTO-Markush) vs MarkushGlyph greedy = 54.1%, MV@8 = 56.8%
• Claude Opus 4.8 baseline: MarkushGrapher-2 score = 5.5% (IP5-M) vs MarkushGlyph: 58.2%
• GPT-5.6-sol baseline: MarkushGrapher-2 score = 26.3% (IP5-M) vs MarkushGlyph: 58.2%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.28532.

Fig 1: From a single molecule to a family of related compounds. Ordinary optical chemical structure recognition converts an

Fig 2 (page 2).

Fig 4: Markush qualitative comparison. Five illustrative examples from the Markush benchmarks. Checks and crosses

Fig 6: A comparison of the MarkushGrapher-2 and strict parsed-graph equality scoring functions. (a) An incorrect model

Fig 5: (a) A capability landscape. Markush ability (horizontal) is the MarkushGrapher-2 combined cxsmi_equality; OCSR

Limitations

• The strict parsed-graph equality metric does not evaluate substituent-definition text correctness, focusing only on structured CXSMILES chemical graph annotations.
• No explicit robustness or adversarial attack experiments were performed; evaluation focuses on standard, well-curated benchmark datasets without distribution shift scenarios.
• MarkushGlyph model is large (2B parameters) and computationally expensive compared to OCSRGlyph, possibly limiting deployment in constrained environments.
• Minor disagreement observed between reproduced MarkushGrapher-2 scores and published numbers on USPTO-Markush benchmark, indicating potential evaluation variability.
• MarkushGlyph was trained with up to 15% ordinary molecule data—performance behavior beyond this range and for out-of-distribution molecule types remains untested.
• The translation to CXSMILES simplifies representation but chemical canonicalization and serialization challenges remain, potentially limiting practical downstream use without further normalization.

Open questions / follow-ons

• How would the models perform under real-world noise and distortion in patent images or scanned documents?
• Can the strict parsed-graph equality scorer be extended to also validate substituent-defining text and conditional phrasing in Markush structures?
• What are the efficiency trade-offs for deploying MarkushGlyph in large-scale patent mining or real-time applications?
• Could joint learning with chemical text (e.g., patent claim text) improve variable-site constraint resolution beyond visual-only input?

Why it matters for bot defense

While this work does not directly address bot-defense or CAPTCHA, the methodology of rigorous image-to-text translation for complex scientific diagrams, alongside the design of strict evaluation metrics, is relevant for bots designed to parse and interpret intricate visual data embedded with contextual annotations. Bot-defense engineers aiming to detect or disrupt automated extraction of intellectual-property-sensitive drawing content (such as chemical patents) might draw insights from the multi-modal early-fusion approach and the stringent scoring mechanisms introduced here. Moreover, the challenge of precisely recognizing both structured images and accompanying text hints at potential future CAPTCHA designs leveraging multi-modal locked content that is difficult for naive OCR plus parsing pipelines to handle. The integration of vision-language models that fuse image and text inputs might also inspire defensive techniques against advanced automated solvers.

Cite

@article{arxiv2607_28532,
  title={ MarkushGlyph and OCSRGlyph: Improved Chemical Structure Recognition },
  author={ Alex Andonian and Samuel G Rodriques and Andrew D White and Siddharth M Narayanan },
  journal={arXiv preprint arXiv:2607.28532},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.28532}
}

Read the full paper

• arXiv:2607.28532 (abstract)
• arXiv:2607.28532 (PDF)