Foreseeing the Invisible: Amodal Reconstruction of Leaf Fossil Images

Source: arXiv:2608.04423 · Published 2026-08-05 · By Liuxiang Yue, Ailin Zhang, Ziyue Zhao, Yikun Duan

TL;DR

This paper addresses the challenge of recovering the complete shape and venation pattern of fossil leaves, which are typically only partially visible due to occlusion by sedimentary rock. Unlike prior amodal segmentation work that relies on given visible masks, the authors propose AmodalDINO, a multi-head dense-prediction transformer-based model that predicts visible and complete leaf masks, as well as main and fine vein masks, directly from a single RGB image with no visible mask input. The model fully fine-tunes a DINOv3 ViT-L/16 backbone at a carefully chosen learning rate and adds auxiliary vein prediction heads to enforce structural priors related to leaf shape and venation.

Trained solely on a large synthetic dataset of rendered leaf fossils created from a small set of carefully labeled cleared-leaf images, AmodalDINO achieves 95.0% Dice and 90.5% IoU for amodal leaf shape reconstruction on synthetic validation data and transfers well to real fossil photographs qualitatively. When stripped to just the visible and amodal heads, the same model architecture also attains competitive amodal segmentation results on the KINS and COCOA-cls datasets without any visible mask input, notably outperforming some baselines on occluded regions. The authors demonstrate practical deployment in a fully offline, 4-bit quantized browser demo with ruler-based surface area calibration and generative leaf visualizations.

The key contribution is that by abandoning the standard assumption of visible mask input and by incorporating venation structure supervision, the model learns a strong shape prior enabling high-fidelity amodal completion of complex, organic fossil shapes with scarce labeled data, advancing amodal segmentation beyond common rigid-object domains.

Key findings

• AmodalDINO achieves 95.0% Dice and 90.5% IoU on synthetic leaf fossil validation for amodal leaf reconstruction.
• The visible leaf head obtains 98.5% Dice and 97.1% IoU, showing near-saturation on visible content.
• AmodalDINO predicts primary veins with 65.4% Dice and 48.6% IoU, and fine veins with 55.3% Dice and 38.2% IoU, reflecting the difficulty of thin structure recovery.
• Fully fine-tuning DINOv3 backbone at learning rate 10^-5 instead of freezing it improves boundary adherence, lobe recovery, and structural fidelity of amodal masks significantly (Fig. 5).
• Adding auxiliary vein prediction heads substantially improves structural shape priors and completion quality over amodal leaf head alone (Fig. 5).
• A two-head RGB-only variant (visible + amodal) reaches 85.05% full mIoU and 66.65% occluded mIoU on KINS and 80.90% / 38.15% on COCOA-cls without any visible-mask input.
• 4-bit quantization reduces model size from 1.27GB to 269MB and runs offline in the browser with negligible IoU drop (0.910 IoU compared to FP32).
• The model generalizes convincingly to nine real fossil photos, producing plausible completions and midrib recovery unseen during training.

Methodology — deep read

The paper frames the problem as amodal reconstruction of fossil leaf images obscured by stone, aiming to predict four mask outputs from a single RGB input: visible leaf, amodal complete leaf, amodal primary vein, and fine detail veins. A key departure from prior art is that the network receives no explicit visible mask input; instead, it predicts visible and amodal regions jointly from pixels alone, optionally conditioned on a coarse user-provided region-of-interest (ROI) hint treated as a weak suggestion.

The backbone is a DINOv3 ViT-L/16 (24 transformer blocks) pretrained self-supervised vision transformer. The authors widen the patch embedding convolution to ingest a 4th ROI channel (with neutral 0.5 encoding mapped to zero after normalization) so the model can exploit hints but does not rely on them. The backbone is fully fine-tuned at a very small learning rate (10^-5), while a lightweight DPT decoder and four independent convolutional heads predict each mask. This full fine-tuning is critical to enable shape adaptation; frozen backbones produce overly smooth, elliptical completions.

The loss function balances per-head binary cross-entropy weighted strongly on mask edges, soft Dice (for visible and detail veins), and soft Tversky loss with tuned false positive/negative penalties (for amodal leaf and vein heads) to encourage tight, precise completion without overgrowth into stone. Reweighting boosts loss emphasis in occluded (completion) regions. Containment constraints ensure visible masks lie within amodal masks, and vein masks lie within leaf masks.

Because no large real fossil dataset exists with dense labels, the authors synthesize 11,000 images from only 160 carefully labeled cleared-leaf specimens and apply randomized virtual stone occlusion and breakage masks in Blender with randomized lighting, pose, and texture to generate realistic synthetic training data with exact ground truth for all four masks. Data augmentations include flips, rotations, color jitter skewed toward darker tones, and ROI hint corruption.

Training uses AdamW with a two-rate schedule (10^-4 for decoder heads and DPT trunk, 10^-5 for backbone) over 40 epochs at batch size 64 on 448x448 inputs, taking ~2 hours on a NVIDIA RTX PRO 6000 GPU. The model checkpoint is selected by amodal Dice on held-out synthetic validation split.

For evaluation, standard semantic segmentation metrics (Dice, IoU, precision, recall) at threshold 0.5 measure the predicted masks. Public amodal benchmarks KINS and COCOA-cls are tested with a stripped two-head version and no visible mask input, evaluated on full and occluded mean IoUs. Nine real fossil photographs provide qualitative generalization validation.

Deployment includes quantizing weights to 4-bit for small offline browser inference using ONNX Runtime Web with WebGPU, achieving size reduction from 1.27GB to 269MB and near-original accuracy. A pipeline detects stone and ruler in photos to segment fossils, applies AmodalDINO, and estimates leaf surface area from ruler scale. Optional generative visualizations of living leaves augment output.

An example end-to-end flow: User provides fossil leaf photo with a ruler. Stone and ruler are detected and segmented. The stone region RGB is passed (with neutral ROI) to AmodalDINO. The model outputs visible, amodal leaf, and vein masks. Masks estimate occluded leaf shape and venation. Area estimate uses ruler calibration. An optional generative module produces visual living leaf renderings.

The core technical insight is that fully fine-tuning the self-supervised backbone and adding structural vein heads overcomes scarcity of labeled amodal examples and enables high-fidelity shape completion without visible-mask input or category templates.

Technical innovations

• Fully fine-tuning a DINOv3 ViT-L/16 backbone at a carefully tuned low learning rate (10^-5) for amodal segmentation instead of freezing and training only the decoder.
• Joint multi-head dense prediction of visible leaf, amodal complete leaf, primary vein, and fine veins from a single RGB image without any visible mask input.
• Auxiliary vein segmentation heads providing an implicit structural shape prior that regularizes amodal completion.
• A synthetic data generation pipeline leveraging a small, carefully labeled leaf specimen set composited with randomized virtual stone occlusions and damage to produce realistic amodal supervision.
• 4-bit quantized transformer model running entirely offline in browser via ONNX Runtime Web with near-original accuracy.

Datasets

• Synthetic Leaf Fossils — 11,000 images (10,000 train / 1,000 val) created from 160 cleared leaf specimens composited with Blender and post-processed damage — private
• KINS — 92,625 annotated instances, 50,894 occluded — public amodal segmentation benchmark
• COCOA-cls — 3,799 instances, 1,881 occluded — public amodal segmentation benchmark

Baselines vs proposed

• VRSP on KINS (predicted visible mask): full mIoU = 80.70%, occluded mIoU = 47.33% vs AmodalDINO (none) full mIoU=85.05%, occluded mIoU=66.65%
• AISFormer on KINS (predicted visible mask): full mIoU = 81.53%, occluded mIoU = 48.54% vs AmodalDINO (none) full mIoU=85.05%, occluded mIoU=66.65%
• C2F-Seg on KINS (predicted visible mask): full mIoU = 82.22%, occluded mIoU = 53.60% vs AmodalDINO (none) full mIoU=85.05%, occluded mIoU=66.65%
• GRASP on KINS (predicted visible mask): full mIoU = 82.37%, occluded mIoU = 55.24% vs AmodalDINO (none) full mIoU=85.05%, occluded mIoU=66.65%
• Amodal SAM on COCOA-cls (GT box + mask): full mIoU = 87.65%, occluded mIoU = 54.34% vs AmodalDINO (none) full mIoU=80.90%, occluded mIoU=38.15%

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2608.04423.

Fig 1: AmodalDINO on a real fossil specimen it never saw during training.

Fig 2: Top: the deployed system. A photograph of a specimen is segmented into stone and ruler, the stone cut-out is passed to AmodalDINO, and the four

Fig 3: Synthetic leaf fossil images used for training. Pose, scale, lighting,

Fig 4: Soft outputs of the four heads for one specimen (left to right: input

Fig 6: The two-head, RGB-only variant on occluded KINS instances. From

Fig 7: Generalization to real fossil specimens (photograph, predicted masks,

Fig 8: The offline browser demo, running locally in WebGPU on the 4-bit

Fig 9: Failure mode: two leaf fossils on one slab. The amodal head (pale

Limitations

• Vein prediction accuracy is limited (around 65% Dice for primary vein, 55% for fine veins) due to thin, low-contrast structures and confusion with rock cracks.
• Synthetic training data may not capture full variability or domain shift present in real fossil specimens, limiting generalization.
• The model assumes single-center cropped instances; off-center objects or multiple overlapping leaves cause failure modes.
• No ground-truth amodal labels exist for real fossils, so evaluation on real data is qualitative rather than quantitative.
• The model does not incorporate explicit biological or botanical shape priors beyond vein supervision.
• Dependency on a key hyperparameter tuning (two learning rates, full fine-tuning) may reduce robustness to training setup changes.

Open questions / follow-ons

• Can incorporation of explicit botanical or morphological priors improve vein and boundary recovery beyond indirect supervision via vein heads?
• How would the model perform on larger fossil datasets or other organic amodal segmentation tasks with more diverse shapes?
• Can one incorporate multi-view or 3D data to improve amodal completion of occluded leaves?
• What are the failure modes and robustness limits of fully fine-tuned transformers for amodal completion under heavy distribution shift?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, this work illustrates how challenging amodal segmentation can be successfully tackled without the usual aid of visible mask input or rigid object templates, by leveraging self-supervised vision transformers fully fine-tuned and guided by structural auxiliary tasks. Analogously, bot detection systems might benefit from joint multi-headed modeling of visible and inferred hidden behavioral patterns, rather than relying exclusively on explicit initial signals.

The synthetic-to-real transfer with minimal labeled real data underscores the potential of domain-randomized synthetic datasets paired with architecture and training design to enable robust inference on occluded or partially visible entities. The emphasis on structural priors enforced by auxiliary outputs parallels the idea of enforcing invariants or consistency constraints in CAPTCHAs to detect bots that attempt to distort or game the system where direct signals are missing or ambiguous.

While the domain differs significantly from common bot scenarios, the principle of ditching strong upstream segmentation assumptions and learning end-to-end amodal completion could inspire new CAPTCHA challenges that force agents to infer hidden states or completion tasks rather than simply recognizing visible objects. Finally, the successful deployment of a quantized large transformer in a resource-constrained, offline browser setting suggests room for richer client-side bot-defense tasks beyond simple pixel-level classification.

Cite

@article{arxiv2608_04423,
  title={ Foreseeing the Invisible: Amodal Reconstruction of Leaf Fossil Images },
  author={ Liuxiang Yue and Ailin Zhang and Ziyue Zhao and Yikun Duan },
  journal={arXiv preprint arXiv:2608.04423},
  year={ 2026 },
  url={https://arxiv.org/abs/2608.04423}
}

Read the full paper

• arXiv:2608.04423 (abstract)
• arXiv:2608.04423 (PDF)