An Agentic AI Framework to Accelerate Scientific Discovery in Plant Phenotyping

Source: arXiv:2606.31831 · Published 2026-06-30 · By Renan Souza, Daniel Rosendo, Kelsey Carter, John Lagergren, Frédéric Suter, Shelaine L. Curd et al.

TL;DR

This paper addresses a critical bottleneck in high-throughput plant phenotyping workflows: the gap between massive imaging data acquisition and slow, manual trait extraction and analysis. At Oak Ridge National Laboratory's Advanced Plant Phenotyping Laboratory (APPL), terabytes of multimodal images per experiment generate rich genotype-to-phenotype data, but current analysis remains expert-bound, post-hoc, and modality-specific. The authors present an end-to-end agentic AI framework that integrates natural-language conversational interfaces, HPC-scale Vision Transformer segmentation and trait extraction, and rigorous provenance tracking to transform APPL into an interactive, autonomous discovery platform. Their architecture treats analysis as an ongoing dialog where scientists query and refine hypotheses with AI co-scientists running on both cloud and the Frontier exascale supercomputer, shortening analysis from days or weeks to seconds for follow-ups.

Key innovations include a federated multi-agent architecture supporting asynchronous, secure communication across separate compute and user-facing domains; a ViT-based multi-modal image segmentation model trained specifically on APPL imagery and deployed at exascale for trait extraction across 8 imaging modalities; and a provenance-capturing multi-component system that logs all data, model, and reasoning steps to ensure scientific reproducibility. The framework was demonstrated on multi-week phenotyping campaigns of energy crops, enabling longitudinal, modality-agnostic trait computation and dynamic interaction with results via a chat interface. This work exemplifies how agentic AI can overcome human and computational bottlenecks in complex biological data analysis at scale.

Key findings

• The APPL facility can image up to 10,400 plants per multi-week campaign across 8 distinct imaging modalities, generating terabytes of data per experiment.
• Vision Transformer segmentation achieves robust plant-background masks across variable size, shape, and developmental stages, outperforming classical and prompt-based models which degrade on fine structures and modalities.
• Sliding-window ViT inference with Hann-window blending and graph-based pruning yields full-resolution, artifact-minimized plant segmentation masks suitable for downstream trait extraction.
• The extracted traits include morphological measures (height, projected leaf area, width, shape indices) and physiological proxies (Fv/Fm fluorescence ratio, leaf temperature) aggregated longitudinally per plant per imaging round.
• The conversational Co-Scientist Agent translates natural-language scientist queries into structured JSON analysis plans covering date range, modality, trait, and plant selection criteria.
• A Compute Agent dispatches these plans as HPC batch jobs on the Frontier supercomputer via Parsl, enabling scalable inference over terabyte-scale data in parallel.
• The secure token-authenticated streaming channel (S3M) connects user-facing and HPC zones, preserving provenance by logging experiment IDs, model versions, intermediate artifacts, LLM prompts, and agent decisions at every step.
• The end-to-end agentic loop turns what was a multi-day or multi-week offline analysis into an interactive session with follow-up queries answered within seconds reusing cached context and computed traits.

Threat model

The primary threat model considers a trustworthy scientific environment with authenticated users and institutional infrastructure; adversaries cannot arbitrarily tamper with data or agents inside the secured DOE facilities. The system enforces strict access control, token-based authenticated communication, and thorough provenance tracking to prevent unauthorized access, data corruption, or loss of reproducibility. The framework does not explicitly consider external attackers or data poisoning adversaries but prioritizes integrity and auditability within a federated multi-agent HPC and cloud environment.

Methodology — deep read

1. Threat Model & Assumptions: The system assumes an internal scientific environment at a DOE national laboratory where authenticated scientists query phenotyping experiments. Adversarial tampering or external attackers are not the primary threat, but the design enforces strict access controls, token-based secure streaming, and provenance logging to maintain integrity and reproducibility. Data provenance and auditability are first-class concerns to support trust in findings.

2. Data: The dataset consists of large-scale, multimodal plant imaging campaigns from APPL, covering species such as poplar, switchgrass, pennycress, and Arabidopsis over multiple experimental rounds. The facility produces terabytes of raw RGB, hyperspectral (VNIR, SWIR), thermal, fluorescence, multispectral, and 3D laser scans daily across up to 520 plants per imaging cycle. Each image is associated with rich metadata (experiment ID, plant ID, genotype, round ID, modality, treatment) and indexed in an AI-ready, versioned data lakehouse. Training annotation data was collected on APPL imagery to fine-tune a Vision Transformer segmentation model. Specific dataset sizes and splits for model training are not explicitly detailed, but involve multiple annotated modalities.

3. Architecture/Algorithm: The pipeline breaks down into four layers—Data (lakehouse, model artifact catalog), Compute (AI readiness preprocessing + ViT segmentation + trait extraction via Parsl on Frontier), Agent (Co-Scientist and Compute Agent multi-agent orchestration), and UI (Chainlit-based chat and report generation). The segmentation backbone is a vit_base Vision Transformer with 8-pixel patches and 448 pixel window size pretrained on ImageNet and fine-tuned on APPL annotations. The decoder is a lightweight convolutional neural network producing full-resolution binary plant masks. The ViT processes overlapping 448x448 tiles with 50% stride and blends per-tile probability maps using Hann weighting to suppress edge artifacts. A graph-based pruning algorithm removes isolated connected components to yield the largest plant cluster mask. Quantitative traits (height, area, shape indices, physiological proxies) are computed by aggregating pixel statistics over segmentation masks.

The Co-Scientist Agent uses an LLM guided by system prompts to conduct structured natural-language dialogs. It converts queries into JSON analysis plans specifying temporal windows, modalities, traits, and plant selection (by enumeration or ranked by trait values). These plans are sent over a secure streaming channel (S3M) to the Compute Agent, a headless HPC-resident agent orchestrating ViT inference jobs with Parsl across GPU nodes on Frontier. Upon job completion, results are sent back and visualized along with textual biological reports generated by the LLM.

4. Training Regime: Details on epoch counts, batch sizes, optimizer parameters, or training hardware for the segmentation ViT are not explicitly stated. The model was fine-tuned on domain-specific annotated APPL imagery after starting from ImageNet pretrained weights. The agentic framework components run in Docker containers on AWS for the user zone and on Frontier supercomputer for HPC workflows. Seed strategies and hyperparameters are not specified.

5. Evaluation Protocol: The paper articulates the workflow's functional correctness and integration rather than quantitative model benchmarks or cross-validation. Segmentation quality is qualitatively assessed by visualizations confirming performance across diverse modalities and growth stages. The system's key metric is interactive throughput: the ability to answer follow-up questions within seconds after initial trait extraction runs, vastly improving over the traditional days-to-weeks post-hoc analysis. No adversarial robustness testing or domain shift experiments are reported.

6. Reproducibility: The framework logs comprehensive provenance including experiment IDs, all model and LLM versions, intermediate outputs, and agent decisions to support auditability and reproducibility. OPAL uses an artifact catalog and FlowCept provenance system. Code availability and exact training datasets are not mentioned, limiting external reproducibility currently.

Example End-to-End: A scientist uses the Chainlit web interface to ask "Which poplar genotypes showed the greatest height gain over the last two weeks?" The Co-Scientist Agent parses this request, resolves date ranges and trait names, and produces a JSON plan. Over S3M, the plan is sent to the Compute Agent on Frontier, which triggers ViT segmentation on relevant image tiles for those plants and rounds using Parsl distributed GPU inference. The extracted traits are aggregated and returned. The Co-Scientist presents segmentation overlays, time-series plots, and a biological summary interpretation within minutes. Follow-ups narrowing or changing ranking criteria reuse cached traits for near-instant answers, closing the human-AI discovery loop.

Technical innovations

• A novel two-zone agentic framework integrating cloud-based conversational Co-Scientist Agents with HPC-resident Compute Agents connected by a secure, token-authenticated streaming channel (S3M), designed to respect security domains and federated resource management.
• Deployment of a Vision Transformer-based segmentation model at exascale on an HPC supercomputer (Frontier) for scalable, multi-modal plant image processing using a sliding-window inference and graph-based pruning pipeline.
• Integration of detailed end-to-end provenance capturing system (FlowCept) logging every data, model, and interaction artifact across federated agents and compute domains, ensuring reproducibility and auditability.
• A multi-agent orchestration middleware (Academy) tailored to heterogeneous scientific infrastructure enabling asynchronous, provenance-aware coordination between agents for autonomous scientific workflows.

Datasets

• APPL Multimodal Phenotyping Imagery — Terabytes per experiment — Oak Ridge National Laboratory (not publicly released)

Baselines vs proposed

• Classical thresholding and hand-crafted image segmentation: qualitative failure under varied plant size and modalities versus ViT segmentation with robust generalization across developmental stages and modalities
• Promptable models like Segment Anything Model (SAM): degrade on small, occluded or fine-edge structures and require accurate prompts, versus ViT fine-tuned on APPL imagery with better performance and self-contained inference
• Traditional manual or batch post-hoc trait extraction workflows: multi-day to weeks turnaround versus agentic framework enabling interactive follow-up queries answered in seconds

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.31831.

Fig 1: System architecture. Raw multimodal imagery flows left to right through AI-ready storage, on-demand ViT trait extraction on Frontier, and a results

Fig 2: Even as APPL’s automated imaging stations relieved

Fig 3: From a raw image to traits. A representative RGB plant image is

Fig 4: ViT-based segmentation: a high-resolution input is split into overlap-

Fig 5 (page 3).

Fig 5: End-to-end architecture of the APPL agentic framework. Two execution domains are bridged by S3M, a secure, token-authenticated streaming channel.

Fig 6: Co-Scientist agent Chainlit-based chat + dashboard user interface.

Fig 8 (page 6).

Limitations

• No quantitative segmentation accuracy metrics (e.g., IoU) or standardized benchmarks are reported, limiting evaluation of model generalization beyond APPL data.
• Training and evaluation are limited to existing APPL imagery and modalities; integration of additional traits requiring other measurement modalities (e.g., gas exchange, biomass) remains future work.
• The paper lacks adversarial robustness or stress tests under distribution shifts or novel plant species/modality combinations.
• Reproducibility is hindered by the lack of public release of code, trained weights, and full annotation data.
• The agentic AI currently depends on substantial HPC infrastructure (Frontier supercomputer), limiting transferability to lower-resource environments.
• Evaluations focus on system integration and qualitative benefits; formal user studies measuring impact on scientific productivity are not reported.

Open questions / follow-ons

• How well does the ViT segmentation model generalize to novel plant species, imaging conditions, or unseen modalities beyond those annotated in the training set?
• Can the framework integrate additional phenotypic traits that require non-imaging data or destructive measurements, and how to extend multi-modal fusion capabilities accordingly?
• What are the impacts of agentic AI workflows on biological discovery speed and accuracy in quantitative user studies with plant scientists?
• How can the architecture be adapted or optimized to support lower-scale HPC resources or edge deployments to broaden applicability?

Why it matters for bot defense

For bot-defense or CAPTCHA practitioners, this work offers a detailed example of a federated, multi-agent AI system operating across heterogeneous compute domains with strict provenance and secure communication channels. The design principles of separating user-facing conversational agents from compute-intensive backend agents, linked by token-authenticated secure streaming, illustrate architectures applicable where sensitive data or high-assurance workflows require compartmentalization and audit trails. Moreover, the demonstrated speedup from multi-day offline analyses to interactive second-scale queries reveals the potential gains obtained by orchestrating multi-agent AI systems atop high-performance resources. However, the biological domain-specific trait extraction and ViT segmentation techniques are less directly applicable to CAPTCHA contexts. Instead, the key lessons are around agent orchestration, secure kernel of trust implementations (S3M), and end-to-end provenance capture — all critical considerations when deploying AI systems guarding against automated abuse or needing explainable audit logs. Bot-defense engineers may consider these architectural patterns for building robust, federated AI models that maintain security and auditability while enabling real-time interactive systems.

Cite

@article{arxiv2606_31831,
  title={ An Agentic AI Framework to Accelerate Scientific Discovery in Plant Phenotyping },
  author={ Renan Souza and Daniel Rosendo and Kelsey Carter and John Lagergren and Frédéric Suter and Shelaine L. Curd and Gerald A. Tuskan and Rafael Ferreira da Silva and David Weston },
  journal={arXiv preprint arXiv:2606.31831},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.31831}
}

Read the full paper

• arXiv:2606.31831 (abstract)
• arXiv:2606.31831 (PDF)