STeMP: Spatio-Temporal Modelling Protocol

Source: arXiv:2607.20592 · Published 2026-07-22 · By Jan Linnenbrink, Jakub Nowosad, Marvin Ludwig, Anna Frederike Jablotschkin, Fabian Schumacher, Teja Kattenborn et al.

TL;DR

Spatio-temporal machine-learning models are crucial in environmental sciences but highly sensitive to training data characteristics, methodology choices, and cross-validation strategies. These sensitivities can lead to misleading model quality estimates and hinder trust and applicability assessments. This paper introduces STeMP (Spatio-Temporal Modelling Protocol) to address the lack of standardized reporting and guidance specifically tailored for spatio-temporal ML models. STeMP structures reporting into three sections—Overview, Model, and Prediction—to systematically capture metadata, modeling decisions, evaluation choices, and prediction details. Accompanied by an R-package and a user-friendly web application, STeMP supports semi-automated protocol generation from modeling objects, raising warnings for common pitfalls such as inappropriate data splitting or lack of uncertainty quantification. The goal is to improve transparency, reproducibility, and critical assessment of spatio-temporal models in environmental applications. An illustrative case predicting plant species richness in South America demonstrates that STeMP can reveal evaluation biases and help researchers and users ask pertinent questions about model applicability and limitations.

Key findings

• STeMP formalizes spatio-temporal machine-learning reporting into three main protocol sections: Overview, Model, and Prediction, covering crucial metadata, modeling steps, and prediction contexts.
• The accompanying R-package supports automated extraction of protocol fields from R model objects (caret, tidymodels, mlr3) and spatial datasets, streamlining documentation.
• STeMP’s web application detects and warns about common modeling issues, e.g., random cross-validation on clustered training data can yield overoptimistic accuracy estimates.
• In the example task modeling South American plant species richness, the model reported an R2 of 0.7 but random CV on clustered samples led to likely overoptimistic performance, flagged by STeMP.
• The protocol captures details on sampling design, predictor resolution, evaluation strategy, uncertainty quantification, and predictor selection to elucidate model reliability and usability.
• STeMP extends and specializes prior protocols like ODMAP and ODD to address spatio-temporal autocorrelation challenges unique to geoscience ML.
• The protocol and software are open source, versioned (v2026.07.00), and encourage community contributions for maintenance and extension.
• Warnings raised during protocol filling can be summarized and exported to aid peer reviewers in assessing spatio-temporal ML studies efficiently.

Methodology — deep read

1. Threat model & assumptions: The paper does not model adversaries or security threats; rather, it assumes model developers, reviewers, and users who seek transparent and reliable model reporting in spatio-temporal environmental ML tasks.

2. Data provenance and size: The authors illustrate STeMP using plant species richness data from the sPlotOpen database and spatial predictors like elevation (Jarvis et al., 2008) and WorldClim climate variables. The protocol itself is agnostic to dataset size or domain, focusing on structured metadata capture.

3. Architecture/algorithm: STeMP is a reporting protocol, not a model architecture. The accompanying software includes an R-package with a Shiny web application structured modularly into sections aligned to the protocol: Overview, Model, and Prediction.

• The Model section further breaks down into data characteristics, learning method, predictor details, model evaluation and selection, explainability, uncertainty, bias, limitations, and software documentation.

• The Prediction section optionally covers prediction extent, map evaluation, uncertainty quantification, and post-processing.

4. Training regime: N/A for STeMP itself; the example random forest model used training data from sPlotOpen combined with spatial predictors.

5. Evaluation protocol: STeMP incorporates reporting of the model evaluation strategy used (e.g., spatial or random cross-validation), performance metrics, results, hyperparameter tuning, and predictor selection approaches. Warnings are raised if the evaluation is mismatched with the sampling design (e.g., random CV on clustered data).

6. Reproducibility: The protocol and software are open source, available on GitHub under GPL, with a Zenodo-archived version (v2026.07.00). It supports export of completed protocols in CSV and PDF, with future plans for JSON export and Zenodo integration for DOIs. Currently automated extraction supports R model objects only.

Example workflow:

• Model developers upload trained model objects and spatial data (training locations and prediction area) into the web application.
• The app auto-fills many protocol fields (sample size, evaluation metrics, hyperparameters, CRS, sampling pattern) and generates maps.
• The app assesses inputs for common pitfalls and raises warnings.
• Completed protocols can be downloaded and analyzed with provided R functions to produce concise summaries of warnings.

The protocol helps answer key questions about model applicability, trustworthiness of reported metrics, and spatial/temporal generalization capacity by making these metadata explicit and machine-checkable.

Technical innovations

• Design of a domain-specific standardized protocol (STeMP) for transparent reporting of spatio-temporal ML models addressing unique challenges like spatial autocorrelation and data splitting pitfalls.
• Integration of STeMP with a modular, extensible R-package and Shiny web application enabling semi-automated protocol generation from common R ML model objects and spatial datasets.
• Automated detection and warning system for common modeling issues such as evaluation strategies misaligned with sampling designs (e.g., random CV on clustered samples).
• Provision of a machine-readable and human-interpretable capture of key metadata fields critical for assessing model generalization, uncertainty, and limitations in spatio-temporal environmental predictions.

Datasets

• sPlotOpen database — plant species richness plot data — public environmental vegetation database
• Elevation data (Jarvis et al., 2008) — global spatial predictor layer
• WorldClim climate data (Fick and Hijmans, 2017) — global climate predictor dataset

Baselines vs proposed

• N/A — The paper focuses on protocol development and reporting standard rather than model performance comparison.

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.20592.

Fig 2: The prediction resulting from the example modelling task.

Fig 3: An overview of selected parts of the web application’s user interface filled based on the example (top). The

Fig 3 (page 7).

Fig 4 (page 7).

Fig 5 (page 7).

Fig 6 (page 7).

Fig 7 (page 7).

Fig 8 (page 7).

Limitations

• Currently, automated protocol extraction supports only R model objects using specific packages (caret, tidymodels, mlr3); no support yet for Python or ONNX models.
• The protocol primarily focuses on spatial aspects; temporal modeling aspects receive less comprehensive coverage and require further extension.
• The automated warning system identifies common pitfalls but does not enforce corrections, relying on user interpretation and justification.
• The protocol and software do not yet exploit large language models or automated extraction from unstructured documents.
• Export formats are limited to CSV and PDF currently; a more flexible JSON export is planned but not yet available.
• The example application illustrates utility but does not include extensive validation or user studies on protocol adoption or impact.

Open questions / follow-ons

• How to extend STeMP to fully address temporal dependencies and align temporal cross-validation strategies with spatio-temporal data structure.
• How to broaden automated extraction support to models and workflows beyond R, including Python-based ML frameworks and interoperable formats like ONNX.
• How to integrate natural language processing or large language models for semi-automated protocol completion from published manuscripts or documentation.
• How to foster widespread community adoption and iteratively refine STeMP through open contributions and real-world case studies.

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, STeMP offers a compelling example of how standardizing transparent reporting can mitigate subtle pitfalls caused by data dependencies and evaluation biases—issues analogous to challenges in behavioral or challenge-response modeling used in bot detection. Applying a protocol-driven approach to model reporting can improve reproducibility, interpretability, and trust in spatio-temporal or dynamic behavioral classifiers relevant in CAPTCHA system evaluation. Furthermore, the implementation as an automated, user-friendly tool shows how embedding domain-specific quality checks and warnings during model development can prevent overoptimistic evaluation and undetected data leakage, lessons important for robust bot detection models which also often deal with temporal data and spatial network patterns. While STeMP’s direct focus is environmental modeling, its principles around transparency, metadata capture, evaluation strategy alignment, and uncertainty quantification transfer well to captcha anti-bot modeling pipelines under complex spatio-temporal adversarial data conditions.

Cite

@article{arxiv2607_20592,
  title={ STeMP: Spatio-Temporal Modelling Protocol },
  author={ Jan Linnenbrink and Jakub Nowosad and Marvin Ludwig and Anna Frederike Jablotschkin and Fabian Schumacher and Teja Kattenborn and Hanna Meyer },
  journal={arXiv preprint arXiv:2607.20592},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.20592}
}

Read the full paper

• arXiv:2607.20592 (abstract)
• arXiv:2607.20592 (PDF)