Modeling Decisions in Blockchain Analytics: A Leakage-Aware Evaluation of Tree-Based vs. Sequential Models
Source: arXiv:2607.27350 · Published 2026-07-29 · By Michał Bartnicki, Jarosław A. Chudziak
TL;DR
This paper addresses the challenge of detecting Sybil bots on the Ethereum blockchain—automated accounts mimicking legitimate users to exploit airdrops or governance. Prior work heavily relied on complex deep learning sequence models treating blockchain transaction histories as if they were language sequences, but these models risk inflated performance due to label leakage from transactions involving high-signal smart contracts. The authors propose a leakage-aware evaluation protocol (Blind-Spot) that removes such shortcuts, and introduce a novel Transaction Grammar representation encoding transaction rhythm, structure, and intent derived from Ethereum Virtual Machine traces.
Using this framework, the paper rigorously compares sequential deep models (Transformers, BiLSTMs) against tree-based tabular models (XGBoost, linear SVM) for classifying Ethereum actors as organic users, MEV bots, or Sybil bots. Results on a large, leakage-filtered dataset show that organic users exhibit higher structural and timing entropy indicating more complex behavior, but surprisingly the XGBoost model outperforms all sequential models in accuracy (MCC 0.75 versus 0.66 for the best Transformer), while running orders of magnitude faster and consuming far less energy. Ablations reveal that transaction order adds little benefit to Transformer models which essentially act like high-dimensional density estimators. Thus, blockchain transaction data behaves more like a bag-of-grammar tokens than a meaningful sequence, making tabular models both more accurate and practical for real-time deployment.
Key findings
- The Blind-Spot protocol removes interactions with high-signal contracts (e.g. Uniswap Router, OpenSea) reducing dataset volume by 21.8%, mitigating label leakage.
- Organic users have significantly higher structure entropy (HG=3.50) compared to MEV bots (HG=3.12) and Sybil bots (HG=2.66) with p<0.01 (Kruskal-Wallis test).
- XGBoost achieves highest classification performance with MCC=0.7535, Macro-F1=0.8141 and accuracy=0.8745, outperforming Flattened Transformer (MCC=0.6602) and Hierarchical BiLSTM (MCC=0.6187).
- Flattened Transformer outperforms XGBoost only on recall for MEV (0.90 vs 0.79) and Sybil bots (0.59 vs 0.53) but recalls fewer Organic users (0.82 vs 0.93).
- Sequence ablation experiments on Hierarchical Transformer show that full scrambling of transaction order reduces MCC by only 0.0248, indicating minimal reliance on sequential dependencies.
- Grammar token order is more important than timing order, but overall local token frequency distributions dominate Transformer predictions.
- XGBoost is 100× faster in inference latency and has 30× lower estimated energy consumption compared to Transformer, highlighting efficiency advantage for real-time blockchain analytics.
- The findings validate prior tabular learning literature where tree ensembles outperform deep networks on structured, noise-prone datasets.
Threat model
The adversary consists of automated Ethereum accounts, including Sybil bots coordinating to farm airdrop rewards or manipulate governance, and MEV arbitrage bots executing profitable transactions. These adversaries may emulate legitimate user transaction behaviors to evade detection. They cannot bypass the removal of high-signal contract interactions imposed by the Blind-Spot protocol, nor have direct insider knowledge of model parameters or training data. The threat involves misleading classification to gain unfair economic advantage in decentralized systems.
Methodology — deep read
The paper's threat model assumes adversaries are Sybil bots and MEV bots acting to exploit airdrops and governance, attempting to mimic organic users. The key risk addressed is label leakage caused by high-signal smart contracts strongly correlated with actor class, which models can exploit as trivial shortcuts. To mitigate this, the authors define a Blind-Spot protocol which removes all interactions involving such contracts (e.g. Uniswap Router for MEV bots, OpenSea for Organic), reducing spurious correlations.
Data provenance includes three labeled actor classes: organic users from Hop Protocol's official investigations, Sybil bots from detected coordinated groups, and MEV arbitrage bots identified via Dune Analytics. The dataset consists of 15,813 Ethereum wallet transaction histories with at least 10 transactions each, split 80/20 into train and validation sets (12,650 / 3,163), with class imbalance handled by weighted loss. Internal Ethereum Virtual Machine (EVM) execution traces for each transaction are exploited to form features.
A novel Transaction Grammar is introduced to represent each transaction as a triplet of Rhythm (inter-arrival time discretized into 16 bins), Structure (9 discretized EVM trace properties concatenated as a composite token), and Intent (function selector merged into a vocabulary of 100 tokens). This grammar is consumed differently depending on model family.
Sequential models include a Flattened Transformer (4 layers, d=256, 4 heads) operating on flattened token lists with positional embeddings, and a Hierarchical BiLSTM (2 layers, 256 hidden units) which sums within-transaction tokens then passes transaction vectors recurrently capturing temporal dependencies. Both use weighted cross-entropy loss, trained for multiple epochs on NVIDIA A100.
Tree-based baselines include XGBoost trained on aggregated "Bag-of-Grammar" features (token counts, entropy, Lempel-Ziv compression metrics), and a linear SVM on TF-IDF and bigram counts. These models use global statistics abstracting away transaction order.
The evaluation protocol enforces identical data splits and random seeds for fair comparison, with metrics including Matthews Correlation Coefficient (MCC), macro-F1, and accuracy reported over five runs. Ablations randomly scramble timing and grammar order within transactions to assess dependency on sequence structure. Latency and energy use are estimated to evaluate deployment viability.
A concrete example follows: A wallet's transaction history is parsed into Transaction Grammar tokens capturing timing bins, execution depth and error flags, and function selectors; these tokens are aggregated into frequency histograms plus entropy and complexity metrics for XGBoost input, or passed as sequences for Transformer and BiLSTM; after training and evaluation using the Blind-Spot filtered dataset, the XGBoost model achieves best accuracy with minimal latency, demonstrating that granular sequential modeling is not necessary once leakage is controlled.
Technical innovations
- Design of a Blind-Spot protocol that systematically removes label leakage from high-signal smart contracts to enable leakage-aware evaluation of blockchain actor classification.
- Introduction of Transaction Grammar, a hierarchical representation of Ethereum transactions capturing rhythm, EVM execution structure, and intent tokens derived from internal execution traces.
- Comprehensive comparison of sequential neural models (Transformer, BiLSTM) versus aggregated tree-based models (XGBoost) on identical leakage-controlled data and features.
- Ablation methodology disentangling the effects of timing order and execution structure order on model performance, revealing minimal benefit from transaction sequencing.
- Integration of energy and latency benchmarking to evaluate the real-world deployment viability of blockchain behavioral models with respect to efficiency.
Datasets
- Blind-Spot Leakage-Controlled Ethereum Actor Dataset — 15,813 labeled wallets with structured transaction histories and internal EVM traces — labeled by Hop Protocol investigation list, Dune Analytics MEV classification
Baselines vs proposed
- XGBoost (tabular): MCC = 0.7535, Macro-F1 = 0.8141, Accuracy = 0.8745 vs Flattened Transformer: MCC = 0.6602, Macro-F1 = 0.7566, Accuracy = 0.8151
- XGBoost vs Hierarchical BiLSTM: MCC = 0.7535 vs 0.6187
- XGBoost vs Linear SVM: MCC = 0.7535 vs 0.6298
- Flattened Transformer recall on MEV bots: 0.90 vs XGBoost: 0.79
- Flattened Transformer recall on Sybil bots: 0.59 vs XGBoost: 0.53
- Transformer full sequence shuffle ablation MCC drop: −0.0248 from 0.5733
- XGBoost inference latency approximately 100× faster than Transformer with estimated energy use ~30× lower
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.27350.
Fig 2: Impact of sequence ablations on Transformer performance (MCC).
Fig 3: Efficiency vs. Accuracy. XGBoost (top left) achieves the most optimal trade-off
Limitations
- Dataset limited to Ethereum and labeled wallets available via Hop Protocol and Dune Analytics, which may bias findings to these actors.
- Leakage removal excludes high-frequency contracts but may not capture all subtle or future forms of label leakage.
- No evaluation against adaptive adversaries attempting to evade classification by manipulating entropy or timing patterns.
- Sequential models tested are standard Transformer and BiLSTM; more advanced architectures or self-supervised pretraining might improve results but were not explored.
- Energy estimates are approximate rather than measured from production-scale deployment.
- Only transaction order and timing were ablated; other behavioral features like graph interactions remain unexplored.
Open questions / follow-ons
- Can graph neural networks leveraging heterogeneous interaction graphs better capture actor behavior beyond transaction sequences and aggregate features?
- Will continuous, self-supervised embedding representations of transactions close the performance gap by capturing richer semantics than discrete tokenization?
- How robust are these models against adversarial adaptation where bots inject noise or camouflage behavioral entropy to evade classification?
- What are efficient sampling or pruning strategies enabling low-latency inference for graph-based or hierarchical models suitable for real-time deployment?
Why it matters for bot defense
This study highlights the importance of leakage-aware evaluation in blockchain actor classification relevant to bot-defense practitioners. It shows that naively applying deep sequence models to blockchain transaction histories can lead to misleadingly high performance by exploiting label leakage via high-signal contracts. Bot-defense engineers should emphasize rigorous leakage controls, verifying that models rely on intrinsic behavioral signals rather than trivial shortcuts.
The empirical evidence that simple tree-based models trained on aggregated statistics outperform complex sequence models also advises caution in model selection for real-time detection. Lower latency and energy costs facilitate scalable deployment in latency-sensitive monitoring scenarios common in CAPTCHA or bot-detection pipelines. Finally, the introduced Transaction Grammar offers a structured feature extraction methodology for Ethereum wallets potentially adaptable for enriched bot profiling beyond naive token sequences.
Cite
@article{arxiv2607_27350,
title={ Modeling Decisions in Blockchain Analytics: A Leakage-Aware Evaluation of Tree-Based vs. Sequential Models },
author={ Michał Bartnicki and Jarosław A. Chudziak },
journal={arXiv preprint arXiv:2607.27350},
year={ 2026 },
url={https://arxiv.org/abs/2607.27350}
}