FlexViT: A Flexible FPGA-based Accelerator for Edge Vision Transformers

Source: arXiv:2606.31938 · Published 2026-06-30 · By Hubert Dymarkowski, Xingjian Fu, Rappy Saha, Jude Haris, José Cano

TL;DR

This paper addresses the challenge of deploying Vision Transformer (ViT) models on resource-constrained edge devices, focusing on FPGA acceleration. Modern edge ViTs exhibit architectural heterogeneity—they combine fully connected (FC) and convolutional (CONV) layers with diverse tensor dimensions—making static accelerator designs inefficient. FlexViT proposes a flexible FPGA-based accelerator that uses a unified INT8 GEMM engine to handle both FC and CONV layers by performing an im2col transformation at runtime. A dual-mode dynamic dataflow switches between input-broadcast and weight-broadcast reuse depending on layer shape, optimizing utilization for varying matrix dimensions. A depth-first tiling approach completes accumulation in a single pass, eliminating off-chip partial sum transfers and reducing memory bandwidth demand.

Implemented on a low-cost PYNQ-Z2 FPGA, FlexViT achieves up to 2.74x speedup on accelerated layers and up to 1.40x end-to-end speedup compared to CPU-only execution across five representative ViT models spanning standard and hybrid architectures like ViT-T, DeiT-T, MobileViT-S, and EfficientViT-b1. The design consumes 83.9% BRAM, 71.8% DSPs, and 69.3% LUTs on the FPGA, indicating a resource-dense but balanced mapping. Energy measurements show modest net savings or slight overhead depending on the model. Overall, FlexViT demonstrates effective hardware-software co-design enabling flexible, efficient deployment of diverse ViT workloads on embedded FPGA platforms.

Key findings

• FlexViT achieves up to 2.74× speedup on accelerator-executed INT8 FC and CONV layers relative to CPU-only execution (Table V).
• End-to-end latency is improved by up to 1.40× across five representative ViT models, including standard and hybrid types (e.g., ViT-T and DeiT-T).
• FlexViT supports both standard ViT (FC-heavy) and hybrid ViT models (with balanced FC and CONV layers) via a unified GEMM engine using runtime im2col transformation.
• The dual-mode dataflow dynamically switches between Input-Broadcast and Weight-Broadcast modes per layer, improving hardware utilization across varied matrix dimensions (Algorithm 1).
• Single-pass depth-first tiling eliminates off-chip partial-sum transfers by completing accumulation entirely on-chip, reducing memory bandwidth requirements.
• On the PYNQ-Z2 FPGA, FlexViT uses 83.9% of available BRAM, 71.8% of DSPs, and 69.3% of LUTs, maximizing resource use within a constrained edge platform (Fig. 3).
• Energy per inference is comparable to or better than CPU baseline for FC-dominant models (e.g., DeiT-T: 1.09× energy improvement), with slight overheads in hybrid models due to simultaneous CPU-FPGA activity.
• Execution outputs on hardware match CPU baseline within 99% cosine similarity, preserving quantized INT8 inference accuracy.

Methodology — deep read

1. Threat Model & Assumptions: The paper’s threat model is implicitly the deployment of computationally intensive ViT neural networks on resource-limited edge devices with a goal of maximizing throughput and energy efficiency. The adversary is essentially the resource constraints and architectural heterogeneity in modern ViTs; no security adversary is modeled.

2. Data: The evaluation uses five fully quantized INT8 TFLite ViT models pretrained on ImageNet-21k or ImageNet-1k datasets: ViT-T, DeiT-T, Swin-T, MobileViT-S, and EfficientViT-b1. Table IV summarizes their parameter counts and input image sizes (e.g., ViT-T with 5.7M parameters at 224x224 resolution). No training was performed; the models were converted to INT8 TFLite format preserving accuracy.

3. Architecture / Algorithm: FlexViT implements a configurable FPGA accelerator comprising six hardware units: a scheduler, input/weight/bias read units, a high-throughput GEMM engine with 3 parallel cores, and a post-processing unit (PPU). The design treats FC and CONV layers uniformly by transforming CONV inputs into 2D matrices via a CPU runtime im2col operation, mapping both layer types to a unified GEMM (matrix multiply) operation:

• Scheduler dynamically configures scheduler modes: input-broadcast or weight-broadcast to optimize data reuse based on matrix dimension heuristics (Algorithm 1).
• GEMM engine employs an output-stationary dataflow completing accumulation in local registers, avoiding off-chip partial sums.
• PPE handles requantization and bias addition with per-tensor or per-channel quantization depending on layer type.
• The GEMM cores have tile sizes TN=64, TM=64, buffer depth TK=1024, and SIMD width Kf=16, with 3 parallel cores to maximize throughput within BRAM and DSP constraints.
• Arithmetic units split 8-bit dot-products between LUT-based logics and DSP slices to increase MAC throughput beyond DSP-only limits.

4. Training Regime: No training regime was applicable since models are pretrained externally and quantized. The focus was on inference acceleration and functional verification ensuring hardware output matched CPU results within 99% cosine similarity.

5. Evaluation Protocol: Latency and energy measurements were averaged over 100 runs on the PYNQ-Z2 board. CPU baseline used NEON SIMD instructions without FPGA running. Resource utilization reported in % BRAM, DSP, and LUT on FPGA. Five models span both FC-heavy and hybrid architectures to evaluate generality. Speedups were reported per accelerated layers and end-to-end inference. Non-offloaded operators including Softmax and LayerNorm were left on CPU to preserve FPGA resource efficiency. Energy was measured with a USB power meter. Correctness validated by output similarity.

6. Reproducibility: FlexViT’s codebase is publicly available at https://github.com/gicLAB/FlexViT. The accelerator is integrated with SECDA-TFLite toolkit enabling TFLite model offloading. The FPGA bitstream targeting PYNQ-Z2 and evaluation scripts are provided.

Concrete example: For a given MobileViT-S CONV layer, inputs are transformed on CPU to 2D im2col matrices. Scheduler chooses Weight-Broadcast mode due to tall-skinny matrix shape: weights are stored once and broadcast to cores while inputs partition among cores. Each GEMM core computes a 64x64 tile of output over 1024 depth with full accumulation performed locally. Output tiles are requantized and returned to host. This dynamic selection maximizes utilization and minimizes off-chip memory bandwidth.

Overall, the methodology combines software runtime adaptations with a flexible, reconfigurable hardware architecture leveraging FPGA-resource-aware design to achieve efficient, flexible ViT inference on edge devices.

Technical innovations

• Unified execution of both fully connected and convolution layers on a single INT8 GEMM engine via runtime im2col transformations.
• Dynamic dual-mode dataflow scheduler that switches between input-broadcast and weight-broadcast reuse strategies at runtime based on layer matrix shape to optimize utilization.
• Depth-first tiling approach that completes accumulation within on-chip buffers in a single pass, eliminating expensive off-chip partial-sum transfers.
• Hybrid arithmetic datapath splitting 8-bit operations between FPGA LUT logic and hardened DSP slices to increase effective MAC density and throughput on resource-constrained FPGA.

Datasets

• ViT-T — 5.7M parameters, ImageNet-21k pretrained
• DeiT-T — 5.7M parameters, ImageNet-1k pretrained
• Swin-T — 28M parameters, ImageNet-1k pretrained
• MobileViT-S — 5.6M parameters, ImageNet-1k pretrained
• EfficientViT-b1 — 9.1M parameters, ImageNet-1k pretrained

Baselines vs proposed

• CPU-only baseline: end-to-end latency normalized to 1.0 vs FlexViT: up to 1.40× speedup (ViT-T and DeiT-T)
• CPU-only baseline layer execution time normalized to 1.0 vs FlexViT accelerated layers: up to 2.74× speedup (DeiT-T)
• Energy per inference CPU baseline normalized to 1.0 vs FlexViT: between 0.93× (slight overhead, EfficientViT-b1) and 1.09× (energy reduction, DeiT-T)

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.31938.

Fig 1: Vision Transformer Architecture.

Fig 2: FlexViT architecture and computational flow.

Fig 3: FlexViT resource utilization on PYNQ-Z2 board.

Fig 4 (page 7).

Limitations

• The accelerator offloads only FC and CONV layers within a GEMM primitive, leaving other operations such as Softmax, LayerNorm, memory reshaping on the CPU, which limits potential end-to-end speedups especially for hybrid models.
• Evaluation is limited to INT8 quantized models pre-trained off-chip; no training or accuracy trade-offs were studied in joining FlexViT inference.
• Experiments use a single low-resource FPGA platform (PYNQ-Z2 with Zynq-7000), so generalization to larger FPGAs or other edge platforms is unclear.
• No adversarial or security evaluation is presented; robustness to adversarial inputs or fault injection is not addressed.
• Dynamic scheduler heuristics depend on runtime dimension padding and estimation which may incur overhead or suboptimal decisions in some cases; effects not deeply evaluated.
• Non-linear operations remain costly to offload, limiting improvements for ViTs with complex layer types beyond GEMM.

Open questions / follow-ons

• Can the accelerator architecture be extended to support more diverse operations (e.g., Softmax, LayerNorm) in hardware without exceeding resource budgets?
• How would FlexViT scale or need adaptation on more powerful FPGAs with more BRAM and DSP resources to handle larger ViT models or training workloads?
• What are the trade-offs and accuracy impacts of integrating higher precision or mixed-precision computations in the accelerator to support non-quantized models?
• Can the dynamic scheduling be enhanced with machine learning or runtime profiling to further improve utilization under varied workloads or model updates?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners focused on deploying robust and efficient vision models on edge or client devices, FlexViT provides a compelling template for FPGA-based acceleration of complex transformer architectures. The key insight of unifying convolutional and fully connected layers into a shared GEMM primitive via runtime transformations enables flexible hardware deployment across evolving model families. The dynamic dataflow scheduling approach addresses practical challenges due to heterogeneous layer compositions and varying tensor shapes encountered in modern vision architectures. The depth-first tiling strategy minimizing memory bandwidth and off-chip transfers is particularly relevant for constrained edge environments where latency and power consumption matter.

While CAPTCHAs often rely on fast, accurate image understanding or challenge generation models, efficient FPGA-based acceleration designed to handle complex vision transformers could improve responsiveness and enable client-side tasks in contexts with limited compute resources. However, the approach also highlights the limitations of partial hardware offloading when models contain diverse operators—practitioners should consider end-to-end profiling and balance hardware/software partitioning depending on the workload composition. Overall, FlexViT advances the state-of-the-art of flexible edge-accelerator design, which is directly applicable to specialized ML inference in bot detection and CAPTCHA generation endpoints.

Cite

@article{arxiv2606_31938,
  title={ FlexViT: A Flexible FPGA-based Accelerator for Edge Vision Transformers },
  author={ Hubert Dymarkowski and Xingjian Fu and Rappy Saha and Jude Haris and José Cano },
  journal={arXiv preprint arXiv:2606.31938},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.31938}
}

Read the full paper

• arXiv:2606.31938 (abstract)
• arXiv:2606.31938 (PDF)