Bridging Online and Offline Handwriting via Differentiable Physical Rendering

Source: arXiv:2608.03198 · Published 2026-08-04 · By Seonmi Park, Seunghyun Shin, Vihaan Misra, Dongmin Shin, Ukcheol Shin, Jean Oh et al.

TL;DR

This paper addresses the long-standing challenge in handwriting generation of unifying online trajectory-based and offline image-based methods into a single differentiable framework. Traditional online methods accurately capture the temporal dynamics and motion structure of handwriting but lack fine-grained texture and visual realism. Conversely, offline methods synthesize realistic images preserving stroke textures but discard stroke order and dynamics, limiting physical executability. Bridging these paradigms is difficult due to the absence of explicit physical models linking stroke kinematics to pixel-level appearance and the lack of paired online trajectory–offline image datasets. The authors propose a compact, differentiable physical brush rendering model parameterized by six core physical parameters that serve as an interpretable bridge from trajectories to realistic stroke images. This brush renderer enables end-to-end gradient backpropagation, allowing joint learning of motion and appearance.

The framework includes four modules: a text-to-stroke generator predicting pen trajectory conditioned on text and style images, a brush parameter observer extracting physical brush parameters from style references, a differentiable brush renderer converting stroke sequences and brush parameters into rendered handwriting images, and a zero-shot diffusion-based offline image enhancer for realistic refinement. They synthesize a large paired online-offline dataset by applying the differentiable renderer to online trajectory datasets augmented with diverse backgrounds, overcoming the scarcity of paired data. Extensive experiments on IAM-OnDB, CASIA, IAM, and CVL datasets show the approach outperforms state-of-the-art baselines on multiple metrics for both stroke generation and image realism. Real robotic calligraphy demonstrations validate the framework’s physical executability. Overall, the paper advances handwriting generation by tightly coupling motion and appearance via differentiable physics-inspired rendering and enabling cross-modal joint online–offline learning.

Key findings

• The differentiable brush renderer uses six physical parameters (w_base, k_spread, ρ_ink, σ_sharp, p_min, p_max) to link stroke kinematics to pixel-level appearance.
• Synthesizing paired online-offline data using the differentiable renderer on IAM-OnDB and CASIA yields 155,840 word-level samples with realistic styled images and corresponding trajectories.
• The proposed stroke generator achieves a Dynamic Time Warping (DTW) error of 0.2936 for multi-letter online handwriting, outperforming the SDT baseline with 0.8155 (Fig. 2/Table 2).
• Our renderer dramatically improves offline image quality with FID of 11.81 and BFID of 11.03 vs VATr++ baseline FID 74.28 and BFID 33.44 on synthetic paired data (Table 3).
• Attaching the renderer as a guidance module to diffusion models reduces FID from 29.00 to 25.89 on IAM Words and from 19.45 to 14.45 on CVL Words (Table 4), enhancing visual realism.
• The zero-shot diffusion image enhancer refines physical renderer output without retraining, maintaining stroke structure while improving offline image texture and artifacts.
• Synthesized images with varied backgrounds (notebook, cardboard, wood) improve robustness to real-world textures uncommon in prior offline-only datasets.
• Robotic calligraphy experiments demonstrate the physical executability of generated online trajectories and consistency of rendered images.

Methodology — deep read

The paper designs a unified online-offline handwriting generation framework based on a differentiable physical brush rendering model.

1. Threat Model & Assumptions: The adversary is not explicitly modeled since the goal is generation, not security; the approach assumes access to text strings, style images, and online trajectory datasets lacking paired offline images. The model must bridge unpaired domains with differentiable physics.

2. Data: The training pairs are constructed synthetically by applying the differentiable renderer to online handwriting datasets IAM-OnDB and CASIA-OLHWDB. Stroke trajectories with text labels are combined with randomly sampled brush parameters and composited over varied background textures to produce offline style reference images. The final synthetic dataset has 155,840 word samples from 303 writers for training and 78 for testing. Offline evaluation uses IAM and CVL test sets.

3. Architecture/Algorithm:

• Brush Parameterization: Six interpretable parameters govern stroke width base and spread (w_base, k_spread), ink opacity (ρ_ink), stroke edge softness (σ_sharp), and pressure proxy range (p_min, p_max).
• Differentiable Renderer R: Given stroke sequence S (2D pen coords + pen states) and brush params θ, computes per-pixel coverage maps using a differentiable kernel over distance to stroke segments, modulated by pressure proxy derived from inverse velocity. Max-alpha composition models overlapping strokes.
• Brush Parameter Observer O: Extracts brush parameters θ from style images using DINOv3 features aggregated via transformer layers with inter- and intra-image attentions, followed by MLP.
• Online Stroke Generator G: Transformer-based autoregressive model predicts 2D pen trajectories (K=20 Gaussian mixtures) and pen states from text and style images, extended from prior single-character models to word-level with cross-attention to entire text.
• Zero-shot Image Enhancer E: Off-the-shelf handwriting diffusion model (with VAE encoder/decoder) starting from noised structure latent z_rend to improve visual realism without retraining or modifying the diffusion model.

4. Training Regime: The generator G and observer O are trained jointly using synthetic paired data with losses for trajectory likelihood, pen state cross-entropy, brush parameter MSE, and renderer output MSE. Training details such as epochs and batch size are in supplementary material.

5. Evaluation Protocol: Quantitative metrics on online handwriting include DTW error comparing predicted strokes to ground truth. Offline images are evaluated with FID, BFID, HWD, CER, and LPIPS across IAM and CVL datasets. Baselines include SDT (online) and VATr++, DiffPen, One-DM, Emuru (offline). Ablations on noise injection start step for diffusion enhance evaluation robustness.

6. Reproducibility: Code and data release status is not explicitly stated; the paired dataset is synthetically constructed from publicly available IAM-OnDB and CASIA. The diffusion enhancer uses off-the-shelf pretrained models.

Concrete example: Given a text string and style images, O extracts brush parameters. G autoregressively predicts the sequence of pen positions and pen states encoding strokes. The renderer R, using the brush parameters and strokes, produces a differentiable grayscale image simulating brush ink deposition. This output is passed through zero-shot diffusion E to refine texture and noise realism to yield the final offline handwriting image. Training matches rendered images to synthetic ground truth and trajectories to original data, enabling end-to-end updates bridging trajectories and images.

Technical innovations

• A compact, differentiable physical brush model parameterized by six interpretable parameters enables gradient flow from pixel-level appearance to stroke kinematics.
• A differentiable brush rendering module maps sequential pen trajectories and physical brush parameters into stylized grayscale images with max-alpha composition.
• Synthetic paired online-offline handwriting datasets created by applying the renderer to existing trajectory datasets, augmented with varied real-world backgrounds for visual realism.
• A unified transformer-based online stroke generator extended from single-character to word-level with cross-attention for multi-character context and style conditioning.
• A zero-shot diffusion image enhancer uses pretrained handwriting diffusion models as structural prior refinement without retraining, preserving stroke fidelity.

Datasets

• IAM-OnDB — approx. 360k online handwritten words from 420 writers — public
• CASIA OLHWDB (v1.0–1.2) — large-scale Chinese online handwriting with stroke trajectories — public
• IAM Handwriting (test set) — 161 writers offline images — public
• CVL Handwriting (test set) — 283 writers offline images — public
• Synthetic Online-Offline Paired Dataset — 155,840 paired words from IAM-OnDB + CASIA + renderer — constructed by authors

Baselines vs proposed

• SDT (online handwriting): multi-letter DTW = 0.8155 vs Ours 0.2936 (Table 2)
• VATr++ (offline images): FID = 74.28 vs Our Renderer 11.81 on synthetic paired data (Table 3)
• DiffPen (offline images): HWD = 1.6275 vs DiffPen+Ours 2.3116 (tradeoff in metrics) (Table 4 IAM Words)
• One-DM: FID = 29.00 vs One-DM+Ours 25.89 (Table 4 IAM Words)
• One-DM: BFID = 16.35 vs One-DM+Ours 8.15 (Table 4 IAM Words)
• DiffPen: CER = 0.6672 vs DiffPen+Ours 0.7103 (Table 4 IAM Words)
• Our Renderer significantly reduces FID and BFID across IAM and CVL datasets demonstrating improved realism (Table 3,4)

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2608.03198.

Fig 1: A unified framework for online and offline handwriting generation.

Fig 2: Illustration of online

Fig 3 (page 1).

Fig 4 (page 1).

Fig 5 (page 1).

Fig 6 (page 1).

Fig 7 (page 1).

Fig 8 (page 1).

Limitations

• The physical brush model is a simplified surrogate omitting full 3D dynamics, wetness diffusion, and ink-paper physics to maintain differentiability.
• The pressure proxy is estimated from inverse stroke velocity, which may not capture true pen pressure variations in all handwriting styles.
• Synthetic paired dataset relies on randomized brush parameters and background textures, which may not fully capture real-world variability.
• The zero-shot diffusion enhancer depends on pretrained models and requires tuning noise injection parameters per dataset and backbone.
• Evaluation is limited to standard handwriting datasets with relatively short words; generalization to cursive, artistic, or other languages untested.
• Robotic calligraphy demonstrations are qualitative; no quantitative robotic execution error metrics or robustness to perturbations reported.

Open questions / follow-ons

• How to incorporate true physical pressure and 3D brush-tip dynamics into differentiable rendering for improved realism?
• Can the approach generalize to longer text lines, cursive handwriting, or languages with complex stroke orders beyond English and Chinese?
• How robust is the model to out-of-domain styles and unseen brushes or writing instruments in realistic deployment?
• How to quantitatively evaluate and optimize robotic execution accuracy of trajectories generated by the model?

Why it matters for bot defense

This paper’s framework advances handwriting generation by unifying the generation of executable stroke trajectories and realistic offline handwriting images through interpretable, differentiable physical modeling. For bot-defense and CAPTCHA practitioners, the ability to generate physically consistent handwritten trajectories alongside realistic images is valuable. It enables creating CAPTCHA challenges that are resistant to simplistic image-based forgeries, since the stroke ordering and dynamics can be validated or tied to physical writing constraints. Additionally, the interpretable brush parameters and differentiable rendering open avenues for synthetic generation and augmentation of handwriting data to improve anti-bot detectors. However, practical deployment would require further testing on robustness, adversarial resilience, and integration with existing CAPTCHA pipelines. The modular design combining learned style extraction, stroke sequence prediction, and image rendering is conceptually useful for designing handwriting-based challenge systems that require cross-modal consistency. Overall, the physical differentiable rendering bridges discrete online motion and continuous offline image features relevant for sophisticated handwriting verification and challenge generation.

Cite

@article{arxiv2608_03198,
  title={ Bridging Online and Offline Handwriting via Differentiable Physical Rendering },
  author={ Seonmi Park and Seunghyun Shin and Vihaan Misra and Dongmin Shin and Ukcheol Shin and Jean Oh and Hae-Gon Jeon },
  journal={arXiv preprint arXiv:2608.03198},
  year={ 2026 },
  url={https://arxiv.org/abs/2608.03198}
}

Read the full paper

• arXiv:2608.03198 (abstract)
• arXiv:2608.03198 (PDF)