The Educational Proof Assistant Waterproof in an Introductory Proof Course: Proof Construction and Learning Processes

Source: arXiv:2606.26809 · Published 2026-06-25 · By Pim Otte, Rogier Bos, Johan Commelin, Jim Portegies

TL;DR

This paper investigates the use of Waterproof, an educational proof assistant using controlled natural language, in a first-year undergraduate introductory proof course at Utrecht University. The study was conducted as a quasi-experiment involving multiple instructors, students from various study programs, and a mixed Dutch-English language environment. Waterproof aims to support the transfer of formal proof-writing skills to pen-and-paper proofs. The key focus was on how Waterproof affects students' ability to construct mathematically valid proofs and their overall learning process.

Analysis combined in-class observation, surveys, student proof grading, and proof structure coding. Results suggest that students actively using Waterproof write proofs with greater explicitness — demonstrated by more occurrences of explicit logical steps such as existential introduction and case signposting. This increased explicitness also transferred to pen-and-paper proofs written in Dutch, despite Waterproof being English-based. While improvements in quiz grades were not statistically significant overall, students in the Mathematics-Computer Science program who used Waterproof scored noticeably higher. Impact on learning behaviors such as time spent or perceived feedback was inconclusive.

Since students self-selected whether to use Waterproof, the findings are suggestive rather than causal. However, the study provides evidence that an educational proof assistant designed with controlled natural language can encourage more structured and explicit proof writing and support skill transfer beyond the digital environment.

Key findings

• Active Waterproof users exhibit significantly more explicit proof steps: ∃-intro (2.31 vs 1.48 avg), ∨-elim (1.44 vs 0.70), and signpost-case (4.31 vs 2.78) compared to control (Table 4).
• Active Waterproof users produce fewer implicit signpost-concl steps (1.62 vs 2.20 control), indicating less reliance on summarizing intermediate conclusions.
• Evidence of transfer: Waterproof users write Dutch pen-and-paper proofs using Waterproof-inspired syntax and structures (Table 5 example).
• Mathematics-Computer Science students using Waterproof scored higher on the first quiz (mean 8.00) than inactive users (6.78) but no control group data was available (Table 2).
• Linear mixed model shows number of Waterproof exercises positively correlates with quiz grade (estimate 0.189) but effect is not statistically significant (p≈0.1) controlling for prior knowledge (Table 3).
• Survey data on perceived feedback shows active users feel they receive less useful feedback despite Waterproof providing many error messages (Figure 1).
• Observation suggests Waterproof users ask slightly more questions during class sessions (Figure 4), but absolute question numbers remain low.
• No clear differences in self-reported or observed time spent on coursework between Waterproof users and non-users.

Methodology — deep read

1. Threat model & assumptions: The study considers educational use of Waterproof by first-year university students in an introductory proof course. The adversary is not applicable; the experiment assumes students self-selected into groups, so possible confounding factors and selection bias exist.

2. Data: The dataset consists of proofs, quiz grades, surveys, and in-class observations collected from 199 enrolled students, of which 80 consented to research participation. Students came from varied study programs (mathematics, CS, physics, economics). Three groups studied: full control (no Waterproof), inactive Waterproof users (in Waterproof groups but did not engage), and active Waterproof users (completed ≥5 Waterproof exercises). Proofs analyzed included the first quiz proofs given after about two weeks of course.

3. Architecture/algorithm: Waterproof is a proof assistant based on the Rocq Prover accepting controlled natural language input in English. It enforces explicit proof structures such as case splits, witnesses for existential introduction, and signposting proof goals. Feedback is automated via error messages enforcing style.

4. Training regime: Students received an introduction to Waterproof in the first lesson. Waterproof exercises were available voluntarily mostly during first 3 weeks. Instructors demonstrated Waterproof proofs live. No formal training regime was imposed. Surveys were administered at start and end of course. Observations recorded student engagement every 5 minutes in tutorial sessions.

5. Evaluation protocol: Quantitative analysis combined linear mixed modeling of quiz grades, controlling for prior math knowledge using pretest and secondary school grades, with teacher and program as random effects. Proofs from the first quiz were coded line-by-line with logical step codes (e.g., ∃-intro, ∨-elim) to quantify explicitness. Surveys measured time-on-task, feedback perception, and help-seeking. Observations tracked in-class time allocation and questions asked. Analysis focused on differences between Waterproof engagement groups.

6. Reproducibility: Some research material, proof exercises, and coded datasets are publicly available via GitHub. The full data is partially closed due to privacy. No pretrained models as such; the study focuses on pedagogical effect rather than ML model reproducibility.

Concrete example: A student’s proof in the first quiz was coded, showing explicit case signposting and witness selection as required by Waterproof's controlled language syntax. This was quantitatively compared across groups, revealing that active Waterproof users exhibited on average 2.31 ∃-intro steps vs ~1.48 in controls, illustrating increased explicitness fostered by the tool.

Overall, the methodology integrates qualitative classroom observation and quantitative mixed modeling of student outcomes, proof structure, and survey self-reports to assess impacts of an educational proof assistant in a realistic, multi-instructor teaching environment.

Technical innovations

• Applying a controlled natural language input proof assistant (Waterproof) in a mixed-language (English tool, Dutch proofs) undergraduate introductory proof course.
• Analysis of proof structure by coding explicit logical steps line-by-line to measure explicitness and structure induced by the assistant.
• A quasi-experimental design separating active users, inactive users, and controls within multiple concurrent classes to control for teacher and program effects.
• Demonstration of cross-language skill transfer from English proof assistant use to constructing Dutch-written paper proofs.

Datasets

• Course participant data — 199 enrolled, 80 survey participants, proofs and quiz grades from Utrecht University Proof course
• Coded proofs from first in-class quiz with detailed line-by-line logical step annotations

Baselines vs proposed

• Control group average first quiz grade (Mathematics only): 5.96 vs Active Waterproof users: 6.12 (non-significant)
• Mathematics-Computer Science inactive Waterproof users: 6.78 vs active Waterproof users: 8.00 (no control group for this subgroup)
• Linear mixed model: Number of Waterproof exercises estimate on grade = +0.189 vs p=0.1 (non-significant)
• Control group ∃-intro steps average = 1.48 vs Active Waterproof users = 2.31
• Control group ∨-elim average = 0.70 vs Active Waterproof users = 1.44
• Control group signpost-case average = 2.78 vs Active Waterproof users = 4.31

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.26809.

Fig 1: Survey results on perceptions about feedback

Fig 2: presents the self-reported time spent on the course, aggregated by Wateproof use.

Fig 3: Observations on time spent in class. The vertical axis shows the proportion of observed in-

Fig 4: Observations on number of questions asked by students

Fig 5: Screenshot showing a Waterproof proof and example of feedback.

Limitations

• Quasi-experiment with self-selection: students chose whether to use Waterproof, introducing potential selection bias.
• Effect of teachers confounded: teachers using Waterproof were also researchers experienced with proof assistants, hard to separate instructor effect from tool effect.
• No random assignment or fully matched control group, limiting causal claims.
• Limited sample size, especially in active Waterproof subgroup and subgroups of study programs.
• No long-term follow-up to assess persistence of skill transfer beyond the course.
• Feedback perception results inconclusive, possibly due to students interpreting automated errors differently from pedagogical feedback.
• Language mismatch (English-based tool vs Dutch paper proofs) might limit generalizability to other language settings.

Open questions / follow-ons

• How to improve the design and framing of feedback in educational proof assistants so students perceive it as helpful pedagogical feedback rather than error messages?
• What are the long-term impacts of early use of proof assistants like Waterproof on mathematical proof competency in later courses?
• How can selection bias in voluntary uptake be mitigated or controlled in future studies to strengthen causal claims?
• Could Waterproof or similar systems be adapted to support multiple languages more natively to reduce language transfer overhead?

Why it matters for bot defense

While this paper focuses on educational proof assistants rather than bot-defense or CAPTCHAs, some insights are relevant for human-computer interaction in security challenges. The study highlights the value of controlled natural language interfaces in scaffolding complex logical reasoning skill transfer across media—analogous to designing CAPTCHA tasks that leverage humans’ reasoning strengths. The mixed language environment results indicate that language and interface constraints can affect usability and skill transfer, a consideration for CAPTCHA design targeting multilingual populations. Also, the role of explicit stepwise structuring required by Waterproof suggests that forcing rigorous explicit reasoning in challenge interfaces could increase difficulty for automated solvers while remaining accessible to humans. Finally, the inconclusive and sometimes counter-intuitive perception of automated feedback from the system underscores the importance of clear, user-friendly feedback in human-facing security tools to maintain engagement. In sum, the principles of explicit, controlled natural language input and carefully designed feedback loops from this educational setting may inform the design of more robust, user-aligned bot-defense puzzles, especially those requiring logical or stepwise problem solving.

Cite

@article{arxiv2606_26809,
  title={ The Educational Proof Assistant Waterproof in an Introductory Proof Course: Proof Construction and Learning Processes },
  author={ Pim Otte and Rogier Bos and Johan Commelin and Jim Portegies },
  journal={arXiv preprint arXiv:2606.26809},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.26809}
}

Read the full paper

• arXiv:2606.26809 (abstract)
• arXiv:2606.26809 (PDF)