Decomposing the Doer Effect in Programming Practice: Code Writing Stands Out Among Active Practice
Source: arXiv:2608.02541 · Published 2026-08-03 · By Arun Balajiee Lekshmi Narayanan, Gillian Gold, Jordan Barria-Pineda, Quinn K Wolter, Peter Brusilovsky, Paulo Carvalho
TL;DR
This paper investigates the "doer effect"—the phenomenon that active practice leads to stronger learning outcomes than passive learning—in the context of computer science education, specifically programming practice. Prior work categorized active practice as a single homogeneous category; this study decomposes active practice into distinct types to discover if some forms are more effective than others. The authors analyze log data from 334 students over 11 semesters in introductory and intermediate Java courses using the Mastery Grids platform, which integrates five types of practice activities: Code Writing, Code Tracing, Code Completion (all active), and Code Visualizations and Code Explanations (both passive, yet interactive). The study finds that active practice is associated with roughly 3.2 times greater learning gains compared to passive activities even though the passive content is itself interactive. Among active practice types, only code writing shows a statistically significant positive association with posttest performance after controlling for other forms, while code tracing and code completion do not show such associations. This suggests code writing’s combination of generative solution construction plus timely corrective feedback drives its superior effectiveness. The authors interpret these findings to mean that the doer effect extends beyond conventional passive content to interactive passive content and is nuanced by the specific cognitive demands of active practice types.
Key findings
- Active practice activities were associated with a 3.2× stronger relationship to posttest learning outcomes compared to passive activities, replicating the doer effect (Model 1, β=0.351 vs β=−0.112, p<.001).
- Among active types, only code writing showed a statistically significant positive association with posttest performance (Model 2, β=0.288, t=3.39, p=0.001).
- Other active types (code tracing β=−0.107 and code completion β=0.183) and passive types (explanation β=0.297, visualization β=−0.279) were not significantly associated with posttest scores.
- The dataset included 334 students but the final analytic sample for regression models was smaller (n=96) due to requiring pre- and posttest completion.
- Pretest scores significantly predicted posttest scores (e.g., Model 1 β=0.513), confirming the control for prior knowledge.
- Mixed-effects modeling with class-level random intercepts showed no detectable clustering, indicating results hold across different course sections.
- Code writing uniquely combines generative problem-solving from scratch and immediate corrective feedback, hypothesized as critical to its superior association.
- Practice system content included 362 problems (115 writing, 117 tracing, 130 completion) and 143 example items (43 visualization, 100 explanation).
Methodology — deep read
The authors analyzed log and assessment data from 334 students enrolled in 11 semesters of introductory and intermediate Java programming courses at large U.S. universities from 2021 through 2025. The students used the Mastery Grids system, which provided unified access to five activity types of smart learning content (SLC): Code Writing (write Java code from scratch with autograder feedback), Code Tracing (predict code outputs with correctness feedback but no explanations), Code Completion (assemble code fragments with explanations), Code Visualization (view animated code execution passively), and Code Explanation (read line-by-line code explanations). The dataset contained a total of 362 practice problems (115 writing, 117 tracing, 130 completion) and 143 example items (100 explanations, 43 visualizations).
Participants engaged voluntarily; not all students used the system extensively, leading to a final analytic sample of n=96 students who completed both a 10-item pretest and a 10-item posttest. Pretest measured prior programming knowledge; posttest combined tracing and construction problems roughly equally.
Outcome variable: posttest score (fraction correct). Predictors: counts of attempts per activity type during the course. The authors standardized predictors as z-scores.
Two primary regression analyses were conducted using ordinary least squares (OLS) in R:
- Model 1 (aggregate doer effect): regressed posttest on pretest plus total problem attempts (sum of active types) and total example visits (sum of passive types).
- Model 2 (type decomposition): included pretest plus counts of each of the five activity types as separate predictors.
Mixed effects versions with class as random intercept were fit, but class-level variance was estimated as zero, so results matched OLS.
Interpretation focused on standardized beta coefficients and t-statistics; significance thresholds followed conventional p<0.05 criteria. The analysis was correlational and cross-sectional, adjusting for prior knowledge but not other confounds like motivation or time spent outside the system. The study treated all attempts equivalently, without separating unique vs repeated engagement.
A concrete example: a student engaging heavily in code writing activities (autograded coding problems requiring solution generation) showed higher posttest scores than a student focused primarily on passive content like code visualizations, even though the passive content was cognitively engaging.
The study did not perform experimental manipulation or causal inference, but rigorously controlled for pretest and examined multiple content types simultaneously to decompose the doer effect nuance.
Technical innovations
- Decomposition of the doer effect in programming education by distinct active practice types rather than treating active practice as a single homogeneous category.
- Analysis of the doer effect in a programming learning environment with interactive passive content (code visualizations, explanations), extending prior work focused on video/text passive content.
- Application of regression models to log data from an integrated multi-content practice system (Mastery Grids) incorporating multiple SLC types from different providers.
- Empirical evidence that code writing, combining generative construction and immediate corrective feedback, outperforms other active practice types in association with learning gains.
Datasets
- Mastery Grids dataset — 334 students total, 96 analytic sample with pre/post tests — available via DataShop
- Practice content: 362 problems (115 writing, 117 tracing, 130 completion) and 143 examples (100 explanations, 43 visualizations)
Baselines vs proposed
- Aggregate active practice attempts: β=0.351, t=3.713, p<0.001 vs passive examples attempts: β=−0.112, t=−1.181, p=0.240 (Model 1, n=111)
- Code writing attempts: β=0.288, t=3.391, p=0.001 vs code tracing attempts: β=−0.107, t=−0.638, p=ns; code completion attempts: β=0.183, t=1.322, p=ns; passive activities non-significant (Model 2, n=96)
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2608.02541.
Fig 1: An example of an active content type [29]– code
Fig 2: An example of a passive activity type – code visual-
Limitations
- Final analytic sample was relatively small (n=96) due to requiring completion of both pre- and posttests, reducing power and possibly introducing selection bias toward more motivated students.
- Observational correlational design with students self-selecting activities; thus causal claims about activity effects are not supported and confounding variables may influence results.
- Mastery Grids was a supplemental, voluntary practice system; students also learned via other course activities not captured here, limiting attribution of effects exclusively to practice activities.
- The analysis counted all attempts equally without distinguishing unique vs repeated practice of the same item, potentially conflating different learning effects.
- The study was limited to Java programming courses at a few U.S. institutions; generalizability to other programming languages, platforms, or learner populations remains untested.
- Mixed-effects models showed no class-level variance, but other hierarchical or longitudinal effects (e.g., within-student changes) were not explored.
Open questions / follow-ons
- To what extent does immediate corrective feedback, as opposed to activity type alone, drive the stronger association of code writing with learning gains?
- How should the sequencing or integration of code tracing, completion, and writing activities be optimized to scaffold learning most effectively?
- What is the causal impact of code writing practice compared to other activities in controlled experimental settings?
- How do repeated attempts versus unique attempts across different activity types differentially affect learning outcomes?
Why it matters for bot defense
For bot-defense and CAPTCHA practitioners, this paper illustrates the importance of finely decomposing active user engagement types rather than a binary active/passive classification. Just as programming education benefits most from generative, feedback-rich activities like code writing, bot-defense systems may find that nuanced user interactions requiring creative or generative input with immediate validation are more effective at distinguishing humans from bots than simpler active tasks or passive content consumption. This aligns with design principles for CAPTCHAs and behavioral challenges that demand complex, constructive input rather than mere cursory clicks or passive waiting.
Furthermore, the study underscores that interactive but passive content—analogous to visually rich or animated challenges that do not require user-produced responses—may engage users without reliably improving verification effectiveness. Therefore, in designing CAPTCHA or bot-defense challenges, privileging task types that combine generative action with instant, corrective feedback could improve bot detection reliability and user engagement. The methodology of log-based activity decomposition and the recognition of different cognitive demands in engagement types could also inspire more granular modeling of human interaction patterns to improve bot-detection models.
Cite
@article{arxiv2608_02541,
title={ Decomposing the Doer Effect in Programming Practice: Code Writing Stands Out Among Active Practice },
author={ Arun Balajiee Lekshmi Narayanan and Gillian Gold and Jordan Barria-Pineda and Quinn K Wolter and Peter Brusilovsky and Paulo Carvalho },
journal={arXiv preprint arXiv:2608.02541},
year={ 2026 },
url={https://arxiv.org/abs/2608.02541}
}