The Boundaries of Automation: A Theory of Persistent Human Participation
Source: arXiv:2607.21547 · Published 2026-07-23 · By Fares Fourati, Hinrich Schütze, Eyke Hüllermeier, Iryna Gurevych
TL;DR
This paper addresses a fundamental question in AI automation: are humans only kept in the loop because current AI is incapable, or are there deeper, conceptual limits to automation where human participation persists even with highly capable AI? The authors argue that persistent human involvement in human–AI systems is not simply a transient need due to technical shortcomings. Instead, it can be required on three grounds: (1) technical or complementarity reasons where humans contribute capabilities AI lacks; (2) normative or developmental reasons where participation itself is valuable for human agency or learning; and most critically, (3) emergence grounds where the "target" or goal of the interaction is not fully specified in advance but instead dynamically emerges through the human–AI interaction.
The core contribution is the development of the theory of target emergence, which formalizes how evaluative criteria, preferences, and objectives can evolve during interaction rather than being fixed up front. The authors show that in many real-world tasks—such as scientific discovery, design, or creative work—the goals and success standards are progressively constituted through iterative engagement with AI-generated artifacts. As a result, human–AI co-construction is not just a transitional workaround for imperfect AI but a lasting feature of tasks whose objectives unfold through collaboration. This perspective suggests new directions for human–AI system design, evaluation, and ethical considerations beyond capability improvement.
Empirically and conceptually, the paper distinguishes three levels of interaction: artifact-level (refining outputs), execution-level (refining workflows), and target-level (refining or changing what counts as success). Using examples (e.g., iterative image generation where the user changes their intended prompt through interaction), they show that target emergence complicates the classical view of static objectives in automated systems. The authors develop a formal dynamic model capturing the co-evolution of targets, artifacts, execution strategies, and human evaluative states over interaction rounds. The taxonomy of revelation, refinement, and constitution further clarifies how targets can emerge from tacit or vague goals to explicit and transformed evaluative standards.
Key findings
- Human participation persists in human–AI interaction for three distinct grounds: technical/complementarity, normative/developmental, and emergence (Table 1).
- Technical or complementarity justification for human input fades as AI capabilities improve, but normative and emergence grounds remain robust to AI advances.
- Target emergence occurs when the evaluative target is not fixed a priori but progressively revealed, refined, or constituted through interaction (Definition 2 and 3-5).
- Human–AI interaction can shift not just artifacts or workflows but the target evaluation criteria itself, requiring persistent co-construction (Fig 2).
- A formal dynamic model treats target (Gt), execution state (Et), artifact (Xt), and evaluative state (St) as coupled evolving variables over interaction rounds (Section 3.3).
- Target emergence is qualitatively classified into three forms: revelation (making latent targets explicit), refinement (operationalizing vague targets), and constitution (transforming evaluative standards).
- The model explains complex tasks like scientific discovery, where initial broad aims become concretely specified only through iterative AI-supported exploration and human judgment.
- Empirical and theoretical literature across philosophy, decision theory, AI alignment, and design supports the view of preferences and objectives as constructed rather than fixed.
Threat model
Not applicable. The paper does not focus on security threats or adversaries but on conceptual limits of automation and the persistence of human participation in human–AI systems despite advances in AI capabilities.
Methodology — deep read
Threat Model & Assumptions: The work is primarily conceptual/theoretical and does not center on a traditional security threat model. Adversaries are not explicitly considered. Instead, the threat is conceptual: overautomation that prematurely excludes humans from tasks where human participation remains essential. The paper assumes AI systems can become highly capable, possibly surpassing human performance in many technical tasks, but that human evaluative agency and target fluidity impose inherent limits on automation.
Data: This is a theoretical paper drawing on interdisciplinary literature rather than empirical datasets. The authors cite examples from domains such as scientific discovery, design, and image generation to illustrate phenomena but do not present new quantitative datasets or large-scale experiments.
Architecture / Algorithm: The main construct is a formal dynamic model of human–AI co-construction delineating evolving states over interaction rounds: target (Gt), execution state (Et), artifact (Xt), and evaluative state (St). The target-update process ΦG maps human–AI interaction history (Ht) to the next target Gt+1, capturing how interaction shapes objectives and criteria over time (Equation 1). This departs from classical fixed-objective optimization by integrating interaction-dependent changes in what counts as success.
Training Regime: Not applicable, as this is a theory paper rather than an empirical ML study.
Evaluation Protocol: The paper does not report traditional experimental metrics but supports claims through literature review, conceptual argumentation, and illustrative examples. Figure 1 metaphorically depicts target emergence as tandem biking where the destination is discovered en route. Figure 2 illustrates three interaction levels (artifact, execution, target). Appendix B gives a concrete use case in image generation where a user’s evaluative target shifts during interaction. Formal definitions and taxonomies operationalize concepts but are not empirically validated.
Reproducibility: No code or datasets are released or required, given the conceptual nature. The paper is situated within a broader interdisciplinary research program linking AI, decision theory, philosophy, and HCI.
Illustrative Example (Image Generation): A user initially requests a "modern coffee shop" image (artifact-level target). Over several interactive refinements, they realize they want a "warm, quiet neighborhood cafe" (target-level shift). This demonstrates target emergence where human intent and success criteria evolve through engagement rather than being fixed upfront, emphasizing why human oversight and iteration remain crucial even with capable AI.
Technical innovations
- Formulation of target emergence as a fundamental conceptual ground for persistent human participation beyond AI capability gaps.
- A dynamic model representing human–AI co-construction as coupled evolution of target, execution state, artifact, and evaluative states over interaction rounds (Equation 1).
- Taxonomy distinguishing three qualitatively different types of target emergence: revelation, refinement, and constitution.
- Explicit integration and differentiation of three levels of human–AI interaction—artifact, execution, and target—with theoretical and practical implications.
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.21547.
Fig 1: A motivating metaphor for target emergence in human–AI co-construction. The
Fig 2: Levels of human–AI interaction. Artifact-level interaction modifies the output,
Limitations
- The paper is primarily conceptual and theoretical without empirical validation of the proposed dynamic model of target emergence.
- No quantitative experiments or longitudinal user studies demonstrating target emergence in practice are presented.
- The taxonomy and formalism rely on abstract definitions whose applicability may vary significantly across domains and user populations.
- The approach does not address adversarial or malicious human actors, focusing instead on cooperative human–AI interaction.
- Evaluation and design recommendations remain high-level, without concrete system-building guidelines or tested prototypes.
Open questions / follow-ons
- How can concrete human–AI systems be designed to explicitly support and detect target emergence during interaction?
- What methods can quantitatively measure target emergence and its impact on task success and human satisfaction?
- How does target emergence manifest in different domains (e.g., creative writing, scientific discovery, code generation), and how domain-general is the theory?
- What are the ethical implications and best practices for balancing human agency and AI autonomy in tasks with emergent objectives?
Why it matters for bot defense
For bot-defense and CAPTCHA practitioners, this theory cautions against assuming that tasks requiring human involvement will naturally vanish as AI systems improve technically. Target emergence highlights that in interactive challenge-response or verification tasks, the validation criteria themselves may evolve through human interaction rather than being fixed targets. This underscores the importance of designing systems that enable flexible human participation rather than purely automated flows.
Moreover, recognizing normativity and emergence as persistent grounds for human co-construction implies CAPTCHAs and bot defenses may need to maintain human engagement not just to compensate for AI errors, but to preserve adaptive evaluative judgment sensitive to context changes or latent intents that purely algorithmic checks cannot predefine. The conceptual framework encourages practitioners to rethink automation limits and adopt interaction paradigms designed for ongoing human oversight, especially where verification targets are inherently fluid or jointly formed.
Cite
@article{arxiv2607_21547,
title={ The Boundaries of Automation: A Theory of Persistent Human Participation },
author={ Fares Fourati and Hinrich Schütze and Eyke Hüllermeier and Iryna Gurevych },
journal={arXiv preprint arXiv:2607.21547},
year={ 2026 },
url={https://arxiv.org/abs/2607.21547}
}