KC-Agent: A Dual-Process Cognitive Architecture for Efficient ML Model Improvement

Source: arXiv:2608.02351 · Published 2026-08-03 · By Gusseppe Bravo-Rocca, Jordi Guitart, Ajay Dholakia, David Ellison, Puneet Jain

TL;DR

KC-Agent addresses the critical challenge of maintaining and improving ML models in production under data drift by introducing a novel dual-process cognitive architecture inspired by Kahneman’s System 1 and System 2 cognitive theory. It integrates fast pattern recognition mechanisms for rapid adaptive responses with slower, incremental, atomic model updates to ensure reliable and verifiable improvements. The architecture incorporates structured semantic, episodic, and working memory modules enabling the agent to learn from past improvements and transfer successful strategies for efficient future adaptations. Evaluated on five datasets—including challenging real-world NASA turbofan data exhibiting authentic temporal data drift and multiple synthetic scenarios with controlled drift—KC-Agent outperformed several state-of-the-art LLM-based autonomous agent frameworks in terms of accuracy, efficiency, and strategic quality. The knowledge consolidation mechanism reduced computational cost by 91% compared to the slow deliberative process while improving accuracy.

Key findings

• KC-Agent achieved an average accuracy of 76.8% ± 11.6% across five datasets, outperforming baselines such as CodeAct (+2.4%), Tree of Thoughts (+3.6%), ReAct (+8.0%), and Reflexion (+8.9%).
• Execution time averaged 13.2 seconds with low variance, significantly faster than KC-Slow’s 141.2 seconds (91% speedup).
• Token consumption was 847 tokens on average, 18.7× fewer than Reflexion and substantially lower than other baselines.
• System 1 resolved 68% of scenarios using learned patterns, enabling rapid improvement without deliberation.
• KC-Agent maintained 100% execution success rate, compared to 34% failure rate in baseline approaches.
• On NASA FD001 dataset, KC-Agent improved accuracy by 24% relative (61.3% vs. 49.3% baseline).
• Qualitative evaluation by three state-of-the-art LLM judges rated KC-Agent highest for Smartness (8.33/10), outperforming others in effectiveness of improvement strategies.
• Ablation study showed full dual-process system outperformed single-system variants by 1.8% accuracy and drastically reduced computational cost.

Threat model

The adversary consists of natural or operational data drift phenomena causing ML model performance degradation over time. There is no assumption of targeted adversarial attacks on the model or agent. The agent must autonomously detect and adapt to distribution shifts without immediate manual intervention. The system assumes the LLM and computing environment remain uncompromised.

Methodology — deep read

1. Threat Model & Assumptions: The adversary is an environment causing data distribution drift that degrades model performance. The agent must detect and adapt the model without manual intervention. There is no adversarial tampering with the agent itself, but drift patterns can be subtle and unknown.

2. Data: Five datasets used - two real-world NASA turbofan datasets exhibiting natural temporal degradation with 10,000 and 16,000 samples respectively, and three synthetic datasets (Financial, Healthcare, Eligibility) generated/refined by GPT-4 to introduce controlled realistic drift validated by KL divergence analysis. Each dataset contains paired old/new distributions simulating drift.

3. Architecture / Algorithm: KC-Agent employs a dual-process cognitive architecture with two reasoning systems: System 1 (Fast Graph) performs rapid pattern-matching-based updates reusing prior successful solutions stored in semantic and episodic memories. System 2 (Slow Graph) performs deliberate incremental improvements via atomic changes constrained by norm ≤ η, allowing precise, reversible code modifications (e.g. hyperparameter tuning, model selection). Knowledge consolidation continuously distills System 2 discoveries into semantic memory enabling System 1 to handle increasingly complex scenarios.

Memory incorporates three components: Semantic Memory stores general improvement knowledge and code, Episodic Memory stores specific scenarios and improvement outcomes, and Working Memory maintains context during reasoning.

4. Training Regime: The agent leverages Llama-3.1-8b LLM with token context 8192, optimized on Intel Xeon CPUs. Batch processes consist of reasoning sessions iterating until improvement or max iterations. Proxy decision threshold µ=0.05 controls acceptance of System 1 fast fixes. Atomic changes evaluated individually and reverted on failure to ensure monotonic improvement.

5. Evaluation Protocol: Metrics include accuracy on old/new data distributions, execution time, and token consumption reflecting real production constraints. Baselines include ReAct, Reflexion, Tree of Thoughts, CodeAct, Self-Discovery, Plan-and-Execute, and a standard single-call baseline. Quantitative evaluation averaged across datasets with multiple runs and fixed seeds for reproducibility. Qualitative evaluation uses consensus panel of three state-of-the-art LLMs scoring Stability, Readability, and Smartness on code improvements.

6. Reproducibility: Code and datasets managed with version control, unique experiment IDs, and fixed random seeds. Memory indexing and workflow orchestration via LangGraph. Implementation details and hardware environment documented. Datasets include proprietary (NASA turbofan) and synthetic data refined by GPT-4, partially non-public.

Technical innovations

• Dual-process cognitive architecture combining fast pattern matching with slow incremental atomic model improvements for ML drift adaptation.
• Structured tri-memory system (semantic, episodic, working) enabling knowledge consolidation and effective strategy transfer across tasks.
• Atomic change principle constraining model updates to small, reversible steps with formal monotonic improvement guarantees.
• Proxy decision mechanism that dynamically delegates between fast and slow reasoning based on performance delta thresholds.

Datasets

• NASA FD001 — 10,000 samples — real-world industrial turbofan engine data with temporal drift
• NASA FD002 — 16,000 samples — real-world industrial turbofan engine data with temporal drift
• Financial — 1,000 samples — synthetic dataset modeling loan default prediction with market noise drift
• Healthcare — 1,000 samples — synthetic dataset modeling chronic condition prediction with population noise drift
• Eligibility — 1,000 samples — synthetic dataset modeling administrative policy decision noise drift

Baselines vs proposed

• CodeAct: accuracy = 74.4% vs KC-Agent: 76.8%
• Tree of Thoughts: accuracy = 74.1% vs KC-Agent: 76.8%
• ReAct: accuracy = 71.1% vs KC-Agent: 76.8%
• Reflexion: accuracy = 70.5% vs KC-Agent: 76.8%
• KC-Slow (slow deliberative only): accuracy = 76.2% vs KC-Agent: 76.8%
• KC-Fast (fast pattern only): accuracy = 75.4% vs KC-Agent: 76.8%

Limitations

• Evaluation datasets are limited in scale and scope; larger and more diverse real-world production datasets would strengthen generalizability.
• No explicit adversarial or malicious data drift testing was performed, limiting analysis under hostile conditions.
• No evaluation of performance or stability under abrupt or catastrophic distribution shifts.
• LLM evaluation relies on consensus of other language models, not human domain experts, potentially biasing qualitative assessments.
• The proxy decision threshold µ is fixed; adaptive thresholding could optimize trade-off between speed and reliability but was unexplored.
• The system’s reliance on Llama-3.1-8b limits understanding of scalability or performance with larger/smaller models.

Open questions / follow-ons

• How effectively can the dual-process architecture handle abrupt, extreme, or adversarial distribution shifts beyond smooth drift?
• Can adaptive or dynamic proxy thresholding improve the balance of fast vs. slow processing in different drift contexts?
• How does scaling the language model size or memory architecture complexity impact model improvement quality and efficiency?
• What are the limits of knowledge consolidation when encountering highly novel or non-stationary drift scenarios over longer operational horizons?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, KC-Agent’s dual-process cognitive architecture provides a promising paradigm for automated, reliable adaptation of ML models under evolving attack patterns or behavioral shifts. The fast System 1 mechanism enables quick, pattern-based responses to known threat adaptations, while the slow System 2 component allows deliberate incremental improvements to counter novel or complex evasion strategies. The atomic change principle and rollback capabilities ensure updates can be safely tested and reverted, critical for production security-sensitive systems where stability is paramount. Additionally, the structured memory and knowledge consolidation mechanisms could enable bot-defense systems to learn from past threat response successes, reducing repeated exploration and improving responsiveness over time. Applying such dual-process architectures to CAPTCHA or bot-detection ML pipelines may enhance resilience against evolving adversary behaviors and reduce manual retraining burdens.

Cite

@article{arxiv2608_02351,
  title={ KC-Agent: A Dual-Process Cognitive Architecture for Efficient ML Model Improvement },
  author={ Gusseppe Bravo-Rocca and Jordi Guitart and Ajay Dholakia and David Ellison and Puneet Jain },
  journal={arXiv preprint arXiv:2608.02351},
  year={ 2026 },
  url={https://arxiv.org/abs/2608.02351}
}

Read the full paper

• arXiv:2608.02351 (abstract)
• arXiv:2608.02351 (PDF)