IS-CoT: Breaking the Long-form Generation Collapse via Interleaved Structural Thinking

Source: arXiv:2606.09709 · Published 2026-06-08 · By Zechen Sun, Yuyang Sun, Zecheng Tang, Juntao Li, Wenpeng Hu, Wenliang Chen et al.

TL;DR

This paper addresses a critical limitation in current large language models (LLMs) for open-ended long-form text generation, a difficulty known as "Length Collapse" where generation quality and length adherence sharply degrade beyond about 2,000 words. The authors attribute this failure to static, one-shot global hierarchical planning approaches that struggle to provide dynamic and context-sensitive guidance over extended text. To mitigate this, they propose the Interleaved Structural Chain-of-Thought (IS-CoT) framework, which interleaves global planning, local step-wise planning, content generation, and reflection cycles dynamically throughout the generation process. This Plan-Write-Reflect strategy allows the model to continuously update its strategy and alignment, reducing divergence and early termination issues common in ultra-long generation. They build a high-quality dataset of 5,000 samples with explicit interleaved reasoning traces via multi-teacher distillation involving two teacher models, and fine-tune an 8B parameter LLM, IS-Writer-8B, on this dataset. Experiments demonstrate that IS-Writer-8B significantly outperforms strong proprietary (e.g., Gemini-2.5-Flash) and open-source baselines on long-form benchmarks like LongBench-Write and WritingBench, especially in the ultra-long [4k-20k] word range, achieving superior length adherence and coherence despite its smaller size. Ablations confirm that both interleaved planning and reflection modules critically contribute to performance. The paper highlights the necessity of dynamic, continuous planning mechanisms for controllable, high-quality ultra-long text generation.

Key findings

• LLMs including GPT-4o, LLaMA-3.3, DeepSeek-V3, and reasoning-enhanced DeepSeek-R1 experience "Length Collapse" where quality and length compliance degrade sharply beyond 2,000 words, with length following scores dropping by ~68.5 points between 2,000 and 8,000 words (Fig 2).
• Reasoning-enhanced "Thinking" models outperform standard instruction-tuned versions across all lengths but also suffer significant performance degradation as target length increases (e.g., Qwen3-Thinking loses ~10.89 points in quality score and 13.53 points in length following beyond 2,000 words).
• IS-Writer-8B achieves an average LongBench-Write score of 88.25, surpassing the best proprietary model Gemini-2.5-Flash by +4.58 and outperforming larger open-source LLMs like DeepSeek-V3.2-671B by +3.08 (Table 1).
• In the challenging ultra-long [4k, 20k) word range, IS-Writer-8B achieves a length following score (Sl) of 88.31 and quality score (Sq) of 88.19, outperforming the best open-source and writing-specific baselines by +13.55 and +10.09 points respectively.
• Ablation studies show removing the reflection module reduces overall LongBench-Write score gains from +4.73 to +2.45, and removing both interleaved planning and reflection drops gains to +0.55, confirming the importance of iterative plan-reflect cycles (Table 2).
• IS-Writer-8B consistently meets or slightly exceeds target lengths across [0,20k) range with much lower variance compared to baselines, mitigating “early stopping” common in long-form generation (Fig 5).
• Dynamic Plan-Write-Reflect process supervision allows IS-Writer to surpass teacher models (Qwen3-235B, DeepSeek-V3.2) despite their larger size, as teachers struggle with single-pass ultra-long context organization.
• Case studies show IS-Writer explicitly tracks word count progress and thematic coherence after each segment allowing near-perfect length adherence (e.g., length compliance score of 99.63 in a narrative task).

Threat model

The paper does not focus on adversarial security threats but examines the failure modes of LLMs for controllable long-form text generation. The implicit adversary is model architecture and static planning strategies that fail to maintain coherence and adherence over very long generated documents exceeding 2,000 words. The capability limitation involves models only having one-shot static plans at generation start, without ability to dynamically adapt plans during generation. The threat is the collapse in output quality and constraint adherence during ultra-long open-ended generation. The approach aims to mitigate this internal production threat by embedding iterative strategic reasoning within the generation process.

Methodology — deep read

The authors first define their threat model as the limitation of current LLMs to maintain coherence and length constraints over long-form text (>2,000 words) due to weakening static plan guidance during generation.

They sample 6,000 seed writing prompts from the DeepWriting dataset evenly balanced in English and Chinese, with half having explicit length constraints. These prompts undergo an instruction refinement process via a reasoning-enhanced LLM (DeepSeek-V3.2) to clarify task requirements and assign length budgets.

A key novelty is the construction of an interleaved reasoning dataset. This is done via a multi-teacher distillation framework combining two large teacher models: DeepSeek-V3.2 and Qwen3-235B-A22B-Instruct. This dual-teacher strategy improves diversity and style robustness. Generation is formulated recursively in four components marked by special tokens: (1) global_plan for high-level outline and length allocation; (2) step_plan for planning the current writing segment; (3) content for actual text generation; and (4) reflection for progress evaluation and plan adjustment.

A heuristic recursive generation control enforces strict adherence to the interleaved format during data synthesis. After generating a reflection token, if length constraints are unmet, generation must continue with another step_plan; otherwise, it terminates with a <FINISHED> token.

Quality filtering removes samples with length following scores below 90 and applies manual verification, yielding a high quality dataset of ~5,000 samples with explicit reasoning traces.

IS-Writer-8B is trained via supervised fine-tuning on this IS-CoT dataset, optimizing a standard autoregressive log-likelihood loss over the entire sequence including planning and reflection tokens. Training is done for 5 epochs with batch size 64 using DeepSpeed ZeRO-3 on 64 NVIDIA H800 GPUs and an extended context window of 32,768 tokens.

Evaluation is conducted on two major benchmarks: LongBench-Write (targeting ultralong open-ended writing) and WritingBench (length-constrained tasks across 6 domains). Quality and length following scores are evaluated using LLM-as-judge with GPT-4o-mini. Baselines include strong proprietary models (GPT-4o, Gemini-2.5-Flash), open-source LLMs (DeepSeek series, Qwen3-235B), and specialized long-context specialists (LongWriter-8B).

Ablation studies compare IS-Writer-8B to variants without the reflection module and without interleaved local planning to quantify the contribution of each.

A case study on narrative writing shows IS-Writer’s plan-reflect loops explicitly track word count progress and thematic relevance, generating more controllable and compliant text than baselines.

The authors do not mention open-sourcing their code or weights but provide detailed dataset construction and training protocols that should support reproducibility. However, the synthesized dataset is not publicly released at this time.

Overall, the methodology carefully integrates continuous dynamic hierarchical planning with reflection checkpoints to overcome the failure modes of static one-shot planning in ultra-long generation, with a sophisticated multi-teacher data generation pipeline and rigorous evaluation on challenging benchmarks.

Technical innovations

• Interleaved Structural Chain-of-Thought (IS-CoT) framework combining Plan-Write-Reflect cycles enabling dynamic, iterative structural planning rather than static one-shot outlines.
• Multi-teacher distillation approach synthesizing high-quality, process-supervised interleaved reasoning traces from diverse powerful models to prevent overfitting to a single style.
• Heuristic-guided recursive data synthesis loop enforcing strict interleaved format and length adherence during dataset construction.
• Training a LLM to predict planning and reflection tokens jointly with content, internalizing a continuous feedback loop that dynamically updates generation strategy.
• Extending context window to 32,768 tokens for ultra-long generation combined with efficient distributed training (ZeRO-3) for scalable supervised fine-tuning.

Datasets

• IS-CoT dataset — ~5,000 samples — constructed via multi-teacher distillation pipeline from DeepSeek-V3.2 and Qwen3-235B teacher models, with manual and heuristic filtration.
• DeepWriting dataset subset — 6,000 seed prompts balanced English/Chinese — used for instruction refinement and seed prompt generation.

Baselines vs proposed

• Gemini-2.5-Flash: LongBench-Write average score = 83.67 vs IS-Writer-8B = 88.25 (+4.58)
• DeepSeek-V3.2-671B: LongBench-Write average score = 85.17 vs IS-Writer-8B = 88.25 (+3.08)
• Qwen3-235B-A22B-Instruct: LongBench-Write average score = 87.15 vs IS-Writer-8B = 88.25 (+1.10)
• Qwen3-8B baseline: LongBench-Write average score = 83.52 vs IS-Writer-8B full = 88.25 (+4.73)
• IS-Writer w/o Reflection: LongBench-Write = 85.97 vs full = 88.25 (-2.28)
• IS-Writer w/o Interleaved Planning: LongBench-Write = 84.07 vs full = 88.25 (-4.18)

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.09709.

Fig 1: Comparison of three long-form generation

Fig 2: Comparison of quality and length-following

Fig 3: Performance comparison between reasoning-enhanced models (“Thinking”) and standard instruct/chat

Fig 4 (page 1).

Fig 5 (page 1).

Fig 4: The overall framework of IS-CoT. We construct the IS-CoT Dataset through a three-stage pipeline: (1)

Fig 7 (page 4).

Fig 8 (page 4).

Limitations

• Model performance remains tied to quality and diversity of teacher models; improvements may be limited by training data synthesis supervision.
• Inference overhead is increased due to interleaved generation of planning and reflection tokens, consuming more tokens and compute than direct generation models.
• Dataset of interleaved reasoning traces (~5,000 samples) is relatively small compared to typical LLM training corpora.
• No adversarial evaluation or rigorous testing under domain shift conditions was reported.
• Performance gains are mostly evaluated on specific long-form writing benchmarks, generalization to highly diverse creative or multi-modal texts remains unclear.

Open questions / follow-ons

• How does IS-CoT perform under different domains outside the studied benchmarks, such as fiction, technical writing, or multi-modal content?
• Can the framework be extended to incorporate user feedback or interactive editing for better dynamic control?
• What are more token-efficient methods to achieve similar dynamic plan-reflect reasoning without the significant inference overhead?
• How would increasing teacher model diversity or combining with reinforcement learning approaches affect the quality and generalization of the IS-CoT framework?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, the findings highlight the intrinsic challenges in generating ultra-long coherent text with strict length adherence, which could inform the detection of machine-generated long-form content. IS-CoT's dynamic interleaved planning suggests that state-of-the-art generative models capable of continuous structural self-reflection during generation might produce more consistent and longer outputs that better mimic human writing patterns. Defenses relying solely on static heuristic or pattern detection in generated text may become less effective as models adopt dynamic reasoning loops. Understanding these generation frameworks could guide development of more robust bot-detection features focused on subtle interleaved planning signals or reflection artifacts in text structure. Additionally, methods to distinguish static planning versus dynamic planning styles might become relevant signals for differentiating human versus machine long-form writing. Overall, the work underlines the sophistication of emerging long-form generation processes, urging continuous evolution of analytical tools in bot-defense and CAPTCHA systems to address such adaptive generation behaviors.

Cite

@article{arxiv2606_09709,
  title={ IS-CoT: Breaking the Long-form Generation Collapse via Interleaved Structural Thinking },
  author={ Zechen Sun and Yuyang Sun and Zecheng Tang and Juntao Li and Wenpeng Hu and Wenliang Chen and Zhunchen Luo and Guotong Geng and Min Zhang },
  journal={arXiv preprint arXiv:2606.09709},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.09709}
}

Read the full paper

• arXiv:2606.09709 (abstract)
• arXiv:2606.09709 (PDF)