Exposure is Optional: Learning Unlike Coordination in Language Models

Source: arXiv:2607.20251 · Published 2026-07-22 · By Jiamu Luo, Shane Steinert-Threlkeld

TL;DR

This paper investigates the linguistic phenomenon of unlike coordination, where conjuncts differ in syntactic category, challenging the classical Law of Coordination of Likes (LCL). Using language models (LMs), specifically GPT-2 architectures, as controlled computational learners, the authors aim to determine whether direct exposure to unlike coordination forms during training is necessary for the models to acquire and generalize this structure. By leveraging a method called Filtered-Corpus Training (FiCT), they train multiple GPT-2 models on corpora with all unlike coordination instances removed and compare their performance to models trained on unfiltered corpora.

Their findings show that direct exposure to unlike coordination is not required: models trained without any unlike coordination examples achieve perplexities and grammaticality judgments comparable to those trained on full data. Analysis of internal model representations reveals two complementary mechanisms by which LMs process unlike coordination: grouping conjuncts into abstract "supercategories" that unify them functionally despite syntactic differences, or using deletion-like mechanisms that implicitly recover elided material. These phenomena emerge organically from exposure to alike coordination alone, offering computational evidence against the necessity of strict LCL-based constraints or specialized grammatical rules for unlike coordination.

Key findings

• GPT-2 models trained on filtered corpora with all unlike coordination instances removed (filtered-all) achieve perplexities statistically indistinguishable from models trained on unfiltered text across several targeted evaluation corpora.
• Grammaticality judgment accuracy on unlike coordination tests is only modestly lower for filtered-all models (mean ~67%) compared to original models (mean ~70%), with filtered-and variants (filtering only 'and' conjunctions) in between.
• Attention analysis shows that models employ two distinct strategies: supercategory-based processing (high attention from query to first conjunct phrase) and deletion-based processing (attention shifted to a potential ellipsis site), with exposure to unlike coordination enhancing the use of deletion-based mechanisms.
• Cosine similarity between conjunct representations is highest for alike coordination, moderate for supercategory-unlike coordination, and lowest for deletion-unlike coordination, supporting differentiated internal representations.
• Clustering of last-layer conjunct embeddings reveals that for deletion-like unlike coordination, conjuncts cluster by strict grammatical categories (nouns, verbs, adjectives), while for supercategory-like cases, conjuncts cluster more by sentence context and functional similarity.
• Filtering all unlike coordination does not prevent models from generalizing and acquiring unlike coordination, refuting the hypothesis that explicit exposure is necessary.
• Models struggle with fine-grained syntactic constraints that modulate category mismatch, indicating limitations in fully capturing nuanced restrictions on unlike coordination.
• Models strongly reject coordination across genuinely mismatched supercategories, suggesting a learned bias toward conjunct similarity without direct categorical constraints.

Threat model

Not applicable; this work is a linguistic and computational study analyzing language model learning rather than adversarial security work.

Methodology — deep read

1. Threat Model & Assumptions: The adversary is represented by language models (GPT-2 variants) trained under controlled conditions where direct exposure to unlike coordination is selectively removed. The study assumes LMs have no innate linguistic biases enforcing the Law of Coordination of Likes (LCL), testing whether they require explicit training examples of unlike coordination to generalize it.

2. Data: Training data originate from the English corpus used by Gulordava et al. (2018) with approximately 1.95 million sentences. Two filters are applied leveraging the Berkeley Neural Parser to remove sentences containing unlike coordination entirely (filtered-all) or just those joined by the conjunction "and" (filtered-and). The original, filtered-and, and filtered-all corpora have identical sentence counts after downsampling but differ by token counts (~91% tokens retained in filtered-and, ~88% in filtered-all).

3. Architecture / Algorithm: GPT-2 Large configuration with 36 layers, 1280 embedding size, 20 attention heads, and context length 1024, trained using causal language modeling objectives. Three model variants (original, filtered-and, filtered-all) are trained from scratch. Models are implemented via Huggingface transformers.

4. Training Regime: Each variant is trained with three random seeds (291, 1543, 9071) for robustness, producing nine trained models. Training details including epoch count, batch size, optimizer specifications, and hardware are in the appendix but not detailed in the truncated text.

5. Evaluation Protocol: Custom evaluation corpora are constructed, comprising 100 unlike coordination sentences (varied syntactic patterns), 100 paired alike-from-unlike sentences, two specialized corpora targeting supercategory and deletion phenomena, and a 22-sentence unlike grammaticality judgment set. Metrics used include perplexity, average surprisal (for grammaticality judgments), average attention scores (quantifying attention from query tokens to key phrases), cosine similarity on conjunct embeddings, and clustering analysis. Statistical testing involves pairwise t-tests, Welch's tests, and linear mixed-effects models.

6. Reproducibility: Code and data will be shared on GitHub upon acceptance. The corpus filtering pipeline is described in detail, enabling reproducibility of the filtered datasets. This includes the filters' performance evaluation with 100-sentence test sets confirming high precision. Models and training procedures are standard and implemented with common frameworks.

Example end-to-end: A model variant is trained on the filtered-all corpus with no exposure to unlike coordination. It is then evaluated on the unlike corpus and achieves perplexity comparable to the original model. Attention patterns during processing of deletion-type unlike coordination sentences show that the model attends more to the potential elided phrase rather than the first conjunct, indicating a deletion-based processing mechanism without direct exposure. Cosine similarity between conjunct embeddings corresponds to theoretical predictions (lowest for deletion). This sequence confirms the model's ability to internally represent and generalize unlike coordination mechanisms by relying on structural cues rather than explicit training data.

Technical innovations

• Use of Filtered-Corpus Training (FiCT) to systematically exclude unlike coordination examples in the training corpus and isolate their effect on language model acquisition.
• Introduction of precise automated filters based on syntactic parsing to remove unlike coordination sentences from large corpora with high precision.
• Novel multi-faceted evaluation paradigm combining perplexity, grammaticality judgments, attention-based probing, cosine similarity, and clustering of internal representations to examine how LMs process unlike coordination.
• Demonstration that unlike coordination can emerge from exposure to only alike coordination, suggesting general compositional abilities and latent structural cues suffice without explicit rules.
• Empirical evidence for dual internal processing mechanisms in LMs—supercategory abstraction and deletion-like recovery—which connect computational behaviors to linguistic theories.

Datasets

• Gulordava et al. English corpus (2018) — ~1,948,104 sentences — public
• Unlike evaluation set — 100 sentences — manual curated from literature and COCA corpus
• Alike-from-unlike evaluation set — 100 sentences — constructed minimal pairs from unlike set
• Supercat-deletion evaluation corpus — 62 sentences — subset of unlike exhibiting supercategory phenomena
• Deletion evaluation corpus — 24 sentences — subset of unlike exhibiting deletion phenomena
• Unlike-judgment corpus — 22 sentences — grammaticality judgment tests from theoretical linguistics literature

Baselines vs proposed

• Original (unfiltered) GPT-2: grammaticality judgment accuracy = 70% (mean), perplexity baseline
• Filtered-all GPT-2 (no unlike coordination exposure): grammaticality judgment accuracy = 67% (mean), perplexity close to original
• Filtered-and GPT-2 (partial filtering): grammaticality judgment accuracy = 65% (mean), perplexity slightly better than filtered-all but lower than original
• Perplexity: original < filtered-and ≤ filtered-all (difference statistically significant but numerically small)
• Attention pattern distinctions for deletion vs supercategory phenomena stronger in original and filtered-and models compared to filtered-all

Limitations

• Parsing errors and strict filtering criteria removed some alike coordination sentences, potentially affecting representativeness of filtered corpora.
• Training regime details such as number of epochs and hyperparameters are not fully described in the truncated text.
• Evaluations rely primarily on perplexity, grammaticality judgments, and analysis of internal representations, but do not test robustness under adversarial or out-of-domain distributions.
• Models show limited sensitivity to nuanced syntactic constraints governing category mismatch, suggesting incomplete linguistic competence.
• No large-scale human evaluation or psycholinguistic validation of LM judgments is presented.
• The study focuses on English and GPT-2 architecture, so generalization to other languages or model types is not tested.

Open questions / follow-ons

• How do alternative cues such as semantic similarity and contextual compatibility influence unlike coordination generalization in LMs?
• Can systematically controlled corpora isolating semantic vs syntactic coordination cues further clarify which features LMs rely on for unlike coordination?
• How do training data distributions shape the balance between supercategory and deletion mechanisms within LMs?
• Do larger or more recent language model architectures replicate these findings across different languages and modalities?

Why it matters for bot defense

This work deepens the understanding of how large language models acquire and internally represent complex syntactic phenomena such as unlike coordination without explicit exposure. While not directly a CAPTCHA or bot-defense study, the insights on compositional generalization and emergent representation learning in language models inform how sophisticated LMs might process—and potentially circumvent—linguistic challenge-response tests. Recognizing that LMs generalize high-level syntactic constraints from indirect evidence underscores the need for bot-defense challenges that exploit subtle linguistic nuances or fine-grained constraints that models struggle with, such as rare category mismatches or semantic incompatibilities in coordination. Moreover, the dual mechanisms (supercategory abstraction and deletion) models employ suggest that CAPTCHA designs targeting compositional and structural linguistic phenomena must consider the possibility that advanced LMs may leverage internal latent representations to solve language tasks beyond surface patterns. Therefore, bot-defense practitioners might apply similar syntactic probes and representation analyses to identify potential model weaknesses or to tailor challenges that expose limits of generalization.

Cite

@article{arxiv2607_20251,
  title={ Exposure is Optional: Learning Unlike Coordination in Language Models },
  author={ Jiamu Luo and Shane Steinert-Threlkeld },
  journal={arXiv preprint arXiv:2607.20251},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.20251}
}

Read the full paper

• arXiv:2607.20251 (abstract)
• arXiv:2607.20251 (PDF)