Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages

Source: arXiv:2606.20517 · Published 2026-06-18 · By Maria Ivanova, Pavel Zadorozhny, Rodion Levichev, Ivan Petrov, Adamenko Pavel, Ivan Lopatin et al.

TL;DR

Multi-LCB addresses a critical limitation of the LiveCodeBench (LCB) benchmark, which evaluates large language models (LLMs) on code generation tasks exclusively in Python. Recognizing that real-world software engineering requires proficiency across diverse programming languages, the authors extend LCB to 12 programming languages including C++, Java, Rust, and others. Multi-LCB preserves LCB's contamination-aware evaluation protocol by converting Python tasks into equivalent tasks in other languages, using a novel pipeline to unify test formats into a language-agnostic STDIN/STDOUT scheme. The benchmark can track future LCB updates automatically and enables rigorous cross-language evaluation of LLM capabilities on identical competitive programming problems.

Key findings

• Python overfitting: Models showing strong Python results degrade significantly on other languages, e.g., OpenRsn-Nmt-32B scored >60% on Python but <30% on other languages (Fig 3).
• Substantial performance gaps: Top models achieve around 48.2% Pass@1 on Python but only around 33% on languages such as C#, Ruby, PHP, Go, Rust, Kotlin, and JavaScript/TypeScript (Fig 4).
• Language-specific contamination: Time analysis reveals inflated scores on pre-cutoff tasks likely due to training data leakage; post-cutoff task performance reflects true generalization (Fig 5).
• Benchmark fidelity: Multi-LCB reproduces original LCB Python results closely, with mean absolute deviation about 3%, confirming the extension does not add artificial difficulty (Table 2).
• Model ranking varies by language: For example, GPT-OSS-120B* outperforms Qwen3-235B-A22B-Thk-2507* on Go, JavaScript, Rust and Kotlin despite lagging behind on Python.
• Zero-shot prompting with unified I/O format supports robust automated evaluation of generative code across languages without language-specific test rewrites.
• Most evaluated models fail to exceed 50% Pass@1 over all languages, underscoring the ongoing challenge of robust multilingual code generation.

Threat model

The evaluation considers an adversary as a large language model with potential knowledge of the training corpus, but with no access to problems published after the specified cutoff dates to avoid contamination. It assumes the adversary can generate code solutions for given natural language programming tasks but does not have privileged knowledge of hidden test cases or task variants in other languages. The threat model focuses on preventing data leakage for fair, zero-shot generalization assessment, with no attempt at adversarial manipulation or evasion of the benchmark setup.

Methodology — deep read

1. Threat Model & Assumptions: The adversary is an LLM developer or evaluator seeking contamination-free assessment of multi-language coding capability. The contamination control assumes the training data cutoff date prevents exposure to post-release problems. The evaluation focuses on the ability to generate correct code given natural language problem statements that do not leak language-specific biases.

2. Data: Multi-LCB builds on the LiveCodeBench dataset versions v4 to v6, which consists of competitively sourced programming problems from LeetCode, AtCoder, and Codeforces, curated with release date metadata for contamination filtering. The dataset includes 1,055+ problems originally in Python for code generation tasks, with test cases hidden and public. Problems requiring multiple valid answers or data structure construction were excluded.

3. Architecture / Algorithm: Instead of retraining a model, the benchmark consists of a zero-shot prompting evaluation where a problem prompt (natural language statement plus input/output examples) is constructed for each task and language. The key novelty is converting all problems into a unified STDIN/STDOUT testing format, enabling language-agnostic evaluation. This is achieved via an automatic pipeline that parses functional-format (LeetCode) problems and rewrites their test cases into standardized input-output streams.

4. Training Regime: Not applicable as the paper evaluates pretrained LLMs without further training. Twenty-four open models ranging from 7B to 685B parameters were tested, covering general and code-specialized models with diverse training paradigms (pure code, mixed, instruction tuning, reasoning enhancement).

5. Evaluation Protocol: Models generate zero-shot code solutions given prompts specifying target language. Code outputs are executed securely inside isolated sandboxes with resource limits (6s runtime, 4GB RAM, no network). Correctness is Pass@1 defined as the fraction of problems for which the first generated solution compiles and passes all hidden official tests. Evaluation is averaged over 10 runs using nucleus sampling (temp=0.2, top-p=0.95). Results include per-language and aggregated accuracy. Performance on tasks released after model training cutoffs are emphasized to minimize contamination bias.

6. Reproducibility: The full code, prompts, and configs for Multi-LCB are publicly released at https://github.com/Multi-LCB/Multi-LCB. The underlying LCB dataset is available via Hugging Face. Models evaluated are open and results reproducible given identical inference setups. Frozen model weights are not altered.

Example: For a LeetCode problem originally in Python functional form, Multi-LCB converts its test cases into STDIN/STDOUT format specifying input dimensions and space-separated values. The zero-shot prompt includes a system message like “You are an expert C++ programmer,” the problem description, sample input/output, and a C++ code block placeholder. The model generates a complete C++ program reading from stdin and writing stdout. This code is compiled with GCC 13 in a sandbox, executed, and output validated against hidden test results to compute Pass@1. This process is repeated across all supported languages for direct performance comparison of the same problem.

Technical innovations

• Development of an automatic test conversion pipeline that transforms functional-format Python code generation problems into a unified, language-agnostic STDIN/STDOUT input-output format enabling consistent multi-language evaluation.
• Seamless integration of Multi-LCB with the continuously updated LiveCodeBench benchmark, preserving contamination controls and allowing automatic tracking of new tasks across twelve programming languages without manual rework.
• Comprehensive analysis revealing Python overfitting and substantial disparities in multilingual performance, highlighting previously unquantified gaps in LLM generalization beyond Python.
• Implementation of a secure, sandboxed, language-agnostic execution framework for automated evaluation of generated code across widely varying languages and runtime environments.

Datasets

• LiveCodeBench (LCB) code generation subset — 1,055+ problems — sourced from LeetCode, AtCoder, Codeforces with release date filtering, public on Hugging Face
• Multi-LCB extension — same 1,055+ problems converted into 12 languages including Python, C++, Java, Rust, Go, JavaScript, TypeScript, C#, Ruby, PHP, Kotlin, Scala

Baselines vs proposed

• Qwen3-235B-A22B-Thinking-2507: Python Pass@1 = 74.0% vs Multi-LCB average across languages = 64.0%
• GPT-OSS-120B* (Medium): Python Pass@1 = 71.1% vs Multi-LCB average = 67.8%
• OpenRsn-Nmt-32B*: Python Pass@1 = 64.4% vs average other languages <30%
• DeepSeek-R1-0528*: Python Pass@1 = 66.3% vs Multi-LCB average = 63.1%
• Qwen3-30B-A3B-Thk-2507*: Python Pass@1 = 64.0% vs Multi-LCB avg = 53.2%
• Reproduction of LCB Python leaderboard on Multi-LCB Python subset within ±3% on average

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.20517.

Fig 2: Top-10 models by Pass@1

Fig 3: Scatter of Python vs. Average Pass@1

Fig 4: Pass@1 distribution across 12 languages

Fig 5: Monthly Pass@1 trends averaged across all

Fig 6: presents the web interface of Multi-LCB, displaying a subset of tasks released between

Fig 7: Monthly distribution of Tasks by Difficulty.

Fig 8: Monthly distribution of LCB tasks by platform.

Fig 9: Monthly task distribution by I/O data dimensionality (LeetCode Functional format).

Limitations

• Benchmark limited to 12 programming languages excludes important languages like Swift, Haskell, R, Julia, limiting domain coverage.
• Tasks are restricted to competitive programming problems, which may not reflect complexities of real-world software engineering tasks such as API usage, debugging, or system integration.
• Strict adherence to STDIN/STDOUT I/O format may cause models to fail due to format parsing or output compliance rather than logical coding errors, confounding true language competence measurement.
• Evaluations exclude proprietary and closed-source state-of-the-art models, reducing generalizability to all available LLMs.
• Conversion from functional to STDIN/STDOUT format may unevenly affect task complexity or difficulty across languages, potentially biasing cross-language comparisons.
• Residual contamination may remain from similar problem patterns and templates even after date-based filtering, especially for popular languages like Python.
• Performance disparities may partly reflect varying ecosystem and tooling maturity (e.g., compilation complexity, language idioms) rather than model capability alone.

Open questions / follow-ons

• How can LLM training be improved to reduce Python overfitting and enhance robust code generation across diverse programming languages?
• What impact would the inclusion of more domain-specific and real-world software engineering tasks have on multilingual code generation benchmarking?
• How can the STDIN/STDOUT evaluation format be extended or adapted to better capture programs requiring multiple valid outputs or complex data structure manipulations?
• What techniques can detect and mitigate subtle contamination effects arising from similar problem patterns or solution templates across languages?

Why it matters for bot defense

From a bot-defense and CAPTCHA perspective, Multi-LCB highlights important challenges in evaluating multilingual code generation capabilities, analogous to assessing bots that may operate across various systems and input modalities. The approach of contamination-aware evaluation through release-date filtering and language-agnostic test conversion offers methodological parallels to designing CAPTCHAs resilient to training data leakage and cross-domain spoofing. Additionally, the observed language-specific performance disparities caution against over-reliance on single-language or modality benchmarks when evaluating adaptive adversaries. The Multi-LCB methodology and findings can inform rigorous contamination controls and cross-environment validation strategies for bot detection and behavior modeling. However, direct application requires careful alignment, since Multi-LCB focuses specifically on competitive programming and code generation rather than interactive human-computer challenges.

Cite

@article{arxiv2606_20517,
  title={ Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages },
  author={ Maria Ivanova and Pavel Zadorozhny and Rodion Levichev and Ivan Petrov and Adamenko Pavel and Ivan Lopatin and Alexey Kutalev and Dmitrii Babaev },
  journal={arXiv preprint arXiv:2606.20517},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.20517}
}

Read the full paper

• arXiv:2606.20517 (abstract)
• arXiv:2606.20517 (PDF)