teLLMe Why (Ain't Nothing but a Jam): Exploratory Causal Analysis of Urban Driving Data
Source: arXiv:2607.15254 · Published 2026-07-16 · By Qiwei Li, Jorge Ortiz
TL;DR
This paper addresses the challenge of performing causal analysis on large-scale observational urban driving data derived from dashcam video. Such data often lack controlled interventions, making it difficult to answer causal questions like how weather or peak hours affect traffic density. The authors present teLLMe, a system that integrates causal structure learning (using the PC algorithm with domain knowledge and bootstrap stability checks), natural language query parsing via a schema-aware large language model, and treatment effect estimation through linear regression and DoWhy. By converting user questions into formal causal queries, automatically selecting backdoor adjustment sets from learned DAGs, and generating human-readable Causal Cards that summarize estimates, assumptions, and uncertainties, teLLMe helps analysts explore plausible causal hypotheses from complex, imbalanced event datasets. The system is demonstrated on a BDD-derived dashcam event table, surfacing plausible effects such as rainy weather reducing urban intersection traffic density and peak hours increasing highway traffic density.
Key findings
- Rainy versus clear weather reduces traffic density at urban intersections during peak hours by an estimated average treatment effect (ATE) of -0.036 (95% CI [-0.047, -0.024]) using 1,840 rainy and 7,681 clear windows.
- Peak-hour periods increase highway traffic density under clear weather by an ATE of 0.024 (95% CI [0.014, 0.034]) based on 4,268 peak-hour and 6,154 off-peak windows.
- Learned causal graphs via PC algorithm with domain constraints and bootstrap stability provide edge-level stability scores (0 to 1) to quantify confidence in causal directions.
- Automatic backdoor adjustment set selection using the learned DAG reduces bias compared to fixed adjustment sets, as ignoring DAG structure sometimes produces larger effects and narrower confidence intervals (potential under-adjustment).
- Analyses on imbalanced datasets produce more extreme effect estimates and greater uncertainty, highlighting the impact of stratified sampling and balancing choices.
- Natural language queries parsed by a schema-aware LLM into structured causal queries reliably prevent hallucinated variables and maintain data schema consistency.
- Causal Cards effectively summarize and visualize treatment effect estimates, adjustment sets, graph evidence, and assumptions, supporting transparent hypothesis exploration.
- Sensitivity checks across alternative adjustment sets and full versus balanced samples demonstrate teLLMe surfaces key modeling decisions impacting causal inference.
Threat model
The adversary is not adversarial but rather the analytic challenge posed by observational, confounded urban dashcam data where no experimental interventions exist. The system must produce causal estimates robust to data imbalance and confounding but cannot defend against maliciously crafted data or adversarial attacks.
Methodology — deep read
The threat model assumes an analyst seeking exploratory causal effects in complex, purely observational dashcam-derived urban driving data; there is no adversarial component. The inputs are structured event tables aggregated from raw dashcam annotations (from BDD100K), where each row corresponds to a fixed-length time window labeled with variables such as weather, traffic density, scene type, peak hour indicator, and temporal features like time of day and weekday/weekend. Missing data rows were dropped and extreme values clipped. To address class imbalance, stratified sampling was used to create balanced analysis subsets, with full datasets retained for sensitivity.
The core causal graph learning relies on the PC algorithm to infer a directed acyclic graph (DAG) over curated relevant variables (e.g., weather, peak hour, traffic density). The PC algorithm iteratively removes edges based on conditional independence tests and orients the graph edges using logical rules, constrained by domain knowledge that forbids implausible directions such as traffic density causing weather. Bootstrap resampling (B=20) is applied to quantify edge stability by counting frequency of each edge across bootstrap DAGs, yielding stability scores between 0 and 1.
On the user query side, natural language questions specifying causal effects of interest are parsed by a schema-aware large language model (LLM) into a formal causal query: treatment variable with treated/control values, outcome variable, and optional conditioning constraints (e.g., filtering by scene type). The parser validates resulting structured queries against schema types and allowed values to prevent hallucinated or invalid queries.
Effect estimation for a given query involves selecting a backdoor adjustment set from the learned DAG by including covariates that are parents of treatment or outcome but not descendants of treatment—approximating adjustment sets from causal theory. Filtering the event table accordingly, average treatment effects (ATE) are estimated with ordinary least squares linear regression (OLS) controlling for treatment and adjustment covariates. A second estimate using the DoWhy library replicates the effect under the same assumptions. Confidence intervals and standard errors are reported, and optional bootstrapping measures empirical effect stability.
Results are reported in a compact "Causal Card" presenting the treatment, outcome, adjustment set, ATE with 95% CI, sample sizes of treated and control, presence/stability of direct DAG edges, and a short natural language explanation to communicate assumptions and uncertainties. The entire workflow integrates offline preprocessing and causal discovery with online interactive querying.
To illustrate, for the query "Does rainy weather affect traffic density at urban intersections during peak hours?" the system sets treatment to weather with rainy vs clear, outcome to traffic density, conditions on peak hour and scene type, selects time of day and total objects as adjustment variables, fits OLS on the filtered balanced subset, and reports an effect with confidence intervals. Parallel estimates using DoWhy and sensitivity tests with imbalanced data and alternate adjustment sets confirm result robustness.
Reproducibility details such as exact hyperparameters or random seeds are not fully specified; code release status is not mentioned. The dataset used is BDD100K-derived but event tables are likely internal. The paper notes limitations around unobserved confounders, lack of temporal structure modeling, and no evaluation of Causal Cards’ interpretability by practitioners.
Technical innovations
- Integration of causal structure learning (PC algorithm with domain constraints and bootstrap stability) with natural language query parsing via a schema-aware LLM to generate structured causal queries grounded in dataset schema.
- Automated selection of backdoor adjustment sets from learned DAGs combined with flexible filtering for subpopulations specific to user queries.
- Use of a dual-effect estimation pipeline employing OLS regression and DoWhy to provide consistent causal effect estimates and quantify uncertainty.
- Introduction of modular, user-friendly Causal Cards that summarize causal query results, adjustment sets, graphical evidence, stability, and assumptions with succinct natural language explanations.
Datasets
- BDD100K-derived event table — large-scale dashcam video event data aggregated into fixed-length windows with support labels for weather, scene type, traffic density, and temporal features — proprietary internal subset balanced via stratified sampling
Baselines vs proposed
- Fixed adjustment set (ignoring DAG): sometimes yields larger effects and narrower confidence intervals compared to DAG-selected adjustment sets, indicating potential under-adjustment bias in fixed sets.
- Imbalanced dataset analysis: produces more extreme effect estimates and greater uncertainty than balanced subsets, demonstrating influence of sampling and data balancing.
Figures from the paper
Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.15254.
Fig 1: teLLMe has an offline phase that aggregates dashcam annotations into fixed-length windows, con-
Fig 2: Example Causal Card summarizing the query, effect estimate, adjustment set, DAG infor-
Limitations
- All causal inferences depend on assumptions encoded in the learned DAG and backdoor adjustment correctness; unobserved confounders like driver intent or road surface conditions not accounted for.
- The system assumes independence between event windows and does not model temporal or spatial autocorrelation common in traffic data.
- Limitations in measurement precision: traffic density is simple to compute, but finer safety outcomes like near-misses are not reliably extracted from video.
- No formal evaluation of how well Causal Cards support domain expert interpretation or decision making.
- Provenance and availability of processed event datasets are limited; exact reproducibility details (code, seeds) are not fully documented.
- Potential model misspecification if the PC algorithm or adjustment selection heuristics are inaccurate in representing true causal structure.
Open questions / follow-ons
- How to incorporate temporal and spatial dependencies in causal structure learning for traffic event sequences?
- What are better methods to handle unobserved confounding variables such as driver behavior or road surface conditions not captured in dashcam footage?
- How do practitioners interpret Causal Cards in real-world traffic planning contexts, and what UI designs best support informed decision making?
- Can the system integrate more sophisticated causal effect estimators (e.g., propensity score or doubly robust methods) and measure their performance on urban driving data?
Why it matters for bot defense
While this work does not directly address bot detection or CAPTCHA challenges, it exemplifies a rigorous approach to exploratory causal analysis on large, observational datasets with complex dependencies—similar to challenges faced in bot-behavior analysis. The methodology of combining causal structure learning with natural language interface, bootstrap stability checks, and transparent adjustment set selection could inspire bot-defense researchers to build interactive causal reasoning tools to understand how external factors impact bot behaviors across different environments. Causal Cards as interpretable summaries may also guide security analysts in explaining causal assumptions and uncertainty to stakeholders. Overall, the paper illustrates practical systems-level approaches for embedding causal inference into applied analytics pipelines, which could analogously improve accountability and clarity in CAPTCHA effectiveness studies or bot mitigation research.
Cite
@article{arxiv2607_15254,
title={ teLLMe Why (Ain't Nothing but a Jam): Exploratory Causal Analysis of Urban Driving Data },
author={ Qiwei Li and Jorge Ortiz },
journal={arXiv preprint arXiv:2607.15254},
year={ 2026 },
url={https://arxiv.org/abs/2607.15254}
}