Toward Robust and 3D-Aware RGB-NIR Imaging in the Dark

Source: arXiv:2607.29684 · Published 2026-07-31 · By Muyao Niu, Mingze Ma, Yifan Zhan, Qingtian Zhu, Zhihang Zhong, Wei Guo et al.

TL;DR

This paper addresses the challenging problem of robust low-light image enhancement by leveraging complementary Near-Infrared (NIR) images fused with extremely noisy RGB inputs. Existing RGB-NIR fusion methods often rely on carefully curated triplets of noisy RGB, NIR, and clean RGB images, limiting their robustness and generalization to varying noise levels and domains. The key innovation here is the introduction of a 3D-aware neural implicit fusion model that operates within a volume rendering framework, allowing multi-view RGB and NIR observations to be fused directly in 3D space without needing any clean RGB supervision. Through novel architecture redesign including a NIR-conditioned positional encoding and a Color Code MLP module for resolving NIR-to-RGB ambiguities, the model demonstrates superior denoising and detail recovery on both synthetic and real-world datasets. Extensive experiments show consistent improvements over state-of-the-art single- and multi-view denoising and RGB-NIR fusion baselines across multiple noise levels and imaging conditions, establishing a robust and practical solution for RGB-NIR dark imaging scenarios.

Key findings

• The proposed 3D-aware fusion method achieves higher SSIM (up to 0.75+), PSNR (up to ~19.7 dB), and lower LPIPS (~0.23) than all compared methods on a synthetic dataset across five noise levels s ∈ {1/10, 1/25, 1/50, 1/100, 1/200} (Tab. 1).
• Without requiring clean RGB supervision, the model generalizes robustly across extreme noise conditions, outperforming both supervised (SANet, NAID) and unsupervised (NVEU) RGB-NIR fusion methods.
• Ablation shows that removing the NIR-based positional encoding introduces checkerboard artifacts and degrades SSIM by up to 0.01 and increases LPIPS by ~0.07 at high noise levels (Tab. 2, Fig. 11).
• Excluding the Color Code MLP causes blending artifacts and reduces the model's ability to disambiguate multiple RGB colors corresponding to the same NIR value, worsening LPIPS and qualitative quality.
• In real-world low-light multi-view captures, the model outperforms SOTA methods in non-reference perceptual image quality metrics PI, MUSIQ, and MANIQA, as well as human subjective scores (Tab. 4, 5).
• The method works well in RAW space as well as sRGB, demonstrating generality beyond standard processed image domains (Tab. 7).
• 2D RGB-NIR fusion followed by NeRF reconstruction performs significantly worse than the integrated 3D-aware fusion, highlighting the importance of joint 3D modeling.
• Training converges in about 3 hours on an NVIDIA RTX 4090 GPU with 9GB VRAM, showing feasibility for practical adoption.

Methodology — deep read

1. Threat Model and Assumptions: The adversary here is not explicitly defined as a security threat; instead, the study assumes challenging imaging conditions in extremely low-light environments producing noisy RGB inputs paired with NIR images. The model assumes multi-view captures with known camera poses for RGB and NIR modalities, and no clean RGB supervision available during training.

2. Data: Synthetic dataset is produced using Mitsuba3 renderer with 5 scenes, each having 40 camera views (960x1280 resolution) with paired RGB and 850 nm NIR images. Noise is simulated by scaling RGB intensities (scale factors s = 1/10 to 1/200) and adding physics-based shot and read noise. 5 views per scene reserved for testing; 35 views for training. Real-world data captured from 4 scenes with a JAI FS-3200T10GE-NNC camera, 49 views each at 768x1024, illuminated by 850 nm NIR LED. Camera poses calibrated by COLMAP. 5 views held for testing, 44 for training.

3. Architecture and Algorithms: The method builds on volume rendering and neural implicit representations (Neural Radiance Fields, NeRF) modified for noisy RGB and NIR fusion.

• Two MLPs: an NIR branch predicts volume density σ exploiting robust structural cues from NIR.
• RGB MLP integrates information from noisy RGB and the NIR output.
• Gradient stopping applied to prevent noisy RGB from degrading NIR branch training.

Key novel components:

• NIR-conditioned positional encoding (PE): Instead of applying PE to 3D coordinates and viewing direction (which causes noise-induced checkerboard artifacts), PE is applied to the NIR feature vector, which has a smoother frequency distribution aligned with clean RGB.
• Color Code MLP: A discrete latent module using Gumbel-Softmax to model the ambiguous mapping from a single NIR value to multiple possible RGB colors. Produces a one-hot color code input to RGB MLP.

4. Training Regime:

• Joint optimization of NIR and RGB MLPs by minimizing L2 photometric loss on rendered NIR and RGB outputs versus noisy RGB and NIR observations.
• Exposure amplification hyperparameter used on the noisy RGB term.
• Adam optimizer with learning rate 5×10^-4, β1=0.9 and β2=0.999, trained for 200,000 iterations.
• Training runs approx. 3 hours on a single NVIDIA RTX 4090 GPU with 9GB memory.

5. Evaluation Protocol:

• Metrics: SSIM, PSNR, LPIPS on synthetic data; PI, MUSIQ, MANIQA, and human subjective evaluation (HSE) scores on real data.
• Baselines include supervised RGB denoisers (Restormer), RGB-NIR fusion methods (SANet, NAID, ScaleMap, NVEU), multi-view RAW-based methods (RawNeRF, LLNeRF).
• Ablations remove NIR positional encoding and Color Code MLP independently.
• Additional comparisons with a two-stage pipeline: 2D fusion followed by NeRF.
• Cross-scene generalization tested with varied noise levels.

6. Reproducibility:

• Code and data released publicly at https://github.com/MyNiuuu/3DarkFusion.
• Synthetic dataset details and real-captured data provided.
• Exact training settings and architecture detailed.

Concrete Example: For a scene under extreme low-light (s=1/200), noisy RGB and NIR images from multiple camera views are fed into the model. The NIR MLP predicts density and structural features, encoded with positional encoding, which are fed to the RGB MLP along with the noisy RGB inputs and color codes predicted by the Color Code MLP. Volume rendering integrates output colors along rays to synthesize clean RGB views. The model is trained end-to-end using noisy RGB and NIR targets without ever accessing clean RGB images. Ablation shows removal of the NIR positional encoding causes strong checkerboard noise, whereas excluding Color Code MLP creates color blending artifacts. With all components, the reconstructed RGB images have significantly improved SSIM (0.70+), PSNR (~19.7 dB), and visually sharper textures compared to baselines.

Technical innovations

• Introduction of a novel 3D-aware implicit neural fusion architecture that jointly models noisy RGB and NIR multi-view observations in 3D space without clean RGB supervision.
• Proposal of a NIR-modulated positional encoding technique that applies sinusoidal encoding to the NIR features rather than to spatial coordinates, mitigating noise-induced checkerboard artifacts in low-light RGB-NIR fusion.
• Development of a Color Code MLP leveraging Gumbel-Softmax to probabilistically resolve the ill-posed one-to-many mapping from NIR intensity to RGB color distributions, improving color fidelity in reconstruction.
• Demonstration that integrated 3D neural fusion outperforms cascaded 2D RGB-NIR fusion followed by NeRF reconstruction, underscoring the benefit of joint multi-view 3D modeling over decoupled processing steps.

Datasets

• Synthetic RGB-NIR multi-view dataset — 5 scenes × 40 views (~200 total) — generated via Mitsuba3 renderer with noise simulation at multiple exposure scales.
• Real-world multi-view RGB-NIR dataset — 4 scenes × 49 views — captured with JAI FS-3200T10GE-NNC camera and 850 nm NIR illumination; camera poses calibrated with COLMAP.

Baselines vs proposed

• Restormer: SSIM=0.65~0.82 across noise levels vs Proposed: SSIM=0.70~0.75 (synthetic, Tab. 1)
• ScaleMap: PSNR~13.9~17.5 dB vs Proposed: PSNR~19.5~19.7 dB (synthetic)
• NVEU: LPIPS=0.36~0.50 vs Proposed: LPIPS ~0.23~0.27 (synthetic)
• SANet: SSIM=0.49~0.75 vs Proposed: SSIM=0.70~0.75 (synthetic)
• NAID: PSNR=13.6~16.1 dB vs Proposed: PSNR=19.5~19.7 dB (synthetic)
• RawNeRF: SSIM=0.57~0.70 vs Proposed: SSIM=0.70~0.75 (synthetic)
• LLNeRF: PSNR=9-17 dB vs Proposed: PSNR=19.5-19.7 dB (synthetic)
• 2D Fusion + NeRF combinations (e.g., NVEU + NeRF): SSIM ~0.36-0.53 vs Proposed: SSIM ~0.70-0.75 (Tab. 3)
• Real-world PI score: best 5.415 vs others ranging 4.8 to 9.7 (Tab. 5)
• Real-world MUSIQ: 56.51 vs competitors max 36.0 (Tab. 5)
• RAW domain: Proposed MUSIQ 55.7 vs NVEU 46.7, NAID 34.7 (Tab. 7)

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2607.29684.

Fig 1: Top: Given extremely noisy multi-view RGB observations in darkness, the proposed method recovers the clean RGB

Fig 2: Architecture illustrations of (a) parallel

Fig 3: Visual comparison results. From left to right: NIR observation,

Fig 4: Results of using vanilla NeRF MLP for our model.

Fig 7: (a) Applying P.E. to x and d in the RGB MLP intro-

Fig 5: Illustration of the final architec-

Fig 6: Visual comparison results. From left to right: NIR observation, (a) result

Fig 8: Clean RGB, NIR, and noisy RGB in frequency do-

Limitations

• Model is restricted to static scenes; cannot handle dynamic or moving objects, limiting use-cases with motion.
• Precise multi-view camera poses for both RGB and NIR images are required; unaligned free-capture scenarios are not addressed.
• No adversarial robustness evaluation or security analysis presented; focus is purely on imaging enhancement.
• Heavy computational and memory requirements could limit deployment on resource-constrained devices.
• Evaluation on wider ranges of real-world lighting conditions, sensor types, or camera setups is not extensively explored.
• No analysis of temporal consistency across video sequences, which is important for real-time low-light enhancement.

Open questions / follow-ons

• How can the model be extended to handle dynamic scenes or moving objects where multi-view consistency is violated?
• Can the requirement for aligned camera poses between RGB and NIR modalities be relaxed to enable more flexible capture setups?
• How to incorporate temporal information or video sequences to improve low-light RGB-NIR fusion robustness over time?
• What are the trade-offs in computational cost when scaling the model for higher resolution or real-time applications?

Why it matters for bot defense

Although focused on low-light image enhancement rather than adversarial bot defense, this work presents a novel implicit neural fusion technique capable of robustly denoising extremely noisy multi-modal multi-view inputs without clean supervision. For CAPTCHA or bot-defense engineers, this approach exemplifies how multi-spectral data fusion and 3D-aware neural representations can improve signal recovery under challenging noise, which could inspire defenses against image-based spoofing or automated recognition attacks in low-visibility conditions. Additionally, the paper’s insights into how to suppress noise-induced training pathologies (e.g., via modality-conditioned positional encoding) could inform design choices for CAPTCHA generation or recognition systems resilient to noisy or adversarial inputs. However, direct application requires adaptation to security scenarios and adversarial threat models, as the current work does not evaluate robustness against deliberate manipulation or attack.

Cite

@article{arxiv2607_29684,
  title={ Toward Robust and 3D-Aware RGB-NIR Imaging in the Dark },
  author={ Muyao Niu and Mingze Ma and Yifan Zhan and Qingtian Zhu and Zhihang Zhong and Wei Guo and Chang Wen Chen and Yinqiang Zheng },
  journal={arXiv preprint arXiv:2607.29684},
  year={ 2026 },
  url={https://arxiv.org/abs/2607.29684}
}

Read the full paper

• arXiv:2607.29684 (abstract)
• arXiv:2607.29684 (PDF)