FaceMoE: Mixture of Experts for Low-Resolution Face Recognition

Source: arXiv:2606.32040 · Published 2026-06-30 · By Kartik Narayan, Vishal M. Patel

TL;DR

Low-resolution face recognition (LR-FR) presents significant challenges due to severe degradation in probe images (blur, occlusion, low contrast) and a domain gap between high-resolution gallery and low-resolution probe images. Traditional single-feature-encoder models struggle to generalize across these domains and suffer from catastrophic forgetting when fine-tuned on LR data. This paper introduces FaceMoE, a novel transformer-based architecture that integrates a Mixture of Experts (MoE) mechanism in the feed-forward network (FFN) layers to specialize feature extraction on different facial semantic regions. A top-k router dynamically assigns tokens to a subset of experts, enabling resolution-aware feature extraction and sparse activation which boosts model capacity without proportional compute increase and helps retain pretrained knowledge during LR fine-tuning. FaceMoE is trained with a combined face recognition loss, along with router z-loss and load balancing losses to ensure stable expert specialization and routing.

Extensive evaluations on eleven datasets spanning high-resolution, mixed-quality, and low-resolution benchmarks—including BRIAR, IJB-S, and TinyFace—demonstrate that FaceMoE significantly outperforms state-of-the-art methods. Notably, on the challenging BRIAR Protocol 3.1 dataset, FaceMoE achieves TAR@FAR of 42.36%, 61.47%, and 81.27% at 0.01%, 0.1%, and 1% FAR, respectively, surpassing prior best results by a considerable margin. FaceMoE also successfully mitigates catastrophic forgetting, showing minimal performance drop on high-resolution and mixed-quality datasets after fine-tuning on low-resolution data. The emergent dynamic routing and expert specialization to different semantic face regions are key factors behind its improved robustness and generalization.

Key findings

• FaceMoE achieves TAR@FAR of 42.36%, 61.47%, 81.27% at 0.01%, 0.1%, and 1% FAR on BRIAR Protocol 3.1, outperforming ProxyFusion and PETALface by at least 2.26%, 7.57%, and 5.84% respectively (Table 1).
• On the IJB-S dataset (Surveillance-to-Surveillance), FaceMoE attains 14.85% TPIR at 1% FPIR, and Rank-1 and Rank-5 retrieval accuracies of 44.81% and 56.12%, surpassing PETALface by 2.6%, 6.49%, and 4.62% respectively (Table 2).
• FaceMoE reaches 76.18%, 79.69%, and 81.75% Rank-1, 5, 10 accuracy on TinyFace, establishing new SOTA and showing minimal performance drop on HR and mixed-quality datasets compared to finetuned CosFace or ArcFace (Fig 3,4).
• The top-k router with N=3 experts and k=2 active experts per token facilitates sparse expert activation, increasing model capacity by 2.17× while only increasing FLOPs by 1.66×.
• Catastrophic forgetting observed in traditional finetuned models (e.g. CosFace, ArcFace) on low-resolution datasets is mitigated in FaceMoE due to modular expert design and sparse activation limiting weight updates (Figures 3 and 4).
• Specialization emerges where experts focus on distinct semantic facial regions (high-frequency edges, low-frequency smooth areas, landmarks), enabling resolution-aware feature extraction (Fig 2).
• The router z-loss and load balancing loss stabilize training and prevent expert collapse, leading to balanced expert utilization.
• Selective drift occurs during fine-tuning where some experts adapt to LR data and others preserve pretrained HR knowledge (Appendix 8.9).

Methodology — deep read

The authors address low-resolution face recognition challenges—primarily the domain gap between high-resolution galleries and low-resolution probes, and catastrophic forgetting during fine-tuning—using a Mixture of Experts (MoE) transformer architecture called FaceMoE.

Threat Model & Assumptions: The adversary context is not explicitly security-focused; the model assumes a domain gap where gallery images are HR and probes are LR, degraded by blur, occlusion, low contrast, pose, illumination, and expression variations. The test domain differs significantly from pretraining HR data.

Data: The model is pretrained on WebFace4M (4 million images, 205,990 identities). For low-resolution fine-tuning and evaluation, datasets used include BRIAR (550k images, 577 identities, with 86,958 gallery images and 5,435 probe clips in Protocol 3.1), TinyFace (169,403 LR images spanning 5,139 identities with a 7,804 image training subset), and IJB-S (398 videos, 202 identities). High-quality datasets for robustness evaluation include LFW, CFP-FP, CPLFW, AgeDB, CALFW, CFP-FF, and mixed-quality datasets IJB-B and IJB-C. Standard splits and protocols are followed.

Architecture: FaceMoE modifies transformer blocks by replacing the standard Feed-Forward Network (FFN) MLP layer with a Mixture of Experts layer composed of N=3 expert MLPs, each a 2-layer fully connected network with GELU activations. A top-k router module dynamically selects k=2 experts per token based on learned routing logits (linear projection from the token embedding). The router outputs softmax probabilities over only the top-k experts per token, enabling sparse activation and conditional specialization.

Training Regime: Pretrain FaceMoE on WebFace4M for 26 epochs with AdamW optimizer, batch size 128 per GPU, initial LR 10^-3, polynomial LR decay, and 1 warm-up epoch on eight NVIDIA A6000 GPUs. Fine-tuning is two-stage: linear probing then full fine-tuning on target LR datasets (TinyFace or BRIAR). Example: For TinyFace linear probe, 10 epochs with 2 warmup, LR 10^-3, batch 16; finetuning 40 epochs with 4 warmup, LR 10^-4, batch 8. For BRIAR, 20 epochs each stage, LR 10^-3 then 5×10^-6, batch sizes 64 then 8. Loss combines CosFace margin-based softmax face recognition loss, router z-loss (L2 penalty on routing logits), and load balancing loss to prevent expert collapse. λ1=10, λ2=10, λz=1, λb=1 to balance losses.

Evaluation Protocol: Models tested across 11 datasets including HR, mixed, and LR domains. Metrics include TAR@FAR on BRIAR, TPIR@FPIR and Rank retrieval on IJB-S, verification accuracy on HR datasets, and rank retrieval on TinyFace. Comparisons are against pretrained and finetuned baselines such as CosFace, ArcFace, PETALface, ProxyFusion. Ablations on expert number and top-k selection validate best config. Performance degradation after fine-tune is compared.

Reproducibility: Code is publicly released at https://github.com/Kartik-3004/FaceMoE. Dataset licensing as per original datasets, no closed datasets involved. Key hyperparameters and training schedules provided. However, frozen pretrained weights provided not explicitly mentioned.

Example Walkthrough: Given a batch of face image tokens (T×d), the model computes routing logits z_t per token via learned projection. For each token, top-k=2 experts are selected based on logits. These experts independently feed-forward the tokens through their two-layer MLP. Output is weighted average of expert outputs by softmax probabilities over top-k logits. The resulting token sequence is passed forward for classification with CosFace loss and auxiliary router regularization. During finetuning on LR data, weight updates mainly affect subset of experts assigned to LR features, preserving others to prevent catastrophic forgetting. Over epochs, experts specialize emergently to face semantic regions (e.g., edges, smooth areas, landmarks).

Technical innovations

• Integration of Mixture of Experts mechanism in transformer FFN layers to enable dynamic, resolution-aware feature extraction for low-resolution face recognition.
• Design of a top-k router that sparsely activates a subset of specialized experts per token, allowing conditional expert specialization and improving capacity-efficiency tradeoff.
• Use of router z-loss and load balancing losses to stabilize expert specialization and prevent expert collapse during MoE training.
• Demonstration that MoE architecture mitigates catastrophic forgetting during fine-tuning on low-resolution datasets through sparse, modular expert updates with selective drift.

Datasets

• WebFace4M — approx. 4 million images, 205,990 identities — public
• BRIAR Protocol 3.1 — 86,958 gallery images, 5,435 probe clips, 615 and 260 identities respectively — public
• TinyFace — 169,403 low-resolution images, 5,139 identities — public
• IJB-S — 398 videos, 202 identities — public
• LFW — standard high-quality benchmark — public
• CFP-FP, CPLFW, AgeDB, CALFW, CFP-FF — high-quality datasets — public
• IJB-B, IJB-C — mixed-quality datasets — public

Baselines vs proposed

• BRIAR Protocol 3.1, TAR@FAR 0.01%: ProxyFusion = 40.10% vs FaceMoE = 42.36%
• BRIAR Protocol 3.1, TAR@FAR 0.1%: ProxyFusion = 53.90% vs FaceMoE = 61.47%
• BRIAR Protocol 3.1, TAR@FAR 1%: PETALface = 75.43% vs FaceMoE = 81.27%
• IJB-S TPIR@FPIR 1%: PETALface = 12.25% vs FaceMoE = 14.85%
• IJB-S Rank-1 retrieval: PETALface = 38.32% vs FaceMoE = 44.81%
• TinyFace Rank-1: PETALface = 75.45% vs FaceMoE = 76.18%
• TinyFace Rank-5: PETALface = 79.05% vs FaceMoE = 79.69%
• TinyFace Rank-10: PETALface = 81.19% vs FaceMoE = 81.75%

Limitations

• The model evaluation is mostly conducted on existing public datasets; robustness to adversarial attacks or deliberate obfuscations is untested.
• While catastrophic forgetting is empirically mitigated, formal theoretical analysis of forgetting reduction via MoE sparse updates is not provided.
• The top-k router hyperparameters (N=3, k=2) are empirically chosen and may not generalize to all scenarios without tuning.
• The emergent semantic specialization of experts is demonstrated visually but not quantitatively evaluated for all datasets.
• The computational overhead and latency for real-time deployment in resource-constrained environments is not detailed.
• Limited discussion of how occlusion and extreme pose beyond low resolution affect routing and expert specialization.

Open questions / follow-ons

• How does FaceMoE perform under intentional adversarial perturbations targeting expert routing to degrade LR face recognition?
• Can the MoE expert routing be further optimized or adapted dynamically during inference to improve efficiency?
• What is the impact of increasing number of experts or varying top-k routing threshold on performance vs compute tradeoffs?
• How well does the architectural principle of emergent semantic specialization generalize to other fine-grained vision tasks under domain shift?

Why it matters for bot defense

FaceMoE introduces an MoE-based transformer architecture that effectively addresses the challenging domain gap between high-resolution gallery and low-resolution probe face images common in surveillance contexts. For bot-defense and CAPTCHA practitioners, the key takeaway is that dynamic expert routing and modular, sparse activation can improve feature robustness under heavy degradations, while mitigating catastrophic forgetting typical of domain adaptation. This suggests MoE architectures could enhance recognition reliability in low-quality input scenarios, potentially applicable in anti-bot face verification or attack detection where input image quality varies widely. The auxiliary router regularization techniques and dynamic semantic region specialization may inspire adaptive feature extraction mechanisms tailored to input quality or attack patterns. However, real-time efficiency and security under adversarial conditions remain open concerns for practical deployment.

Cite

@article{arxiv2606_32040,
  title={ FaceMoE: Mixture of Experts for Low-Resolution Face Recognition },
  author={ Kartik Narayan and Vishal M. Patel },
  journal={arXiv preprint arXiv:2606.32040},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.32040}
}

Read the full paper

• arXiv:2606.32040 (abstract)
• arXiv:2606.32040 (PDF)