A Multi-task Mixture of Experts Framework for Malware Classification, Packing Detection, and Family Attribution

Source: arXiv:2606.30572 · Published 2026-06-29 · By Jithin S., Roshin Sleeba C., Anvin Mariya P. B., Asmitha K. A., Vinod P., Serena Nicolazzo et al.

TL;DR

This paper addresses the challenges of malware classification posed by heterogeneous malware data distributions, the presence of packed binaries, and diverse malware family characteristics. Traditional single-model malware detectors often underperform when generalizing across obfuscation techniques and rare or mutated malware samples. The authors propose a novel multi-task learning framework leveraging Mixture of Experts (MoE) architectures to simultaneously solve three key tasks: malware vs benign classification, packed vs unpacked detection, and malware family attribution. The framework explores three architectural variants—Homogeneous MoE, Heterogeneous MoE, and Multi-Gate MoE (MMoE)—evaluated on two complementary data representations: high-dimensional EMBER static features and raw 1D byte array representations of Portable Executable (PE) files.

The MMoE model, introducing task-specific gating mechanisms to independently route inputs to specialized experts, outperforms other architectures by effectively mitigating negative task transfer. Results demonstrate that MMoE achieves a combined detection rate of 0.9744 with a failure rate of 2.56%, showing superior robustness under mutation-induced adversarial distribution shifts applied to malware samples. The use of multiple expert networks coupled with adaptive gating enables improved specialization and generalization across heterogeneous malware tasks, making this approach promising for scalable and resilient malware detection in real-world complex environments.

Key findings

• Multi-Gate Mixture of Experts (MMoE) model achieves a combined detection rate of 0.9744 and a failure rate of 2.56%, outperforming Homogeneous and Heterogeneous MoE models.
• Task-specific gating in MMoE improves robustness against mutation-based adversarial attacks, maintaining stable performance under distribution shifts.
• EMBER feature-based models outperform raw 1D byte image representations in classification accuracy and computational stability.
• Negative Log-Likelihood loss with expert competition promotes better expert specialization in Heterogeneous MoE compared to Mean Squared Error in Homogeneous MoE.
• Using task-specific gating mitigates negative transfer between malware family classification and packing detection tasks, which rely on different feature cues.
• Latent space reconstruction through gated weighted decoders regularizes expert representations and improves model stability.
• Different expert architectures (residual, hierarchical, bottleneck) in Heterogeneous MoE provide complementary feature abstractions but still underperform MMoE.
• Threshold calibration of output logits for malware-vs-benign classification maximizes F1-score, adapting decision boundaries per random seed variability.

Threat model

The adversary is assumed to produce malware that may be packed or mutated using metamorphic techniques to evade static detection. The attacker cannot compromise the training process or access internal model parameters but aims to cause misclassification through distribution shifts in input features. The model defends against these evasion attempts by enabling expert specialization and task-aware routing, though more advanced adversarial attacks beyond mutation-based transformations are not considered.

Methodology — deep read

1. Threat Model & Assumptions: The adversary is assumed to generate malware binaries potentially employing obfuscation techniques including packing and mutation-based metamorphic transformations to evade detection. The model is designed to classify PE files statically, without dynamic execution. The attacker cannot alter the fundamental PE structure beyond the assessed mutation variations tested; advanced evasions beyond these are not evaluated.

2. Data: The study uses malware and benign samples represented by two modalities: (a) EMBER feature vectors extracted via PEFeatureExtractor library, producing 2381-dimensional feature vectors per sample encoding metadata, headers, sections, imports, byte histograms, and entropy; (b) raw 1D byte arrays truncated or padded to fixed lengths (1024, 4096, 16384 bytes) reshaped as one-dimensional grayscale images. Mutation-based augmentation is applied to test distributional shifts. The exact dataset size and source are not fully detailed, but the malware family distributions are visualized. Data preprocessing includes log(1+x) transform and min-max normalization.

3. Architecture/Algorithm: Three Mixture of Experts architectures are proposed:

   • Homogeneous MoE (Homo-MoE): three identical deep neural network experts share architecture, gated by a single gating network producing a probability distribution over experts; experts output latent vectors regularized by a decoder reconstruction loss; Expected Squared Error loss used.
   • Heterogeneous MoE (Hetero-MoE): three experts with distinct architectures including residual connections, hierarchical shrinking layers, and bottleneck compression; gating remains shared; Negative Log-Likelihood loss promotes competitive expert specialization.
   • Multi-Gate MoE (MMoE): shares three expert networks but replaces unified gating with task-specific gating networks (one per each task: malware family, packer detection, malware vs benign), each independently weighting experts; task-specific towers produce final predictions optimized with cross-entropy loss per task.

4. Training Regime: Models are trained jointly over the three tasks with combined losses including reconstruction regularization weighted by coefficient α (exact hyperparameter values unclear). Threshold calibration for malware/benign final output uses grid search on F1-score. Training parameters such as epoch counts, batch size, optimizer type, hardware, and random seeds are not explicitly detailed in the truncated text.

5. Evaluation Protocol: Performance metrics include combined detection rate (accuracy-like metric), failure rate, and macro F1-scores for multi-task classification. Models are tested under standard conditions and under adversarial data shifts induced by mutation-based augmentation. Architectural variants and input representations are compared head-to-head. Visualizations including t-SNE are used for latent space analysis.

6. Reproducibility: Code and pretrained weights release status is not stated. Dataset source is partially described (EMBER features, Portable Executable binaries) but exact public availability and mutation protocols are not fully documented.

Concrete Example: For a single PE sample, extraction produces a 2381-dimensional EMBER vector, which after log and min-max normalization is input to the three experts. The gating network(s) assign weights per expert per task, and the experts produce latent embeddings reconstructed via decoders to regularize latent space. Each task tower then outputs predictions—e.g., malware family class probabilities, packing binary decision, malware/benign classification. Backpropagation optimizes losses jointly with gating mechanisms dynamically adapting expert contributions task-wise, enabling specialization and mitigating interference.

Technical innovations

• Application of Multi-Gate Mixture of Experts (MMoE) with task-specific gating to malware multi-task classification, enabling independent expert weighting per task and reducing negative transfer.
• Integration of two distinct input modalities (high-dimensional EMBER features and raw 1D byte arrays) evaluated under the MoE framework.
• Use of latent space reconstruction decoders weighted by gating outputs to regularize expert latent representations and stabilize training.
• Comprehensive evaluation of MoE loss functions (Expected Squared Error, Negative Log-Likelihood) showing impact on expert specialization in malware detection.
• Systematic testing of model robustness under mutation-based adversarial distribution shifts simulating real-world malware evasion.

Datasets

• EMBER — 2381-dimensional static feature vectors extracted from Portable Executable files — derived from EMBER feature extraction tools and dataset
• Malware binaries represented as raw 1D byte arrays truncated/padded to lengths 1024, 4096, or 16384 bytes — source not explicitly specified
• Mutated malware samples generated via mutation-based metamorphic transformations for adversarial evaluation — mutation methods not fully detailed

Baselines vs proposed

• Homogeneous MoE: combined detection rate = lower than 0.9744 vs proposed MMoE: 0.9744 with failure rate 2.56%
• Heterogeneous MoE: improved over Homogeneous but underperforms MMoE, exact numbers not specified
• EMBER-based models outperform raw 1D byte image input models in classification accuracy and stability (Fig. 5 comparison)
• MMoE task-specific gating shows better robustness under mutated sample distribution shifts compared to unified gating MoE (Fig. 6 visualization)
• Loss functions comparison: Negative Log-Likelihood loss improves expert specialization over Mean Squared Error loss in heterogeneous expert setting

Figures from the paper

Figures are reproduced from the source paper for academic discussion. Original copyright: the paper authors. See arXiv:2606.30572.

Fig 2: The MoE framework for Windows malware classification

Fig 3: Architecture of the proposed Multi-Gate Mixture of Experts (MMoE) framework for multi-task malware analysis

Fig 7: Distribution of malware samples across different malware families (excluding benign samples).

Fig 8: The t-SNE projections on the experiments which gave the best results across MMoE, HeteroMoE, and HomoMoE

Fig 5 (page 19).

Fig 6 (page 19).

Fig 9: Comparison of EMBER feature representations and 1D image representations (1024-byte length) across Homogeneous

Fig 10: Comparison of Mixture of Experts (MoE) architectures under different evaluation settings using EMBER feature

Limitations

• Exact dataset size, composition, and public availability are not fully detailed, limiting reproducibility and comparison.
• Training details such as epochs, batch sizes, optimizer hyperparameters, and hardware specifications are not explicitly provided.
• Limited evaluation beyond mutation-based adversarial transformations; no testing against other evasion methods or dynamic malware behavior.
• The approach focuses on static analysis; dynamic or hybrid malware analysis scenarios remain unexplored.
• No ablation on number of experts beyond three or scalability to larger expert pools reported.
• Unclear if models generalize well on completely unseen malware families or benign distributions outside the test splits.

Open questions / follow-ons

• How does increasing the number of experts or incorporating dynamic expert expansion affect multi-task malware detection performance and robustness?
• Can the MMoE framework be extended to integrate dynamic behavioral features or other modalities beyond static PE features and raw bytes?
• What are the model's limits under stronger adversarial attacks such as adversarial perturbations crafted specifically against gating networks?
• How does this approach generalize in real-world deployment with continuously evolving malware families and novel packing methods?

Why it matters for bot defense

For bot-defense and CAPTCHA practitioners, this work illustrates the utility of multi-task learning with task-specific gating and expert specialization to manage heterogeneous, evolving input distributions—concepts directly applicable to detection of automated abuse across multiple correlated tasks (e.g., bot/non-bot detection, attack type classification, session risk attribution).

The multi-expert MoE paradigm helps mitigate negative transfer when training models jointly on diverse objectives, and the task-aware gating mechanism improves robustness under distribution shifts induced by adversarial manipulation—paralleling challenges in adversarial bot traffic and CAPTCHA attacks. The methodology of combining structured static features with raw low-level inputs also suggests multimodal approaches for abuse detection. However, the focus on malware static analysis means adaptation to network or behavioral signals would require further investigation. Still, the core architectural principles and robustness evaluation provide valuable design patterns for bot and CAPTCHA defense systems facing heterogeneous and adaptive adversaries.

Cite

@article{arxiv2606_30572,
  title={ A Multi-task Mixture of Experts Framework for Malware Classification, Packing Detection, and Family Attribution },
  author={ Jithin S. and Roshin Sleeba C. and Anvin Mariya P. B. and Asmitha K. A. and Vinod P. and Serena Nicolazzo and Antonino Nocera },
  journal={arXiv preprint arXiv:2606.30572},
  year={ 2026 },
  url={https://arxiv.org/abs/2606.30572}
}

Read the full paper

• arXiv:2606.30572 (abstract)
• arXiv:2606.30572 (PDF)