Fretiq: Browser-Native Electric Guitar String Classification via Engineered Spectral Features and Held-Out Free-Play Evaluation
Source: arXiv:2607.18303 · Published 2026-07-17 · By Aadi Garg
TL;DR
This paper addresses the challenge of classifying which string produces a given pitch in monophonic electric guitar audio, a difficult problem due to pitch redundancy on the fretboard and subtle, often imperceptible timbral differences. Prior work achieved up to 0.90 F-measure using SVM and spectral envelope features but generally relies on offline or non-browser environments. Fretiq innovates by delivering a browser-native, real-time single-instrument, single-player electric guitar string classifier that operates solely from a single direct input USB-C audio stream, requiring no specialized hardware such as hexaphonic pickups or fretboard sensors. The core technical contribution is a 26-dimensional engineered feature vector combining frequency band energies, spectral statistics, and 13 Mel-Frequency Cepstral Coefficients (MFCCs), achieving 97.1% accuracy on a large shuffled validation set of 322,215 balanced frames. Fretiq also proposes a novel "Comparison Training" data collection technique that alternates playing confusable adjacent string pairs to reduce specific confusions, successfully decreasing the D3→A2 confusion by 44% but showing mixed results on others. Held-out free-play evaluation on 103,000 frames unseen during training yielded 87.8% accuracy, highlighting a generalization gap and pointing to temporal modeling as future work. The system runs fully in the browser and is engineered for training-inference parity using identical Python and TypeScript pipelines. Two critical implementation failure modes (feature dimensionality mismatch and model serialization issues) are documented to aid reproducibility.
Key findings
- Fretiq achieves 97.1% shuffled frame-level validation accuracy on 322,215 balanced frames using a 26-dimensional feature vector.
- Removing MFCC features drops accuracy from 97.1% to 92.2%, with largest losses on ambiguous strings D3 (-9.1 pp) and A2 (-5.1 pp).
- Comparison Training reduces D3→A2 frame-level confusion rate by 44%, but increases confusion on G3→D3 (+55%) and B3→E4 (+279%), showing pair-specific effects.
- Held-out free-play evaluation on 103,000 frames yields 87.8% overall accuracy, a 9.3 percentage point drop from controlled validation performance.
- Frame-level temporal leakage during train/validation split likely inflates validation accuracy.
- Dense neural network with engineered features outperforms 1D CNN on raw FFT bins on lower parameter count (final model: 26 input → Dense(128) → Dropout → Dense(32) → Dropout → Dense(6)).
- Feature extraction implemented identically in Python and TypeScript ensures training-inference parity, required to fix a critical 13-feature vs 26-feature mismatch bug.
- The browser-native pipeline executes inference within ~2 ms per frame on typical laptop hardware, validating real-time feasibility.
Threat model
Not applicable; the paper does not assume a security threat model or adversarial scenario but focuses on robust audio classification of guitar strings from monophonic audio in real-time browser conditions.
Methodology — deep read
The authors consider an adversary-free technical development scenario focused on monophonic electric guitar audio string classification from a single DI stream, with no external sensor or video data allowed. The system operates entirely in-browser.
The dataset consists of three recording sessions using a single electric guitar, one player, and a Boss Katana Gen 3 amplifier USB-C output at 44.1 kHz: (1) full neck coverage across six strings (60,294 frames), (2) open strings with varied attacks, and (3) a novel "Comparison Training" session targeting ambiguous adjacent string pairs played in deliberate alternation. Frames below a silence threshold are discarded, and the combined dataset totals 322,215 balanced frames (approx. 16.7% per string label). Data was shuffled before an 80/20 train-validation split; a separate held-out free-play session of about 15 minutes (103,000 frames) was recorded post training with manual string labels.
Audio capture uses Web Audio API with no noise/echo suppression. FFTs of size 2048 yield 1024 bins at ~21.5 Hz resolution. Pitch detection applies the McLeod Pitch Method (MPM) via the Pitchy library with variable clarity thresholds and RMS sustain gating, restricting pitch range to 70-1200 Hz. Detected MIDI pitch constrains candidate strings before classification.
The feature vector per frame is 26-dimensional: 8 psychoacoustic frequency band energies, 5 spectral statistics (centroid, rolloff, flatness, peak bin and magnitude), and 13 Mel-Frequency Cepstral Coefficients (MFCCs) computed with a custom mel filterbank and orthonormal DCT-II transform implemented identically in Python and TypeScript.
The neural network classifier is a fully connected dense architecture: Input(26) → Dense(128, ReLU) → Dropout(0.3) → Dense(32, ReLU) → Dropout(0.2) → Dense(6, softmax). Adam optimizer with batch size 32, early stopping with patience=6 on validation loss, and class weights balanced via sklearn are employed.
Comparison Training targets five canonical adjacent-string pitch pairs (open string vs. fifth-fret notes) to give the model balanced alternating examples designed to reduce confusion specifically between string pairs sharing identical pitches but subtle timbral differences.
Evaluation metrics are frame-level accuracy, confusion matrices for targeted pairs, and held-out free-play accuracy on unseen natural playing data. No statistical significance tests or multiple seeds reported; results are descriptive. The authors document two critical runtime failures: training-inference feature dimensionality mismatch (13 vs 26 features) and TensorFlow.js model serialization incompatibility, both resolved with engineering workarounds.
The full pipeline is publicly released with training code, feature extraction, evaluation scripts, and model weights. The system runs inference within ~2 ms per animation frame on a modern laptop in Chrome, supporting real-time operation.
Technical innovations
- A browser-native electric guitar string classification system running in real-time from a single USB-C DI audio stream without multi-microphone or fretboard sensor setups.
- A 26-dimensional engineered feature vector combining frequency band energies, spectral statistics, and 13 MFCCs computed identically in Python and TypeScript ensuring training-inference parity.
- Comparison Training data collection method that records alternating performances of confusable adjacent-string pitch pairs to reduce targeted misclassifications.
- Use of a dense feedforward neural network architecture over raw FFT due to physical unmotivated translation invariance in frequency axis and better performance with engineered features.
Datasets
- strings_clean_session1 — 60,294 frames — single guitar/player, USB-C DI, clean preset
- strings_open_session1 — size unspecified — open strings varied attack/dynamics
- strings_comparison_session1 — unspecified size — paired adjacent string alternation data
- held_out_free_play_session — ~103,000 frames — natural free-play, manual string labels
Baselines vs proposed
- Abeßer (2012) SVM + spectral envelope: F-measure = 0.90 vs Fretiq validation: 97.1% frame-level accuracy
- Geib et al. (2017) String-Inverse Frequency features: F1 = 0.72 vs Fretiq validation: 97.1%
- Kehling et al. (2014) tablature pipeline string estimation: string accuracy = 82% vs Fretiq validation: 97.1%
- Fretiq baseline (13 features): 92.2% accuracy vs full model (26 features): 97.1% accuracy
- Fretiq w/o Comparison Training (Condition B): 97.8% accuracy vs with Comparison (Condition C): 97.1% accuracy
- Held-out free-play accuracy: 87.8% (no direct comparable baselines)
Limitations
- Single instrument, single player, and single amplifier preset limit generalizability across gear and playing styles.
- Frame-level temporal leakage due to shuffling allows adjacent temporal frames from same sustained notes in both training and validation, inflating validation accuracy.
- No evaluation under different amplifier presets or distortion settings; model fails under Boss Katana distortion preset without retraining.
- Held-out free-play data reveals a 9.3 percentage point drop from validation, showing a generalization gap likely due to natural playing dynamics not captured during controlled recordings.
- Comparison Training is pair-specific; it reduces confusion for some pairs but increases it for others with no net accuracy benefit.
- Only single training run per condition; no reporting of statistical significance or confidence intervals.
- No adversarial or noise robustness evaluation.
- No temporal sequence models yet, which are planned to reduce the generalization gap and smooth predictions.
Open questions / follow-ons
- How well does the approach generalize across different guitars, amplifiers, pickups, strings, and players?
- Can temporal sequence models (e.g., LSTMs) close the 9.3% generalization gap from controlled validation to free-play accuracy?
- How to design more effective data augmentation or contrastive training techniques beyond pair-specific Comparison Training to improve confusion between ambiguous adjacent strings?
- What is the real-time robustness of Fretiq to noisy or distorted inputs, and how can model architectures or features be adapted?
Why it matters for bot defense
Fretiq demonstrates that highly accurate, real-time classification of subtle, acoustically confusable classes (guitar strings with identical pitch but different timbre) is achievable in a constrained browser environment from a single direct audio stream without specialized hardware. CAPTCHA and bot-defense systems that seek to classify or detect human behavioral signals from audio or other complex sensor inputs under similar constraints may find the engineered feature approach and rigorous held-out free-play evaluation methodology informative. The demonstrated mismatch between high shuffled validation accuracy and lower held-out natural performance highlights the importance of realistic data collection and deployment evaluation. The careful implementation of identical feature extraction in training and browser inference to ensure parity is a practical consideration for robust real-world ML deployment. The documented failure modes offer cautionary lessons about common pitfalls in browser-based ML pipelines. While the specific task differs from CAPTCHA, the principles of subtle-signal classification, single-stream inference, and robust model engineering remain highly relevant.
Cite
@article{arxiv2607_18303,
title={ Fretiq: Browser-Native Electric Guitar String Classification via Engineered Spectral Features and Held-Out Free-Play Evaluation },
author={ Aadi Garg },
journal={arXiv preprint arXiv:2607.18303},
year={ 2026 },
url={https://arxiv.org/abs/2607.18303}
}