Evaluating EEG-to-text models through noise-based performance analysis

• EEG-based speech decoding can be evaluated through noise-based performance analysis for real-world robustness ¹.
• Nature-published methodology supports rigor in EEG-to-text and speech prostheses applications ¹. ¹

Weekly enrichment (2026-07-20)

• The primary source is a peer-reviewed paper by Jo et al., “Evaluating EEG-to-text models through noise-based performance analysis,” published in Scientific Reports (2025) by a Kyung Hee University group, not a general-methodology note.^{2 3}
• The core method is a noise-baseline diagnostic: EEG inputs are replaced with random noise of matching dimensionality, and if a model performs the same on noise as on real EEG, it is memorizing linguistic patterns rather than decoding neural signals.^{2 4}
• The critique is shown to be model-agnostic, replicated across three pretrained transformer decoder families—BART, Pegasus, and T5—so the failure is not tied to one architecture.^{2 3}
• Evaluations were run on the ZuCo reading datasets (combining SR v1.0, NR v1.0, TSR v1.0, and NR v2.0 reading tasks), a widely used but demographically limited EEG-while-reading corpus.²
• Teacher-forcing during evaluation (feeding ground-truth tokens at each step) inflated scores by at least 3×, and the GitHub codebase reports the inflation exceeding 3×; without it, models collapse into repetitive generic phrases like “He was…”.^{3 4}
• Across noise manipulations (shuffle-time, similar-Gaussian, shuffle-channel, channel-wise Gaussian), performance stayed essentially unchanged, evidence that the models are insensitive to neural signal structure.²
• In some conditions BLEU-1 for noise inputs exceeded the original BART model by about 1–2%, which the authors attribute to checkpoint-specific variation rather than genuine neural decoding.²
• The paper proposes mandatory reporting standards: noise-baseline testing with specified signal-to-noise ratios, elimination of teacher-forcing from evaluation, bootstrapped significance testing, and confidence intervals/effect sizes.^{2 4}
• Recommended data-handling fixes include strict sentence-level train/validation/test splits with no overlap and full transparency of preprocessing (filter parameters, artifact removal, frequency bands, temporal windows).²
• Practical BCI implication: reported EEG-to-text “successes” may reflect pretrained language priors, so open-vocabulary noninvasive speech prostheses need noise-controlled benchmarks before clinical claims; the official codebase (khu-aims/EEG-To-Text) releases the evaluation pipeline.^{2 4}

Footnotes

1. https://news.google.com/rss/articles/CBMiX0FVX3lxTE9kQ0N3eG9jejY5aWJxSDJKeEJmRFlmZzh0cmtESzFYcDdJWTgyamJRM2tnTEJOSVBvUm44VEtRcVB3NTJpd1hPcWVCNk50QUtjVy1nMld3OFV5cE5PeWVv?oc=5 ↩ ↩² ↩³

2. https://www.nature.com/articles/s41598-025-29587-x ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹

3. https://khu.elsevierpure.com/en/publications/evaluating-eeg-to-text-models-through-noise-based-performance-ana/ ↩ ↩² ↩³

4. https://github.com/khu-aims/EEG-To-Text ↩ ↩² ↩³ ↩⁴