Neural Noise

Accepting the 'bitter lesson' and embracing brain complexity in computational neuroscience

2 min read

  • The “bitter lesson” (that general methods leveraging computation and scale often outperform hand-designed features) is discussed in relation to brain modeling and neural data analysis.1
  • Embracing the brain’s complexity is proposed as a complementary perspective to pure scaling in computational neuroscience and methodology priorities.1 1

Weekly enrichment (2026-07-20)

  • The perspective “Accepting ‘the bitter lesson’ and embracing the brain’s complexity” was published in The Transmitter’s NeuroAI series on 26 March 2025 by Eva Dyer and Blake Richards.2
  • It applies Rich Sutton’s 2019 “bitter lesson”—that general methods scaling with computation (search and learning) beat hand-engineered domain knowledge in the long run—to neural data analysis.23
  • The authors argue neural data is a prime candidate for scale precisely because it is high-dimensional, stochastic and context-dependent, so bespoke reductionist models fail to generalize across tasks, individuals and species.2
  • They prescribe three ingredients: large generalist models over multimodal datasets (electrophysiology, imaging, behavior), high-performance computing infrastructure, and large high-quality datasets spanning species, regions and conditions, including naturalistic settings.2
  • Sutton’s original essay draws on 70 years of AI history (e.g., deep-search engines defeating world chess champion Kasparov in 1997 over human-knowledge approaches) and Moore’s-law-driven falling compute cost.34
  • Konrad Kording’s commentary “An elegy for simplicity in neural coding” agrees complexity has displaced simple single-neuron coding but warns AI models offer predictive power with little interpretable clarity, and that “distillation” to simple explanations rarely fits real data.5
  • A related Transmitter piece notes neural foundation models can generalize to predict new activity, motor output and sensory responses, and can even distinguish up to 11 excitatory cell types from activity alone.6
  • BCI implication: the scaling thesis underpins neural foundation models for decoding (e.g., speech decoding in people who cannot talk), motivating shared infrastructure and large cross-subject datasets over task-specific pipelines.26

Footnotes

  1. https://news.google.com/rss/articles/CBMipgFBVV95cUxNMW94ZWpEQkZQQ3ZjaGVzYUdaT3dtbVZjV1NKaDQxNTNKLXNXUmFTM3hVZlNJVHFPa0NHQXd2M2ZHdW9BQkRZZl9zOTVGUDVZYTh5NUdKUzFoeDRHTlhtYmN3VXY1WGVVWDhoZ3lwWG1Ya3hQZEtqdWZ2N0lFZnJqUzh0ODJVWmZvbDJoeVE5Sk41Sjdzckkzemt2WW5FTlI1WHNpY2Rn?oc=5 ↩ ↩2 ↩3

  2. https://www.thetransmitter.org/neuroai/accepting-the-bitter-lesson-and-embracing-the-brains-complexity/ ↩ ↩2 ↩3 ↩4 ↩5

  3. http://www.incompleteideas.net/IncIdeas/BitterLesson.html ↩ ↩2

  4. https://en.wikipedia.org/wiki/Bitter_lesson ↩

  5. https://kording.substack.com/p/an-elegy-for-simplicity-in-neural ↩

  6. https://www.thetransmitter.org/neuroai/why-neural-foundation-models-work-and-what-they-might-and-might-not-teach-us-about-the-brain/ ↩ ↩2

Graph View