Skip to content
AI Lehel Briefing

Independent signal, primary sources

The important moves in artificial intelligence.

Models, research, coding tools, open source, infrastructure, and major product releases—ordered by publication time.

Chronological feed

Latest stories

2437 stories
Models & research Berkeley AI Research

What exactly does word2vec learn?

What exactly does word2vec learn, and how? Answering this question amounts to understanding representation learning in a minimal yet interesting language modeling task. Despite the fact that word2vec is a well-known precursor to modern language models, for many years, researchers lacked a quantitative and predictive theory describing its learning process. In our new paper, we finally provide such a theory. We prove that there are realistic, practical regimes in which the learning problem reduces to unweighted least-squares matrix factorization. We solve the gradient flow dynamics in closed f…