CS546 Learning Notes: From Co-occurrence Matrix to Word2Vec, Connecting Negative Sampling and Language Models

1. Distributional meaning: Is it just about word frequency?

My initial question was: does distributional meaning simply refer to frequency of occurrence?

My current understanding is that it focuses on the contexts in which a word appears. Word frequency can be used to count these patterns, but looking only at how many times a word appears in total does not reveal its meaning.

Suppose cat appears 100 times and computer also appears 100 times—this does not mean the two words are similar. Even though cat and dog may have different total frequencies, they could both appear near feed, pet, fur. This distribution across contexts is what determines whether they are related.

The phrase "the company it keeps" in the lecture notes refers to observing which words frequently co-occur with a given word.

How to read a word-word co-occurrence matrix?

Each row represents a target word, each column represents a context word, and each cell contains the number of co-occurrences within a specified window.

So when a question asks "a dimension of a row vector represents what," it is asking what corresponds to a position. The answer is context word. A dimension corresponds to digital; the value 2 in that dimension represents the co-occurrence count.

It is important to clarify: two words having similar contexts does not require the two words themselves to appear next to each other frequently.

2. TF-IDF: count(t,d), df, and the final value

I initially found these symbols confusing, but they actually represent statistics over two different scopes:

For example, if apple appears 100 times in an article, it contributes 100 to count but only 1 to df.

The TF used in the lecture notes is log-transformed:

Higher occurrence frequency yields higher TF, but the logarithm prevents growth from being too fast. When a word does not appear, TF is set to 0, avoiding log(0).

IDF is exactly as I confirmed before: "log(total number of documents / number of documents containing this word)". A word appearing in all documents would have IDF of 0 under this unsmoothed definition.

Finally, multiply the two:

This gives the TF-IDF weight of the word in this document. A word that is frequent in the current article but rare in others typically does a better job of distinguishing that article.

The TF-IDF matrix is commonly arranged as "documents × terms," but can also be transposed to "terms × documents." Each cell stores a weight; what the rows and columns represent depends on the labels.

3. PPMI, cosine, and SVD: What does the matrix become?

A PPMI matrix can still be "target word × context word," just replacing co-occurrence counts with association scores:

The denominator $P(w)P(c)$ is the joint probability assuming independence. A ratio greater than 1 means actual co-occurrence is more frequent than under the independence assumption, so PMI is positive. Negative values are clipped to 0; word pairs without co-occurrence are typically also recorded as 0 in the PPMI table.

This is an example table. PPMI weighting itself does not reduce the number of rows or columns, nor does it necessarily solve the problem of unreliable statistics for low-frequency words.

Cosine similarity: Is closer to 1 better?

It compares the direction of two non-zero vectors, with values in $[-1,1]$: 1 for same direction, 0 for perpendicular, −1 for opposite direction. For non-negative co-occurrence counts or PPMI vectors, similarity lies in $[0,1]$; cosine is undefined for zero vectors.

When I asked if "closer to 1 is better," the more accurate understanding is: closer to 1 means the vectors are more similar. If the task is to find similar words, this is the signal we care about, but it does not mean we should push all pairwise similarities toward 1. Antonyms can also share many contexts and have high similarity; high similarity does not guarantee synonymy.

What does SVD compress into what?

Suppose a PPMI matrix has 10,000 target words and 10,000 context dimensions:

Truncated SVD retains the largest $k=100$ singular values and their corresponding directions:

We can take $U_k\Sigma_k$ as the word vector matrix, which becomes $10000\times100$. The number of words remains unchanged; each word goes from 10,000 dimensions to 100 dimensions, yielding a typically dense low-dimensional representation.

Originally each column corresponded to a context word; after compression, the columns are combined features, typically not directly nameable as eat or fruit. This is a low-rank approximation that discards some information and is not lossless compression.

Additionally, the vocabulary symbol $|V|$ denotes the size of a set, not an absolute value. For the corpus apple banana apple orange, $|V|=3$, with repeated words counted only once. The $V_k$ in the SVD formula is a decomposition matrix and is not the same as the vocabulary $V$.

4. Why dense representations might be better: car and automobile

The lecture notes say car and automobile are synonyms, but in sparse vectors they are two independent dimensions. I needed to work through this example concretely.

Suppose we only look at two context dimensions:

The cosine of the two vectors is 0. Although both contexts are discussing cars, the counts fall in different positions, and a direct comparison fails to recognize the association.

Dense representations can learn the relationships between these contexts, allowing similar usage patterns to provide similar information. The point is not that sparse representations cannot handle synonyms, but that independent count dimensions may underestimate such similarity.

5. Word2Vec has no manual labels—how does it train?

"No manual labels" and "no targets during training" are different things. Word2Vec constructs prediction tasks automatically from text, so it can be called self-supervised learning, and is often classified as unsupervised learning in a broad sense. When the lecture notes say "in a supervised fashion," it refers to having prediction targets, loss functions, and parameter updates.

For example, in we enjoy learning language models, with learning as center and window radius 1, positive samples are:

• (learning, enjoy)
• (learning, language)

I once selected (enjoy, language), but that actually pairs two context words together. In Skip-gram, each positive sample is a center word with one context word within the window. When training through the entire sentence, the center position moves; not every position is centered on learning.

6. The two sets of vectors are not one-hot and embedding

This is what I repeatedly clarified this time. Following the arrangement in the lecture notes:

It clicked for me here: $W_{k,:}$ is row $k$, all columns. The colon means take all; $W_{:,k}$ is column $k$.

Multiplying one-hot by $W$ is equivalent to table lookup, extracting the center word's vector. Programs typically index directly without actually performing one-hot matrix multiplication.

"Two sets of vectors" refers to the roles of center word and context word, both $d$-dimensional and both learned. For example, (eat, apples) uses the input vector of eat and the output vector of apples. In reverse, (apples, eat) uses the other two vectors.

For notational consistency later, I write individual word vectors as column vectors $v_t,v'_c\in\mathbb R^d$; the row retrieved from $W$ becomes $v_t$ after transposition.

What does the retrieved vector represent?

It is a set of features learned to predict context. You cannot directly name each dimension as "fruitiness" or "sweetness"; meanings are usually distributed across multiple dimensions.

It is not a probability, allows negative values, and does not require the sum to be 1. Training typically starts with random values and only forms useful structure after training.

7. Skip-gram Three Formulas, and the Vanishing Labels

First Calculate Probability for a Single Candidate Word

This is a dot product: multiply corresponding dimensions and sum them. It is the matching score adopted by the model, not cosine, and not a probability.

Why can dot product represent matching? Because training adjusts vectors so that the relative score of true word pairs increases. At random initialization, this score does not yet have reliable semantic meaning.

Complete softmax is:

The numerator is the current candidate word, and the denominator is the entire vocabulary, not just words in the window.

Then Combine the Targets in the Window

Here we assume that given the center word, each context target is conditionally independent. For example, I [eat] apples, this is $p(\text{I}\mid\text{eat})p(\text{apples}\mid\text{eat})$.

Window radius is something we set to select training samples; the model does not predict the radius. This is also not the probability of "apples appears at least once in the window." Regular Skip-gram does not explicitly distinguish left and right positions.

Finally Convert to Loss

For a true word pair:

I asked at the time: "Didn't you say cross-entropy was used? Why don't I see $y_j$?"

Complete cross-entropy is:

The label for the true word is 1, and for other candidate words is 0. All terms multiplied by 0 vanish, leaving only the negative log probability of the true word. So ground truth is already embodied in the selected $c$.

For example, if the predicted probability of the true word apples is 0.2, the loss is $-\ln0.2\approx1.61$; after increasing to 0.8, the loss is $-\ln0.8\approx0.22$.

For each center position and each actual context in the training text, sum the losses together, which gives the two layers of summation in the slide. The log converts products to sums, and the negative sign converts maximizing probability into minimizing loss.

I also corrected something I said: it is not "reduce the gradient," but rather use the gradient to update parameters and reduce the loss:

8. Negative Sampling: Not Shrinking the Softmax Denominator

This is where I most easily make mistakes. Initially I understood it as "only sample a few words, so softmax doesn't need to sum from 1 to the entire vocabulary." It is true that computation decreases, but the objective function also changes.

Complete softmax predicts context words across the entire vocabulary; negative sampling distinguishes between real data word pairs and random noise word pairs.

For example, a positive sample (eat, apples), paired with two negative samples (eat, computer), (eat, airplane). The positive sample has label 1, and negative samples have label 0.

Put together, for a positive sample $(t,c)$:

Because $1-\sigma(s)=\sigma(-s)$, the second term can also be written as $-\sum_i\log\sigma(-s(t,n_i))$.

Positive samples want sigmoid output close to 1; negative samples want it close to 0. Negative samples do not mean "these two words can never co-occur"; they are just training instances from noise sampling this time.

If you only use positive samples for this binary classification, the model can judge all word pairs as true. Negative samples give it grounds to distinguish. Complete softmax, on the other hand, already makes candidates compete through the denominator, without needing these extra negative samples.

k Negative Samples per Positive Sample: What Exactly Are We Counting?

In the slide, apricot has four context words, so there are 4 positive samples. The figure shows 8 negative samples, corresponding to 2 per positive sample.

The exercise changes $k$ to 3, so the number of negative samples is $4\times3=12$, for a total of 16 positive and negative combined. We count training word pair instances, not different center words, and do not require the sampled noise words to be distinct.

9. Negative Sampling's P and Alpha: Which Words to Sample?

Here $P$ only describes when sampling negative samples, what probability each word has of being selected, not the model's $p(w\mid t)$, nor the classification output of sigmoid.

The denominator sums the "count raised to a power" for the entire vocabulary, normalized into a probability distribution. This distribution can be pre-calculated from the corpus without needing to recalculate for each center word.

Taking $\alpha=3/4$ makes the distribution smoother. For example, if the original frequency ratio is $100:1$, the sampling probability ratio becomes:

I understand this smoothing as follows: the gap between high and low frequency narrows, low-frequency words get higher relative sampling probability than before, but high-frequency words are still more likely to be sampled.

It is not add-1. When count is 0, raising to a power still gives 0, and does not assign new probability to unseen words.

10. After Learning Word Vectors Well, How Do We Evaluate?

Intrinsic evaluation directly tests representations, such as calculating cosine for word pairs and comparing with human similarity ratings for ranking or correlation; you can also do synonym selection questions.

WordSim-353, SimLex-999, and SCWS in the slide are relevant evaluation datasets. The TOEFL example asks which word is closest to levied, with the correct answer being imposed; the model can select based on vector similarity and calculate accuracy.

Extrinsic evaluation puts vectors into downstream models to see actual task performance, such as machine translation, sentiment classification, or named entity recognition.

POS is Part of Speech. POS tagging assigns labels like NOUN, VERB to each word and can be a downstream task. In a book, book is a noun; in book a flight, book is a verb; context must be considered.

Good intrinsic evaluation does not guarantee good performance on every downstream task; final task performance also depends on the model and training method.

11. N-gram: Given the First N−1 Words, Calculate the Conditional Probability of the Next Word

Unigram does not look at context, bigram looks at the previous 1 word, trigram looks at the previous 2 words. This is not just predicting the $n$-th word of the entire sentence; the window slides backward.

For example, trigram can successively calculate $P(\text{apples}\mid\text{I,eat})$, $P(\text{every}\mid\text{eat,apples})$. Multiplying conditional probabilities at each position gives sequence probability, with boundary handling conventions needed at sentence start and end.

Traditional count-based model estimation:

Here we assume consistent counting scope, where nice's count corresponds to occurrences of contexts with subsequent tokens. The problem is: if a combination has not been seen, unsmoothed estimation gives 0, and the product of sequence probabilities becomes 0.

12. Add-1: Why Add 1 to the Numerator and Add the Entire Vocabulary Size to the Denominator?

$c$ is the preceding context. Each candidate word gets +1, so total count increases by $|V|$, not just 1.

Suppose the vocabulary has only apples, rice, bread, with original counts after eat as 3, 2, 0. After smoothing they become 4, 3, 1, and total count changes from 5 to 8, so bread's probability changes from 0 to $1/8$.

The total continuation count in the question is here the sum of occurrences of all follow-up words, not the number of distinct follow-up words. For example, 12 times eat apples and 8 times eat rice total 20.

Therefore if total count is 20, vocabulary size is 5, and target word count is 0:

Smoothing does not make all word probabilities larger. The sum is still 1; we allocate mass to unseen combinations while adjusting probabilities of seen combinations. The "unseen" here refers to combinations in the fixed vocabulary that haven't been seen, not automatically handling all out-of-vocabulary words.

13. Why Does Neural Language Model Automatically Do Smoothing?

In traditional count-based models, dog jump and cat jump are counted separately. If dog jump appears frequently, the model does not automatically increase cat jump's count just because dog and cat have similar meanings.

In neural language models, dog and cat have different but potentially similar vectors, processed through the same network. If the model learns a pattern from dog jump, it has a chance to transfer this pattern to cat jump, even if this combination never appeared in training data.

So the key is similar representations and parameter sharing in the network. This does not mean dog and cat share a single vector, nor does using embedding guarantee superiority over all n-grams.

This automatic smoothing here refers to this kind of generalization of statistical information, not secretly doing add-1. Embeddings can be pre-trained or learned together with the language model.

Also, n-gram describes context restriction, and neural models can also look at only the previous $n-1$ words. When comparing, we should say "count-based models" and "neural network-based models."

14. That Feed-Forward Network Diagram: Where Is the Fixed Window, and What Are W1/W2/W3?

The slide input is $w_{t-n+1},\ldots,w_{t-1}$, each word looks up a vector from matrix $C$, concatenated into $x$. If word vectors have $d$ dimensions, input is fixed at $(n-1)d$ dimensions. The window moves with position, but size remains fixed. Feed-forward is reflected in the computation flowing from input to hidden layer, then to output, with no recurrent hidden state passed across time steps.

$W^{(3)}$ is not a third layer stacked after $W^{(2)}$. It bypasses the hidden layer; its scores are added to those from the other path, and both are softmax'd together.

I even specifically asked "has such a network actually existed in history?" Yes, the slides correspond to Bengio et al.'s A Neural Probabilistic Language Model (2003). It jointly trains word representations and language models, so word embeddings didn't originate with Word2Vec.

15. Contextual embedding: Not just one extra context matrix

When I saw this page, my first thought was: is there an additional context matrix alongside the original embedding matrix?

The actual difference is that the final representation is computed dynamically based on the current input.

In static lookup, the representation of "bank" is fixed. A contextualized model first retrieves the initial token representation, then processes the visible context with a network to obtain the output vector at this position:

In "deposited money at the bank" and "beside the river bank," the initial token embedding for "bank" can be the same, yet the final representations will differ due to context.

So "each word still has only one vector" needs layered understanding: at the lookup stage, usually each token has a fixed row; at the network output stage, each occurrence can have its own representation. It's not that every possible meaning is pre-stored separately.

The slides list ELMo using BiLSTM, BERT using bidirectional Transformer encoder, and GPT using causal Transformer. Causal means only information from the current position and earlier can be used; future unseen words cannot modify the current representation.

Also don't confuse this with Word2Vec's output/context vector: those are static parameters too. "A vector as context words" and "a vector that changes with context" mean different things.

16. What category does GloVe belong to?

I ultimately mistook GloVe for a Skip-gram example. After organizing it's like this:

An intuitive GloVe example: ice and steam. "Solid" leans toward ice's context, "gas" toward steam's, while "water" relates to both. The ratio of co-occurrence probabilities helps capture this difference.

It's a different method from Word2Vec, but both fall under distributional semantics: learning meaning from the contextual distribution in language.

17. Key sentences I most need to remember this time

• Distributional meaning looks at context distribution, not just total word frequency.
• One dimension of co-occurrence vectors corresponds to context words; TF-IDF's count and df respectively count word occurrences and number of documents containing it.
• $|V|$ is vocabulary size; $W_{k,:}$ is the $k$-th row.
• Word2Vec's two sets of vectors are learned, not one-hot plus embedding.
• Dot product is a score; only after softmax do we get a probability distribution across the vocabulary.
• One-hot cross-entropy simplifies to only the negative log probability of the true target.
• Negative sampling switches to a binary classification objective, not shrinking the softmax denominator; $P_\alpha$ only controls how noise words are sampled.
• Add-1 adds one to each candidate word, thus adding $|V|$ to the denominator.
• Neural language models help unseen combinations generalize through similar representations and parameter sharing, but don't guarantee better results in all cases.
• Contextual embedding computes representations by context, not by adding another static table.

After reviewing to this point, I can better connect "lookup → score → probability → loss → update parameters." When seeing another formula next time, I'll first ask: is it doing representation, scoring, prediction, or sampling? Separating these things makes symbols less confusing.

Materials and corresponding slides

• Distributional semantics, sparse representations, and SVD: PDF pages 3–20.
• Word2Vec, two sets of vectors, negative sampling: pages 21–32.
• Word embedding evaluation: pages 42–43.
• N-gram, Add-1, neural language models: pages 45–53.
• Contextual embeddings: page 63; GloVe: pages 67–72.
• Historical sources: Bengio et al., A Neural Probabilistic Language Model, JMLR 2003.