TF-IDF — Hao Wu

TF-IDF stands for term frequency and inverse document frequency. It evaluates the importance of a word to a document in a corpus, which increases with the number of times it appears in the document but is offset by its frequency in the corpus. More precisely, for TF, we define

tf(term| document) = \frac{\mbox{the number of times "term" appears in "document"} }{\mbox{total number of words in "document"} }

For IDF, we define

idf(term) = 1 + \log(\frac{\mbox{the total number of documents} }{\mbox{the number of documents this term appears in} } )

Now suppose we have a query. Let $Q = \{t_1, t_2, ..., t_n \}$ the set of different term in the query. We convert $Q$ into a vector,

V(Q) = [tf(t_1 | Q)* idf(t_1), tf(t_2|Q) * idf(t_2),..., tf(t_n|Q) * idf(t_n)]

Similarily, for a document $D$ , we can get a vector $V(D)$ . Now the similarity between $Q$ and $D$ is given by

\frac{V(Q) \times V(D)}{||V(Q)|| \times ||V(D)||}