How To Implement Bag-Of-Words In Python [2 Ways: scikit-learn & NLTK]

In this guide, we cover how to start with the bag-of-words technique in Python. We first cover what a bag-of-words approach is and provide an example. We then cover the advantages and disadvantages of the technique and provide code samples in scikit-learn and NLTK. Lastly, we suggest alternative algorithms that you could consider for your application.

What is a bag-of-words in Python?

In natural language processing, the bag-of-words model represents text data when modelling text with machine learning algorithms. The model is called a bag-of-words model because it represents each text document as a bag of words, regardless of the order of the words in the document.

A bag-of-words is like cutting all the different words out of a text and working with just the words.

In Python, you can implement a bag-of-words model by creating a vocabulary of all the unique words in your text data and then creating a numerical feature vector for each text document that represents the frequency of each word in the vocabulary.

A bag-of-words example

Here’s an example of a bag of words representation of a set of documents:

Suppose we have the following three documents:

Document 1: "I love dogs and cats"
Document 2: "I hate dogs but love cats"
Document 3: "Dogs are my favorite animal"

First, we create a vocabulary of all the unique words in the documents. In this case, the vocabulary is:

['I', 'love', 'dogs', 'and', 'cats', 'hate', 'but', 'are', 'my', 'favourite', 'animal']

Next, we create a matrix where each row represents a document, and each column represents a word in the vocabulary. The value in each matrix cell is the frequency of the corresponding word in the document.

So, the bag of words representation of the documents would be:

        I  love  dogs  and  cats  hate  but  are  my  favorite  animal
Doc 1   1    1     1    1    1     0     0    0   0      0        0
Doc 2   1    1     0    0    1     1     1    0   0      0        0
Doc 3   0    0     1    0    0     0     0    1   1      1        1 

This representation allows us to quantitatively compare the content of the documents based on the frequency of the words they contain.

For example, we can see that Document 1 has the word “love” as does Document 2.

Where is the bag-of-words technique used?

A bag of words is a simple and widely used representation of text data in many natural language processing (NLP) tasks. Here are some simple use cases for the bag of words:

1. Text classification: Bag of words can be used to represent the input text for a text classification model. The model can then learn to predict the class label based on the presence or absence of certain words in the input text.
2. Information retrieval: Bag of words can be used to represent the content of a document for information retrieval tasks, such as search engines or document recommendation systems.
3. Text similarity: Bag of words can be used to measure the similarity between two or more documents by comparing their word frequencies.
4. Latent Semantic Analysis (LSA): The bag of words can be used as input to LSA, a technique that discovers the underlying meaning of words by analyzing their relationships.
5. Topic modelling: Bag of words can be used as input to topic modelling algorithms, which discover the underlying topics in a collection of documents by analyzing their word frequencies.
6. Language modelling: Bag of words can be used to represent the input text for language modelling tasks, such as machine translation or text generation.

Advantages of a bag-of-words

There are several advantages of using the bag-of-words model for natural language processing tasks:

1. Simplicity: The bag-of-words model is a simple and intuitive approach to representing text data. It is easy to implement and understand, making it a good choice for many NLP tasks.
2. Sparsity: The bag-of-words model is a sparse representation of text data, meaning it only stores non-zero values for the words in a text document. This can be a significant advantage when working with large datasets, as it reduces the memory and computational requirements of the model.
3. Robustness: The bag-of-words model is robust to the order of words in a text document, which makes it resistant to the influence of word order on the meaning of the text. This can be useful when working with noisy or unstructured text data.
4. Ease of feature engineering: The bag-of-words model is a simple representation of text data, which makes it easy to extract additional features from the data. For example, you can easily create features that represent the length of a text document or the presence of specific words or word combinations.
5. Widely used: The bag-of-words model is a widely used and well-established approach to representing text data, and many NLP libraries and frameworks support it. This makes it easy to use and apply in various NLP tasks.

Disadvantages of a bag-of-words

There are also several disadvantages to using the bag-of-words model for natural language processing tasks:

1. Loss of context: The bag-of-words model ignores the order of words in a text document, which means that it loses the context and structure of the original text. This can be a significant disadvantage for tasks that require understanding the relationships between words or the meaning of the text.
2. Sensitivity to stop words: The bag-of-words model treats all words equally, which means that it may give too much weight to common words like “a” and “the” (called stop words), which do not carry much meaning. This can be a disadvantage if you want to identify the important words in a text document.
3. Lack of semantic information: The bag-of-words model must capture the meaning or semantics of the words in a text document. This can be a disadvantage for tasks that require an understanding of the underlying meaning of the text.
4. High dimensionality: The bag-of-words model can create a high-dimensional feature space, challenging some machine learning algorithms. This can make finding a good set of features for your model difficult and may require additional feature selection or dimensionality reduction techniques.
5. Limited ability to handle rare words: The bag-of-words model may not effectively represent rare or out-of-vocabulary words, as it only includes words in the vocabulary. This can be a disadvantage if you want to capture the full range of words in your text data.

Bag-of-words Python code

Scikit-Learn

In Python, you can implement a bag-of-words model by creating a vocabulary of all the unique words in your text data and then creating a numerical feature vector for each text document that represents the frequency of each word in the vocabulary. This can be done using the CountVectorizer class in the sklearn.feature_extraction.text module.

Here is an example of how you can use CountVectorizer to create a bag-of-words model in Python:

from sklearn.feature_extraction.text import CountVectorizer

# create the vocabulary
vectorizer = CountVectorizer()

# fit the vocabulary to the text data
vectorizer.fit(text_data)

# create the bag-of-words model
bow_model = vectorizer.transform(text_data)

# print the bag-of-words model
print(bow_model)

The bow_model variable is a sparse matrix that contains the frequency of each word in the vocabulary for each text document in the text_data list. You can access the vocabulary and the mapping from words to indices using the vocabulary_ attribute of the CountVectorizer object.

# print the vocabulary
print(vectorizer.vocabulary_)

# print the word-to-index mapping
print(vectorizer.vocabulary_['word'])

NLTK

Here is an example of how you can use the Natural Language Toolkit (NLTK) library to create a bag-of-words model in Python:

import nltk

# create the vocabulary
vocab = set()

# create the bag-of-words model
bow_model = []

for text in text_data:
    # create a dictionary to store the word counts
    word_counts = {}
    
    # tokenize the text
    tokens = nltk.word_tokenize(text)
    
    # update the vocabulary
    vocab.update(tokens)
    
    # count the occurrences of each word
    for word in tokens:
        if word in word_counts:
            word_counts[word] += 1
        else:
            word_counts[word] = 1
    
    # add the word counts to the bag-of-words model
    bow_model.append(word_counts)

The vocab variable is a set that contains the unique words in the text_data list. The bow_model variable is a list of dictionaries, where each dictionary represents the word counts for a single text document in the text_data list.

You can access the vocabulary and the word counts for a specific text document using the following code:

# print the vocabulary
print(vocab)

# print the word counts for the first text document
print(bow_model[0])

Alternatives to a bag-of-words in Python

There are several alternatives to the bag-of-words model for representing text data in natural language processing tasks:

1. N-grams: An n-gram is a contiguous sequence of n words in a text document. N-gram models capture the relationship between adjacent words in a text document, which can be helpful in tasks that require an understanding of word order or the meaning of the text.
2. Word embeddings: Word embeddings are dense, low-dimensional representations of words that capture the semantic relationships between words. Word embeddings can represent the meaning of words in a text document and capture the relationships between words.
3. Part-of-speech tags: Part-of-speech tags identify the part of speech (e.g., noun, verb, adjective) of each word in a text document. Part-of-speech tags can capture the syntactic structure of a text document and the relationships between words.
4. Named entity recognition: Named entity recognition is identifying and classifying named entities (e.g., people, organizations, locations) in a text document. Named entity recognition can extract structured information from unstructured text data and identify essential entities in a text document.
5. Syntactic parsing: Syntactic parsing is a process of analyzing the structure of a sentence and determining the relationships between the words in the sentence. Syntactic parsing can capture the syntactic structure of a text document and the relationships between words.

Closing thoughts

At Spot Intelligence, we frequently use the bag-of-words technique. It’s simple and fast and scales well. It’s often far more advantageous to spend more of your time on descent feature engineering, creating better datasets and getting domain knowledge into your model than using more complicated machine learning techniques. This is where the simpler pre-processing tools continue to outshine the more complicated techniques.

Do you use the bag-of-words technique for your projects? Let us know in the comments.