Top 5 Ways To Implement NLP Text Tokenization Techniques In Python With NLTK, SpaCy, TextBlob, Gensim & Keras

Tokenization is a process in natural language processing (NLP) where a piece of text is split into smaller units called tokens. This is important for a lot of NLP tasks because it lets the model work with single words or symbols instead of the whole text.

Tokenizing text can be done in a number of ways, depending on the task at hand and the type of text being processed. For example, in sentiment analysis, a common method is to split the text into individual words, known as word tokenization. This allows the model to analyse the sentiment of each word and make a prediction about the overall sentiment of the text.

Another type of tokenization is sentence tokenization, which splits the text into individual sentences. This can help with tasks like summarization, where the model needs to know how the text is put together in order to make a short summary.

For summarization, sentence tokenization instead of word tokenization is used.

In addition to word and sentence tokenization, other types of tokens can be extracted from text. For example, n-grams are groups of n-consecutive words or symbols, that can be used to capture the context of a word in a sentence. Part-of-speech tagging is another common NLP task, where each word in a sentence is labelled with its part of speech (e.g., noun, verb, adjective, etc.).

Overall, tokenization is a crucial step in many NLP tasks, as it allows the model to work with individual language units instead of the entire text. This makes it possible to analyse the text, figure out what it means, and do other NLP tasks.

What are the different ways of NLP tokenization in text?

In natural language processing (NLP), there are several ways to tokenize text, and each has its pros and cons. Here are some common methods:

1. Word tokenization

Word tokenization involves splitting the text into individual words, also known as tokens. This is a common method for sentiment analysis, where the model needs to analyze the sentiment of each word. Advantages of word tokenization include simplicity and the ability to capture the sentiment of individual words. Disadvantages include the need to handle punctuation and contractions and the possibility of splitting words that should be treated as a single token (e.g. “can’t”).

2. Sentence tokenization

Sentence tokenization involves splitting the text into individual sentences. This can be useful for tasks such as summarization, where the model needs to understand the structure of the text to generate a summary. Advantages of sentence tokenization include the ability to capture the structure of the text and the ability to handle punctuation. Disadvantages include the possibility of splitting sentences that should be treated as a single unit (e.g. if the text contains a list of items).

3. N-gram tokenization

N-gram tokenization involves splitting the text into groups of n-consecutive words or symbols, known as n-grams. This can be useful for tasks such as language modelling, where the model needs to predict the next word in a sentence based on the context provided by the previous words. Advantages of n-gram tokenization include capturing the context of a word in a sentence and handling words that are split by sentence boundaries. Disadvantages include the need to decide on the value of n and the increased complexity of the model due to the larger number of tokens.

4. Part-of-speech tagging

Part-of-speech tagging: This involves labelling each word in the text with its part of speech (e.g. noun, verb, adjective, etc.). This can be useful for tasks such as syntactic parsing, where the model must understand the sentence’s grammatical structure to identify the dependencies between words. Advantages of part-of-speech tagging include capturing the sentence’s grammatical structure and handling words with multiple possible parts of speech (e.g. “fly” can be a verb or a noun). Disadvantages include the need for a large annotated corpus to train the model and the complexity of the model due to a large number of possible labels.

What Python libraries implement NLP tokenization?

There are several libraries in Python that implement tokenization for natural language processing (NLP) tasks. Some of the most common libraries include:

• NLTK (Natural Language Toolkit): This is a widely-used library for NLP tasks in Python. It includes word and sentence tokenization functions and other everyday NLP tasks such as part-of-speech tagging and named entity recognition.
• spaCy: This is another popular NLP library in Python, known for its fast performance and ease of use. It includes word and sentence tokenization functions and other common NLP tasks such as part-of-speech tagging and dependency parsing.
• TextBlob: This is a simple, user-friendly library for everyday NLP tasks in Python. It includes word and sentence tokenization functions, sentiment analysis, spelling correction, and other tasks.
• Gensim: This is a library for topic modelling and document similarity analysis in Python. It includes functions for sentence tokenization and n-gram generation, as well as other everyday NLP tasks such as Latent Semantic Analysis (LSA) and Latent Dirichlet Allocation (LDA).
• Keras: This is a popular deep-learning framework for building and training neural networks in Python. It provides a high-level API for working with TensorFlow, a powerful open-source library for numerical computation and machine learning.

Overall, there are a lot of Python libraries that use tokenization for NLP tasks, and each one has its own pros and cons. Therefore, choosing a suitable library for your specific use case is vital.

The biggest challenge in NLP tokenization

As with other NLP techniques, the biggest issue is often scaling the technique to include all different possible languages.

Tokenization tasks are typically applied to western text corpora where text is written in for example English or French. These languages use white spaces or punctuation marks to demarcate the beginning and end of sentences. Unfortunately, this method can not be used with other languages such as Chinese, Japanese, Korean, Thai, Hindi, Urdu, Tamil, and others. The need to create a universal tokenization tool that combines all languages is still an issue we face today.

Code implementations for NLP tokenization

1. NLTK Tokenization

The NLTK (Natural Language Toolkit) library in Python includes functions for tokenizing text in various ways. Here is an example of how to use NLTK for tokenization:

# First, import the NLTK library and the relevant tokenizers 
import nltk from nltk.tokenize 
import word_tokenize, sent_tokenize 

# Next, define the text that you want to tokenize 
text = "Natural language processing is an exciting field of study." 

# Use the word_tokenize() function to tokenize the text into words 
tokens = word_tokenize(text) 
print(tokens) 
# Output: ['Natural', 'language', 'processing', 'is', 'an', 'exciting', 'field', 'of', 'study', '.'] 

# Use the sent_tokenize() function to tokenize the text into sentences 
sentences = sent_tokenize(text) 
print(sentences) 
# Output: ['Natural language processing is an exciting field of study.']

In this example, we imported the NLTK library and the relevant tokenizers, defined the text that we wanted to tokenize, and then used the word_tokenize() and sent_tokenize() functions to split the text into words and sentences, respectively. This is just one way to use NLTK for tokenization, and the library includes many other functions and options that you can use to customise your tokenization.

2. SpaCy Tokenization

The spaCy library in Python is a popular choice for natural language processing (NLP) tasks, and it includes functions for tokenizing text in a variety of ways. Here is an example of how to use SpaCy for tokenization:

# First, import the spaCy library and the relevant tokenizer 
import spacy nlp = spacy.load("en_core_web_sm") 

# Next, define the text that you want to tokenize 
text = "Natural language processing is an exciting field of study." 

# Use the nlp object to tokenize the text 
doc = nlp(text) 

# Use the token attribute to access the individual tokens in the document 
for token in doc: 
  print(token.text) 
# Output: 
# Natural 
# language 
# processing 
# is 
# an 
# exciting 
# field 
# of 
# study 
# . 

# Use the sents attribute to access the individual sentences in the document 
for sentence in doc.sents: 
  print(sentence) 
# Output: 
# Natural language processing is an exciting field of study.

In this example, we imported the SpaCy library and the relevant tokenizer, defined the text that we wanted to tokenize, and then used the nlp object to tokenize the text into words and sentences. The token attribute allows us to access the individual tokens in the document, and the sents attribute allows us to access the individual sentences. This is just one way to use SpaCy for tokenization, and the library includes many other functions and options that you can use to customise your tokenization.

3. TextBlob Tokenization

TextBlob is a Python library for working with textual data. It provides a simple API for common natural language processing tasks, such as part-of-speech tagging, noun phrase extraction, sentiment analysis, and more.

To use TextBlob for tokenization, you can use the TextBlob class and its words property. This property returns a list of word tokens in the text. Here is an example:

from textblob import TextBlob 
# Define a text string 
text = "This is a sample text. It contains some words that we can use for tokenization." 

# Create a TextBlob object 
blob = TextBlob(text) 

# Print the list of word tokens 
print(blob.words)

In this example, the blob.words property will return a list of tokens like this: ['This', 'is', 'a', 'sample', 'text', 'It', 'contains', 'some', 'words', 'that', 'we', 'can', 'use', 'for', 'tokenization'] .

Keep in mind that the TextBlob class automatically performs some basic text preprocessing, such as lowercasing all words and removing punctuation. If you want to retain the original case and punctuation, you can use the TextBlob.tokenize() method instead.

4. Gensim Tokenization

Gensim is a Python library for topic modelling and document similarity analysis. To use Gensim for tokenization, you can use the gensim.utils.simple_preprocess() function. This function takes a document as input and returns a list of tokens, or words, that make up the document.

Here is an example of how you can use this function:

import gensim 

# Define a document 
document = "This is a sample document. It contains some text that we can use for tokenization." 

# Use the simple_preprocess() function to tokenize the document 
tokens = gensim.utils.simple_preprocess(document) 

# Print the tokens 
print(tokens)

In this example, the simple_preprocess() function will tokenize the document and return a list of tokens like this: ['this', 'is', 'sample', 'document', 'it', 'contains', 'some', 'text', 'that', 'we', 'can', 'use', 'for', 'tokenization'] .

Keep in mind that the simple_preprocess() function also performs some basic text preprocessing, such as lowercasing all words and removing punctuation. If you want to retain the original case and punctuation, you can use the gensim.utils.tokenize() function instead.

5. Keras Tokenization

Keras is not specifically designed for text processing or tokenization. Instead, it focuses on building and training neural networks for tasks such as image classification, natural language processing, and time series prediction.

However, you can use Keras with other libraries specifically designed for text processing and tokenization. For example, you could use the Tokenizer class from the Keras text module to vectorize your text data, and then use the tokenize() method to perform tokenization. Here is an example:

from keras.preprocessing.text import Tokenizer

# Define a text string
text = "This is a sample text. It contains some words that we can use for tokenization."

# Create a Tokenizer object
tokenizer = Tokenizer()

# Use the fit_on_texts() method to tokenize the text
tokenizer.fit_on_texts([text])

# Print the list of tokens
print(tokenizer.word_index)

In this example, the fit_on_texts() method will tokenize the text and return a list of tokens like this: {'this': 1, 'is': 2, 'a': 3, 'sample': 4, 'text': 5, 'it': 6, 'contains': 7, 'some': 8, 'words': 9, 'that': 10, 'we': 11, 'can': 12, 'use': 13, 'for': 14, 'tokenization': 15} .

Keep in mind that the Tokenizer class in Keras is not designed for traditional tokenization, where each word is represented by a single token. Instead, it is used to vectorize text data for neural network models. If you are looking for a more traditional tokenization approach, you may want to use a different library, such as the others described above.

NLP Tokenization Key Takeaways

• There are several types of tokenizers you could consider using depending on your use case. The most common options are: word tokenization, sentence tokenization, n-gram tokenization and part of speach tagging.
• The main limitation for most tokenizers is that they don’t scale to all possible languages.
• SpaCy, NLTK, Gensim, TextBlob and Keras all have tokenization techniques ready to be used in your own projects.

At Spot Intelligence we frequently use tokenization techniques. It’s also one of the top 10 NLP tecniques. The article is worth a read to find other great NLP techniques.

What is your favourite tokenizer and why? Let us know in the comments.