Words and Tokens

Chadi Helwe, PhD

CSC 464:  Deep Learning for Natural

Language Processing

The Building Blocks of Text

Words and Tokens

ELIZA (1966)

Tokenization

Tokenization separates text into distinct units called tokens, representing words or parts of words. It’s the essential first step in natural language processing (NLP), converting raw text into units that algorithms can process.

Words and Tokens

Words and Tokens

What is a Word? (1/2)

The most intuitive definition is the orthographic word:  words separated by whitespace.

Words and Tokens

Word Type and Word Instance (1/2)

Word Type

A unique vocabulary item

Example: "the" is one type

Word Instance

Each occurrence of a word

Example: "the" appears many times

Words and Tokens

Word Type and Word Instance (2/2)

Words and Tokens

What is Considered a Word?

Problem 1: Disfluencies in Speech

"I do uh main-mainly business data processing"

Is "uh" a word? It is a filled pause. Is "main-" a word? It's a fragment. They are often treated as words in speech recognition.

Problem 2: Capitalization

Are "They" and "they" the same word type?

It depends on the task. Sometimes we want to ignore case; other times, capitalization is a crucial feature, such as names.

Problem 3: Compound words and Clitics

"I'm" is orthographically one word; grammatically, it functions as two words: the subject pronoun "I" and the verb "m," short for "am."

Problem 4: Punctuation

Is AT&T one word? What about Ph.D.?

Internal punctuation complicates simple space-based rules.

Words and Tokens

What About Other Languages?

The previous definition of a "word" fails in many languages.

English:
Yao Ming reaches the finals

Chinese:

姚明进入总决赛

Words and Tokens

Heap's Law (1/2)

Heap's Law

A mathematical relationship indicates that as the corpus size increases (word instances |N|), the number of unique word types (|V|) also grows.

Words and Tokens

Heap's Law (2/2)

As we have seen previously, we face two major issues that need to be addressed for NLP models to effectively process text:

- Many languages lack orthographic words

- Unknown words.

How can we solve these problems?

Morphemes

Morphemes: Parts of Words (1/2)

A morpheme is a minimal meaning-bearing unit in a language.morpheme is a minimal meaning-bearing unit in a languge

Examples:

fox → 1 morpheme

cats → 2 morphemes: cat and -s plural 

Words and Tokens

Morphemes: Parts of Words (2/2)

Carefully

Words and Tokens

Morphology Across Languages

Words and Tokens

The fact that morphemes can be hard to define, and that many languages can have complex morphemes that aren’t easy to break up into pieces, makes it very difficult to use morphemes as a standard for cross-lingual tokenization.

Unicode

What is Unicode?

How do we represent characters accross languages and writing systems?

The Unicode standard is a method for representing text written using any character in any script of the language of the world.

Words and Tokens

ASCII (1/3)

The Latin characters used to write English were represented with a code called ASCII (American Standard Code for Information Interchange)

ASCII:

• Each character is represented as one byte
• A byte represents 256 characters; ASCII uses 127 of them. The high-order bit of ASCII bytes is always set to 0. (Actually, it only used 95 of them)

Words and Tokens

ASCII (2/3)

ASCII is of course insufficient since there are lots of other characters in the world’s writing systems! Even for scripts that use Latin characters, there are many more than the 95 in ASCII.

Words and Tokens

ASCII (3/3)

Many languages are not based on Latin characters at all! The Devanagari script is used for 120 languages, including Hindi, Marathi, Nepali, Sindhi, and Sanskrit.

Chinese has about 100,000 Chinese characters in Unicode.

All in all there are more than 150,000 characters and 168 different scripts supported in Unicode 16.0.

Words and Tokens

Code Points

Unicode assigns a unique id, called a code point, for each one of these 150,000 characters.

The code point is an abstract representation of the character, and each code point is represented by a number, traditionally written in hexadecimal, from 0 through 0x10FFFF (which is 1,114,111 decimal).

Words and Tokens

UTF-8 Encoding

While the code point (the unique id) is the abstract Unicode representation of the character, we don’t just stick that id in a text file.

Instead, whenever we need to represent a character in a text string, we write an encoding of the character.

The most common encoding standard is UTF-8, which efficiently represents characters using a variable number of bytes.

Words and Tokens

Subword Tokenization

Why we need to Tokenize?

One reason is that converting an input to a deterministic fixed set of units means that different algorithms and systems can agree on simple questions.

For example, how long is this text? How many units are in it? Or is "don’t" or "New York" one token or two?

Standardizing is thus essential for replicability in NLP experiments, and many algorithms.

Words and Tokens

Subwords

To deal with the unknown word problem, modern tokenizers automatically induce sets of tokens that include tokens smaller than words, called subwords.

Subwords can be arbitrary substrings, or they can be meaning-bearing units like the morphemes -est or -er.

Words and Tokens

Byte-Pair Encoding (BPE)

Two tokenization algorithms are commonly used in modern language models: byte-pair encoding (BPE) and unigram language modeling (ULM). We will introduce the byte-pair encoding (BPE) algorithm.

The BPE algorithm consists of two components: a trainer and an encoder.

Words and Tokens

BPE Training (1/4)

The BPE training algorithm iteratively merges frequent neighboring tokens to create longer and longer tokens.

The algorithm begins with a vocabulary that is just the set of all individual characters. It then examines the training corpus, and finds the two characters that are most frequently adjacent.

1. Initialization

The algorithm starts with a vocabulary consisting only of individual characters. It treats the text as a sequence of these characters.

Initial Vocabulary: {A, B, C, D, E}

Initial Corpus: A B D C A B E C A B (Length: 10)

Words and Tokens

BPE Training (2/4)

2. Iterative Merging

The core of the algorithm is a loop that repeats the following steps:

Step 1: Count and Merge "A B"

The algorithm scans the corpus and finds that the pair "A B" appears most frequently (3 times). It creates a new token AB and updates the corpus.

- New Token: AB

- Updated Corpus: AB D C AB E C AB

- Corpus Length: Reduced to 7 tokens.

Words and Tokens

BPE Training (3/4)

2. Iterative Merging

Step 2: Count and Merge "C AB"

Scanning the new corpus, it finds that the pair "C" followed by "AB" is now the most frequent adjacent pair. It merges them into a single token CAB.

- New Token: CAB

- Updated Corpus: AB D CAB E CAB

- Corpus Length: Reduced to 5 tokens.

Words and Tokens

BPE Training (4/4)

Words and Tokens

BPE Encoder

Once we have learned our vocabulary, we use the BPE encoder to tokenize a test sentence. This encoder operates solely on the test data using the merges we learned from the training data, applying them greedily in the order they were learned.

First, we break down each word in the test sentence into its individual characters. Next, we apply the first merging rule, followed by the second, and continue this process sequentially.

The result: A vocabulary that is a mix of words, morphemes, and characters.

Words and Tokens

BPE in Practice

DEMO

Words and Tokens

Rule-based Tokenization

 Rule-Based Tokenization

Words and Tokens

A method where tokens are defined by pre-set standards and implemented via deterministic rules, rather than learned from data.

Challenges in English:

• Punctuation Handling: e.g., Dr., Inc., Ph.D., AT&T

• Numerical Expressions: Prices (e.g., $45.55), Dates (e.g. 01/02/06), Commas in Numbers (e.g. 4232,232)

• Web and Social Media: URLs (e.g., https://google.com), hashtags (e,g, #NLP)

• Clitics: e.g., I'm, you're, what're

The Penn Treebank Standard

Words and Tokens

A widely used standard for parsed corpora released by the Linguistic Data Consortium (LDC).

Key Rules:

• Clitics: Separates clitics from the root (e.g., doesn't → does + n't).

• Hyphens: Keeps hyphenated words together.

• Punctuation: Separates all punctuation marks.

NLTK: Natural Language Toolkit

Words and Tokens

NLTK (Natural Language Toolkit) is a Python-based software library used for natural language processing that provides tools for tasks such as tokenization and text analysis.

Regular Expressions

Regular Expression (Regex)

One of the most useful tools for text processing in computer science is the regular expression (or regex), a language for specifying text strings.

Regexes are used in every computer language, in text processing tools like Unix grep, and in editors like vim or Emacs.

Core functions:

• Search: Finding a specific pattern within a string.

• Substitution: Replacing one string pattern with another.

• Tokenization: Essential for the pre-tokenization step in algorithms like BPE.

Words and Tokens

Basic Character Matching (1/3)

Simple Sequences:

• Regexes are case-sensitive by default.

• Example: r"Buttercup" matches the substring "Buttercup".

Disjunction (Square Brackets):

• Specifies a choice between characters.

• [mM] matches either 'm' or 'M'.

• [abc] matches 'a', 'b', or 'c'.

Words and Tokens

Basic Character Matching (2/3)

Ranges:

• Use a dash - to specify a range of characters.

• [A-Z]: Any uppercase letter.

• [0-9]: Any single digit.

Words and Tokens

Basic Character Matching (3/3)

Negation:

• The caret ^ inside brackets negates the set.
• [^a]: Matches any character except 'a'.

Words and Tokens

Quantifiers (Counting and Optionality) (1/2)

Basic Operators:

• Question Mark ?: Indicates zero or one occurrence (optional). For example, "colou?r" matches both "color" and "colour."

• Kleene Star *: Zero or more occurrences. baaa* matches "baa", "baaa", "baaaa", etc.

• Kleene Plus +: One or more occurrences. [0-9]+ matches a sequence of digits (integer).

• Specific counts {n}: Exactly nn occurrences. ax{10}z matches 'a' followed by exactly 10 'x's, then 'z'.

• Period .: Matches any single character (except a newline). Often combined with * as .* to mean "any string of characters.

Words and Tokens

Quantifiers (Counting and Optionality) (2/2)

Words and Tokens

Anchors and Boundaries (1/2)

Anchors:

• Caret ^: Matches the start of a line. ^The matches "The" only at the beginning of a line.

• Dollar Sign $: Matches the end of a line. dog.$ matches "dog." at the end of a line.

Boundaries:

• \b: Word boundary (start or end of a word).  \bthe\b matches "the" but not "other".

• \B: Non-word boundary.

Words and Tokens

Anchors and Boundaries (2/2)

Words and Tokens

Disjunction, Grouping and Precedence (1/2)

Disjunction (Pipe|)

• Logical OR operator.

• cat|dog matches "cat" or "dog".

Grouping ()

• Encloses a pattern to apply operators to the whole group.
• gupp(y|ies) matches "guppy" or "guppies". (Without parens, guppy|ies would match "guppy" or "ies").

Words and Tokens

Disjunction, Grouping and Precedence (2/2)

Operator Precedence (High to Low)

1. Parenthesis ()
2. Counters * + ? {}
3. Sequences and Anchors the ^ $
4. Disjunction |​

Words and Tokens

More Operators

Words and Tokens

Examples

Create a regex pattern that matches the US date format. e.g. 10/15/2025

Words and Tokens

Thank You