Free preview lesson · Natural Language Processing Essentials · Module 1: Working with text
Why language is hard for computers
Human language is ambiguous, messy and full of context. Before any model can work with text, that text has to be broken into pieces a computer can count.
Why language is hard
The same word can mean different things (a bank of a river, a bank that holds money), and the same meaning can be written many ways. Spelling errors, slang and emoji add more variation. NLP is largely about managing that variation.
Tokenisation
Tokenisation splits text into units called tokens. Simple approaches split on spaces and punctuation; modern language models use sub-word tokens so that rare words can be built from common pieces. The choice affects vocabulary size and how well a model handles new words.
Normalisation
Normalisation reduces unhelpful variation: lowercasing, removing extra whitespace, standardising quotes and sometimes reducing words to a root form. How much you normalise depends on the task — for sentiment, an exclamation mark may carry meaning you want to keep.
Common pitfall
Applying every cleaning step by default. Removing punctuation, numbers or capital letters can throw away information that matters for your task.
Action step
Take ten real sentences from emails or reviews and tokenise them by hand, noting every decision you had to make.
This is one lesson from Natural Language Processing Essentials.
Educational content only. No income, revenue or earnings results are promised or implied. No certification, licence or professional credential is awarded.