# Natural Language Processing (NLP) The discipline of teaching machines to understand, interpret, and generate human language. NLP is the parent discipline behind much of what powers [[Neural Information Retrieval (IR)]] — but where IR asks "how do I find the right document?", NLP asks "how do I understand what this text means?" In practice, every stage of the [[AnomalyLab]] pipeline — from chunking client blogs, to extracting topics with BERTopic, to generating summaries — is an NLP operation before it becomes an IR one. ## Core Concepts - [[Neural Information Retrieval (IR)/Natural Language Processing (NLP)|Natural Language Processing (NLP)]] - Foundational overview - [[Transformer Architecture]] - "Attention Is All You Need" — the architecture behind BERT, GPT, and modern LLMs - [[Tokens]] - The atomic units of text processing - [[Large Language Model (LLM)]] - Foundation models for language understanding and generation - [[Named Entity Recognition (NER)]] - Extracting people, organizations, locations, and other entities from text ## Keyword Extraction Identifying the most significant words or phrases that describe a document's content. Central to how [[AnomalyLab]] builds topic representations from client blogs. - [[Keyword Extraction Algorithms]] - Overview and comparison of approaches - [[KeyBERT]] - BERT-based extraction (best quality, slower) - [[TextRank]] - Graph-based extraction (better for nuance) - [[RAKE]] - Rapid Automatic Keyword Extraction (better for phrases) - [[YAKE]] - Yet Another Keyword Extractor (better for efficiency) - [[Term Frequency-Inverse Document Frequency (TF-IDF)]] - Classic statistical weighting - [[TF-IDF]] - Implementation details - [[Keywords vs Questions vs Events]] - How keyword types map to different content strategies ## Summarization Condensing longer texts into shorter representations while retaining meaning. The distillation that generative AI performs on retrieved chunks is a form of summarization. - [[Summarization Agorithms]] - Overview: extractive vs abstractive approaches and when to use each - [[Extractive Summarization]] - Selecting existing sentences that best represent the text - [[Abstractive Summarization]] - Generating new sentences that capture the meaning ## Topic Modeling Discovering latent themes across a corpus. This is how [[AnomalyLab]] turns hundreds of client blogs into coherent topic clusters that can be monitored against global news. - [[Topic Modeling]] - Classical vs neural approaches - [[BERTopic]] - Neural topic modeling using transformers, UMAP, HDBSCAN, and c-TF-IDF - [[BERTopic Contribution]] - Contributing to the BERTopic library - [[BERTopic vs Top2Vec and Other Topic Models]] - Model comparison - [[Latent Dirichlet Allocation (LDA)]] - Classical probabilistic topic modeling - [[Dynamic Topic Modelling]] - Tracking how topics evolve over time ## Text Processing Preparing raw text for downstream NLP tasks. The order of operations matters — normalize and chunk identically for both content and queries. - [[Text Normalization]] - Unicode normalization, accent removal, case folding — preserving signal while minimizing noise - [[Text Chunking]] - Segmenting documents into manageable, self-contained pieces (word-level, sentence-level, semantic) - [[Semantic double-pass merging chunking (SDPM)]] - Advanced semantic chunking technique - [[Document Expansion]] - Enriching documents with predicted pseudo-queries (Doc2query) ## Text Representations How text gets converted into mathematical objects that machines can compare. - [[Bag-of-words (BoW)]] - Simplest representation: word frequency counts - [[Bag-of-documents Model]] - Document-level representation - [[Dense Vectors ("Embeddings")]] - Continuous vector representations capturing semantic meaning - [[Sparse Vectors ("Embeddings")]] - High-dimensional sparse representations preserving lexical precision ## Encoder Architectures The neural network designs that produce text representations. - [[Bi-encoder]] - Independent encoding of queries and documents (fast, scalable) - [[Cross-encoder]] - Joint encoding of query-document pairs (more accurate, slower) - [[SPLADE]] - Sparse learned representations combining neural and lexical strengths - [[Train BERT]] - Training BERT models for NLP tasks ## Clustering Grouping similar texts together without predefined categories. Used in topic modeling and corpus analysis. - [[K-means]] - Partitional clustering by centroid proximity - [[HDBSCAN]] - Density-based clustering (handles noise and variable-density clusters) - [[Partitional Clustering]] - Clustering category overview - [[The Elbow Method]] - Determining optimal number of clusters - [[The Gap Statistic]] - Statistical approach to optimal k - [[The Silhouette Method]] - Measuring cluster quality - [[Combined Methods to Determine n_observations for k_means Clustering]] - Multi-method approach - [[Selecting Optimal k in K-Means Clustering - Industry Standards as of March 2026]] ## Similarity & Distance Metrics How to measure whether two pieces of text are "close" in meaning. - [[Cosine Similarity]] - Standard metric for comparing embedding vectors - [[Dot Product vs Cosine Similarity]] - When to use which ## Ranking Models NLP models specifically designed to score document relevance. - [[monoT5]] - T5-based pointwise reranker - [[TILDE]] - Token-level document importance estimation - [[LambdaMART]] - Gradient-boosted learning-to-rank - [[Optimum]] - Hugging Face model optimization library - [[Learning to Rank (LTR)]] - The ranking paradigm overview - [[Reranking Methods]] - Techniques for post-retrieval reranking ## Related These NLP techniques feed directly into search and retrieval systems — see [[Neural Information Retrieval (IR)]] for how they're applied. For practical AI applications of this research, see [[Applied AI]].