📚 How Large Language Models Work & Their Applications | 大型语言模型的工作原理与应用
Large Language Models (LLMs) such as GPT, Claude, and LLaMA have transformed the field of artificial intelligence. But how do they actually work beneath the hood? This article provides a structured, exam-oriented guide to the core principles, architecture, training process, and real-world applications of LLMs.
以 GPT、Claude、LLaMA 为代表的大型语言模型（LLM）已经彻底改变了人工智能领域。然而，它们内部究竟是如何工作的？本文将提供一份结构化、紧扣考点的指南，系统讲解 LLM 的核心原理、架构、训练过程及实际应用。
1. What Is a Large Language Model? | 什么是大型语言模型？
A Large Language Model is a deep neural network trained on massive amounts of text data to understand and generate human-like language. The "large" refers to both the enormous scale of training data and the billions of parameters in the model.
大型语言模型是一种在海量文本数据上训练的深度神经网络，能够理解并生成类人语言。这里的"大"既指训练数据的庞大规模，也指模型中数十亿计的参数数量。
- Parameters: The internal weights learned during training, which encode linguistic knowledge.
- 参数：训练过程中学习到的内部权重，编码了语言知识。
- Scale: Models typically range from 1 billion to over 1 trillion parameters.
- 规模：模型参数通常在 10 亿到超过 1 万亿之间。
- Emergent abilities: As models grow larger, they develop capabilities not present in smaller models, such as reasoning and in-context learning.
- 涌现能力：随着模型规模增大，它们会发展出小模型不具备的能力，如推理和上下文学习。
2. Core Architecture: The Transformer | 核心架构：Transformer
Almost all modern LLMs are built on the Transformer architecture, introduced in the 2017 paper "Attention Is All You Need." Unlike earlier recurrent networks (RNNs), Transformers process entire sequences in parallel, making them far more efficient for large-scale training.
几乎所有现代 LLM 都基于 Transformer 架构，该架构首次提出于 2017 年的论文《Attention Is All You Need》。与早期的循环神经网络（RNN）不同，Transformer 并行处理整个序列，因而在大规模训练中效率远高于 RNN。
Transformer = Encoder + Decoder (+ Attention Mechanism)
The key innovation is the self-attention mechanism, which calculates the importance of each word relative to every other word in the input sequence, enabling the model to capture long-range dependencies and contextual relationships.
其关键创新是自注意力机制。该机制计算输入序列中每个词相对于其他所有词的重要性，使模型能够捕获长距离依赖和上下文关系。
3. Tokenization: How Models Read Text | Tokenization：模型如何读取文本
Before processing, raw text must be converted into tokens — numerical IDs that the model can understand. Tokenization is the process of splitting text into smaller units, which may be words, subwords, or even individual characters.
在正式处理之前，原始文本必须被转换为 token——即模型能够理解的数字标识。Tokenization 是将文本切分为更小单元的过程，这些单元可以是单词、子词，甚至单个字符。
- Word-level tokenization: Simple but fails on rare or unseen words.
- 单词级切分：简单但对生僻词或未见过的词失效。
- Subword tokenization (BPE, WordPiece): Balances vocabulary size and coverage; splits rare words into known subword units.
- 子词切分（BPE、WordPiece）：在词表大小与覆盖率之间取得平衡，将生僻词拆分为已知的子词单元。
- Vocabulary size: Typically 32,000–100,000 tokens for modern LLMs.
- 词表大小：现代 LLM 通常为 32,000 到 100,000 个 token。
For the phrase "unbelievable", the tokenizer might produce: "un" + "believ" + "able", each mapped to a unique integer ID before being fed into the model.
例如，单词 "unbelievable" 可能会被切分为 "un" + "believ" + "able"，每个部分先映射到唯一的整数 ID，再输入模型。
4. Embeddings: From Tokens to Vectors | 嵌入：从 Token 到向量
Once tokens are obtained, each token ID is mapped to a high-dimensional vector known as an embedding. Embeddings capture semantic meaning — words with similar meanings are positioned close together in vector space.
获得 token 后，每个 token ID 会被映射到一个高维向量，称为嵌入（embedding）。嵌入能够捕获语义信息——语义相近的词在向量空间中距离很近。
v(token) = E[token_id]
Positional encoding is added to the embedding to give the model information about the order of tokens, since the Transformer contains no inherent sense of sequence order. The final input to the model is the sum of token embedding and positional encoding, followed by the attention computation.
由于 Transformer 本身不具备序列顺序的概念，因此会在嵌入中加入位置编码，以提供 token 的顺序信息。模型的最终输入是 token 嵌入与位置编码之和，然后进入注意力计算阶段。
5. The Self-Attention Mechanism | 自注意力机制
Self-attention is the mathematical heart of LLMs. For each token, the model computes three vectors: Query (Q), Key (K), and Value (V). The attention score between two tokens is calculated as the dot product of Q and K, scaled and passed through a softmax to produce weights.
自注意力是 LLM 的数学核心。对于每个 token，模型计算三个向量：查询（Q）、键（K）和值（V）。两个 token 之间的注意力分数由 Q 和 K 的点积计算得出，经缩放后通过 softmax 生成权重。
Attention(Q, K, V) = softmax(QKᵀ / √dₖ) V
Here, dₖ is the dimension of the key vectors. Dividing by √dₖ prevents the dot product from becoming too large, which would push softmax into regions with very small gradients. This mechanism allows the model to focus on the most relevant tokens in the sequence when generating each output token.
其中 dₖ 是键向量的维度。除以 √dₖ 可防止点积结果过大，避免 softmax 进入梯度极小区域。这一机制使模型在生成每个输出 token 时，能够聚焦于序列中最相关的 token。这种处理方式使模型可以并行处理所有 token 之间的关系，而不像 RNN 那样必须逐个顺序计算。
6. Training Phases: From Pretraining to Alignment | 训练阶段：从预训练到对齐
The training of an LLM can be divided into several distinct phases, each with a specific purpose and method.
LLM 的训练可以分为几个不同阶段，每个阶段都有特定的目标和方法。
- Phase 1 – Pretraining: The model learns to predict the next token from massive unlabeled text corpora. This is self-supervised learning, requiring no human labels.
- 第一阶段 — 预训练：模型从大规模无标注文本语料中学习预测下一个 token。这是自监督学习，无需人工标注。
- Phase 2 – Supervised Fine-Tuning (SFT): The model is trained on curated instruction-following datasets to align its outputs with human expectations.
- 第二阶段 — 监督微调（SFT）：模型在人工整理的指令遵循数据集上训练，使其输出更符合人类期望。
- Phase 3 – Reinforcement Learning from Human Feedback (RLHF): A reward model is trained from human preference comparisons, and the LLM is further optimized via reinforcement learning to maximize this reward.
- 第三阶段 — 基于人类反馈的强化学习（RLHF）：从人类偏好对比中训练一个奖励模型，再通过强化学习优化 LLM，使其输出获得更高奖励。
Next-token prediction → SFT → RLHF
7. Pretraining Objective: Predicting the Next Token | 预训练目标：预测下一个 Token
The fundamental training objective of an LLM is deceptively simple: given a sequence of tokens, predict the next token. During pretraining, the model is shown billions of sentences and updates its parameters to minimize the cross-entropy loss between its predictions and the actual next tokens.
LLM 的基本训练目标看似简单：给定一串 token，预测下一个 token。在预训练阶段，模型被展示数十亿个句子，并通过最小化其预测与实际下一个 token 之间的交叉熵损失来更新参数。
L = −Σ log P(xₜ | x₁, x₂, …, xₜ₋₁)
This is why LLMs are called "autoregressive" models — each token is generated one at a time, conditioned on all previously generated tokens. Understanding this mechanism is essential for explaining why LLMs sometimes produce hallucinated or repetitive content.
这也是 LLM 被称为"自回归"模型的原因——每个 token 的生成都依赖于之前生成的所有 token，逐个依次产生。理解这一机制对于解释 LLM 为何有时会产生幻觉内容或重复内容至关重要。
8. Key Technologies: RAG, CoT, and Distillation | 关键技术：RAG、CoT 与蒸馏
Several practical techniques enhance LLM performance and mitigate their weaknesses in real-world applications.
若干实用技术能够增强 LLM 的性能并弥补其在实际应用中的不足。
- Retrieval-Augmented Generation (RAG): Combines the LLM with an external retrieval system to fetch relevant documents from a knowledge base. The retrieved information is added to the prompt, enabling the model to access up-to-date or domain-specific facts without retraining.
- 检索增强生成（RAG）：将 LLM 与外部检索系统结合，从知识库中抓取相关文档并添加到提示词中，使模型无需重新训练即可访问最新或特定领域的事实信息。
- Chain-of-Thought (CoT): Prompting the model to produce intermediate reasoning steps before the final answer. This dramatically improves performance on math, logic, and multi-step inference tasks.
- 思维链（CoT）：引导模型在给出最终答案前先输出中间推理步骤。这能显著提升模型在数学、逻辑和多步推理任务上的表现。
- Knowledge Distillation: A smaller "student" model is trained to replicate the behavior of a larger "teacher" model, producing more efficient models for deployment.
- 知识蒸馏：训练一个较小的"学生"模型来复现较大"教师"模型的行为，从而获得更适合部署的高效模型。
9. Applications Across Domains | 跨领域应用场景
LLMs have found applications in nearly every sector, from education to medicine, finance, and software engineering.
LLM 已在几乎每个行业找到应用，从教育、医疗、金融到软件工程，范围极为广泛。
| Domain | 领域 | Application | 应用 |
| Education | 教育 | Personalized tutoring, essay grading, question generation | 个性化辅导、作文批改、题目生成 |
| Software | 软件 | Code generation, debugging assistance, documentation | 代码生成、调试辅助、文档撰写 |
| Healthcare | 医疗 | Clinical note summarization, literature review support | 临床笔记摘要、文献综述辅助 |
| Business | 商业 | Customer service chatbots, sentiment analysis, report generation | 客服聊天机器人、情感分析、报告生成 |
| Creative Arts | 创意领域 | Content generation, translation, style transfer | 内容创作、翻译、风格迁移 |
The key to effective application lies in the design of prompts and the selection of the right model for the task, balancing response quality against computational cost.
有效应用的关键在于精心设计提示词，并根据任务选择合适的模型，在响应质量与计算成本之间取得平衡。
10. Limitations and Challenges | 局限性与挑战
Despite their remarkable capabilities, LLMs face several fundamental limitations that every student should understand.
尽管 LLM 能力非凡，但它们仍面临若干根本性局限，这是每位学生都应掌握的知识点。
- Hallucination: The model generates factually incorrect or fabricated content with high confidence.
- 幻觉：模型以极高置信度生成事实上错误或凭空虚构的内容。
- Knowledge cutoff: The model only knows information up to its training date.
- 知识截止：模型只了解其训练截止日期之前的信息。
- Bias: Training data contains societal biases, which the model may amplify.
- 偏见：训练数据中包含社会偏见，模型可能将其放大。
- Computational cost: Training and inference require enormous GPU resources and energy.
- 计算成本：训练和推理需要巨大的 GPU 资源和能源消耗。
- Lack of true reasoning: LLMs recognize statistical patterns rather than performing genuine logical deduction.
- 缺乏真正推理：LLM 识别的是统计模式而非进行真正的逻辑演绎。
11. How to Work with LLMs: API and Inference | 如何调用 LLM：API 与推理
In practice, most developers interact with LLMs through REST APIs rather than training their own models. A typical API request includes a system prompt (setting the model's behavior), a user prompt (the actual query), and hyperparameters such as temperature.
在实际开发中，大多数开发者通过 REST API 与 LLM 交互，而非自行训练模型。一个典型的 API 请求包含 system prompt（设定模型行为）、user prompt（实际查询）以及诸如 temperature 等超参数。
temperature → randomness of output (0 = deterministic, 1 = highly random)
At inference time, the model generates output token by token using a decoding strategy. Common strategies include greedy decoding, beam search, and top-p (nucleus) sampling. Each affects the quality, diversity, and coherence of the generated text in different ways.
推理阶段，模型采用解码策略逐 token 生成输出。常见策略包括贪心解码、束搜索和 top-p（核）采样。不同策略对生成文本的质量、多样性和连贯性各有不同影响。例如，top-p 采样只从累积概率超过阈值 p 的最小候选集中采样，可以获得质量与多样性之间的平衡。
12. Exam-Focused Summary | 考点速记小结
Here are the high-yield points most likely to appear in exam questions, condensed for rapid revision.
以下是最有可能出现在考题中的高频考点，已为快速复习做了精简归纳。
- LLMs are based on the Transformer architecture with self-attention at its core.
- LLM 基于 Transformer 架构，以自注意力为核心。
- Training pipeline: pretraining → SFT → RLHF.
- 训练流程：预训练 → 监督微调 → 人类反馈强化学习。
- The pretraining objective is next-token prediction (autoregressive).
- 预训练目标是下一个 token 预测（自回归）。
- Embeddings convert tokens into vectors; positional encoding adds order information.
- 嵌入将 token 转换为向量；位置编码补充顺序信息。
- Attention formula: softmax(QKᵀ / √dₖ)V.
- 注意力公式：softmax(QKᵀ / √dₖ)V。
- RAG solves knowledge freshness; CoT improves reasoning; distillation reduces model size.
- RAG 解决知识时效问题；CoT 提升推理能力；蒸馏 压缩模型体积。
- Key limitations: hallucination, bias, knowledge cutoff, high cost.
- 关键局限：幻觉、偏见、知识截止、高成本。
Master these concepts and you will be well-prepared to answer exam questions on large language models — both theoretical explanations and application-based scenarios.
掌握以上概念，你就能游刃有余地回答关于大型语言模型的考试题目——无论是理论解释还是应用场景分析。
Published by TutorHao | Computer Science Revision Series | aleveler.com
更多咨询请联系16621398022（同微信）
或截图后用「扫一扫 → 相册」选择本图