Image by Author | Ideogram
Introduction
Large language models have revolutionized the entire artificial intelligence landscape in the recent few years, marking the beginning of a new era in AI history. Usually referred to by their acronym LLMs, they transformed the way we communicate with machines, whether for retrieving information, asking questions, or generating a variety of human language content.
As LLMs further permeate our daily and professional lives, it is paramount to understand the concepts and foundations surrounding them, both architecturally and in terms of practical use and applications.
In this article, we explore 10 large language model terms that are key to understanding these formidable AI systems.
1. Transformer Architecture
Definition: The transformer is the foundation of large language models. It is a deep neural network architecture raised to its highest exponent, consisting of a variety of components and layers like position-wise feed-forward networks and self-attention that together allow for efficient parallel processing and context-aware representation of input sequences.
Why it’s key: Thanks to the transformer architecture, it has become possible to understand complex language inputs and generate language outputs at an unprecedented level, overcoming the limitations of previous state-of-the-art natural language processing solutions.
2. Attention Mechanism
Definition: Originally envisaged for language translation tasks in recurrent neural networks, attention mechanisms analyze the relevance of every element in a sequence concerning elements in another sequence, both of varying length and complexity. While the basic attention mechanism is not typically part of transformer architectures underlying LLMs, they laid the foundations for enhanced approaches (as we will discuss shortly).
Why it’s key: Attention mechanisms are key in aligning source and target text sequences in tasks like translation and summarization, turning the language understanding and generation processes into highly contextual tasks.
3. Self-Attention
Definition: If there is a type of component within the transformer architecture that is mainly responsible for the success of LLMs, that is the self-attention mechanism. Self-attention overcomes conventional attention mechanisms’ limitations like long-range sequential processing by allowing each word — or token, more precisely — in a sequence to attend to all other words (tokens) simultaneously, regardless of their position.
Why it’s key: Paying attention to dependencies, patterns, and interrelationships among elements of the same sequence is incredibly useful to extract a deep meaning and context of the input sequence being understood, as well as the target sequence being generated as a response — thereby enabling more coherent and context-aware outputs.
4. Encoder and Decoder
Definition: The classical transformer architecture is roughly divided into two main components or halves: the encoder and the decoder. The encoder is responsible for processing and encoding the input sequence into a deeply contextualized representation, whereas the decoder focuses on generating the output sequence step-by-step utilizing both previously generated parts of the output and the encoder’s resulting representation. Both parts are interconnected, so that the decoder receives processed results from the encoder (called hidden states) as input. Furthermore, both the encoder and the decoder innards are “replicated” in the form of multiple encoder layers and decoder layers, respectively: this level of depth helps the model learn more abstract and nuanced features of the input and output sequences.
Why it’s key: The combination of an encoder and a decoder, each with their own self-attention components, is key to balancing input understanding with output generation in an LLM.
5. Pre-Training
Definition: Just like the foundations of a house from scratch, pre-training is the process of training an LLM for the first time, that is, gradually learning all of its model parameters or weights. The magnitude of these models is such that they may take up to billions of parameters. Hence, pre-training is an inherently costly process that takes days to weeks to complete and requires massive and diverse corpora of text data.
Why it’s key: Pre-training is vital to build an LLM that can understand and assimilate the general language patterns and semantics across a wide spectrum of topics.
6. Fine-Tuning
Definition: Contrary to pre-training, fine-tuning is the process of taking an already pre-trained LLM and training it again on a comparatively smaller and more domain-specific set of data examples, thereby making the model specialized in a specific domain or task. While still computationally expensive, fine-tuning is less costly than pre-training a model from scratch, and it often entails updating model weights only in specific layers of the architecture rather than updating the entire set of parameters across the model architecture.
Why it’s key: Having an LLM specialize in very concrete tasks and application domains like legal analysis, medical diagnosis, or customer support is important because general-purpose pre-trained models may fall short in domain-specific accuracy, terminology, and compliance requirements.
7. Embeddings
Definition: Machines and AI models do not truly understand language, but just numbers. This also applies to LLMs, so while we generally speak about models that “understand and generate language”, what they do is handle a numerical representation of such language that keeps its key properties largely intact: these numerical (vector, to be more precise) representations are what we call embeddings.
Why it’s key: Mapping input text sequences into embedding representations enables LLMs to perform reasoning, similarity analysis, and data generalization across contexts, all without losing the main properties of the original text; hence, raw responses generated by the model can be mapped back to semantically coherent and appropriate human language.
8. Prompt Engineering
Definition: End users of LLMs should get familiar with best practices for optimal use of these models to achieve their goals, and prompt engineering stands out as a strategic and practical approach to this end. Prompt engineering encompasses a set of guidelines and techniques for designing effective user prompts that guide the model towards producing useful, accurate, and goal-oriented responses.
Why it’s key: Oftentimes, obtaining high-quality, precise, and relevant LLM outputs is largely a matter of learning how to write high-quality prompts that are clear, specific, and structured to align the LLM’s capabilities and strengths, e.g., by turning a vague user question into a precise and meaningful answer.
9. In-Context Learning
Definition: Also called few-shot learning, this is a method to teach LLMs to perform new tasks predicated on providing examples of desired outcomes and instructions directly in the prompt, without re-training or fine-tuning the model. It can be deemed as a specialized form of prompt engineering, as it fully leverages the model’s gained knowledge during pre-training to extract patterns and adapt to new tasks on the fly.
Why it’s key: In-context learning has been proven as an effective approach to flexibly and efficiently learn to solve new tasks based on examples.
10. Parameter Count
Definition: The size and complexity of an LLM are usually measured by several factors, parameter count being one of them. Well-known model names like GPT-3 (with 175B parameters) and LLaMA-2 (with up to 70B parameters) clearly reflect the importance and significance of the number of parameters in scaling language capabilities and the expressiveness of an LLM in generating language. The number of parameters matters when it comes to measuring an LLM’s capabilities, but other aspects like the amount and quality of training data, architecture design, and fine-tuning approaches used are likewise important.
Why it’s key: The parameter count is instrumental not only in defining the model’s capacity to “store” and handle linguistic knowledge, but also in estimating its performance on challenging reasoning and generation tasks, especially when they entail multi-phase dialogues between the user and the model.
Wrapping Up
This article explored the significance of ten key terms surrounding large language models: the main focus of attention across the entire AI landscape, due to the remarkable achievements made by these models over the last few years. Being familiar with these concepts places you in an advantageous position to stay abreast of new trends and developments in the rapidly evolving LLM landscape.
Iván Palomares Carrascosa is a leader, writer, speaker, and adviser in AI, machine learning, deep learning & LLMs. He trains and guides others in harnessing AI in the real world.
