How To Significantly Enhance LLMs by Leveraging Context Engineering

is the science of providing LLMs with the correct context to maximize performance. When you work with LLMs, you typically create a system prompt, asking the LLM to perform a certain task. However, when working with LLMs from a programmer’s perspective, there are more elements to consider. You have to determine what other data you can feed your LLM to improve its ability to perform the task you asked it to do.

In this article, I will discuss the science of context engineering and how you can apply context engineering techniques to improve your LLM’s performance.

You can also read my articles on Reliability for LLM Applications and Document QA using Multimodal LLMs

Table of Contents

Definition

Before I start, it’s important to define the term context engineering. Context engineering is essentially the science of deciding what to feed into your LLM. This can, for example, be:

• The system prompt, which tells the LLM how to act
• Document data fetch using RAG vector search
• Few-shot examples
• Tools

The closest previous description of this has been the term prompt engineering. However, prompt engineering is a less descriptive term, considering it implies only changing the system prompt you are feeding to the LLM. To get maximum performance out of your LLM, you have to consider all the context you are feeding into it, not only the system prompt.

Motivation

My initial motivation for this article came from reading this Tweet by Andrej Karpathy.

I really agreed with the point Andrej made in this tweet. Prompt engineering is definitely an important science when working with LLMs. However, prompt engineering doesn’t cover everything we input into LLMs. In addition to the system prompt you write, you also have to consider elements such as:

• Which data should you insert into your prompt
• How do you fetch that data
• How to only provide relevant information to the LLM
• Etc.

I will discuss all of these points throughout this article.

API vs Console usage

One important difference to clarify is whether you are using the LLMs from an API (calling it with code), or via the console (for example, via the ChatGPT website or application). Context engineering is definitely important when working with LLMs through the console; however, my focus in this article will be on API usage. The reason for this is that when using an API, you have more options for dynamically changing the context you are feeding the LLM. For example, you can do RAG, where you first perform a vector search, and only feed the LLM the most important bits of information, rather than the entire database.

These dynamic changes are not available in the same way when interacting with LLMs through the console; thus, I will focus on using LLMs through an API.

Context engineering techniques

Zero-shot prompting

Zero-shot prompting is the baseline for context engineering. Doing a task zero-shot means the LLM is performing a task it hasn’t seen before. You are essentially only providing a task description as context for the LLM. For example, providing an LLM with a long text and asking it to classify the text into class A or B, according to some definition of the classes. The context (prompt) you are feeding the LLM could look something like this:

You are an expert text classifier, and tasked with classifying texts into
class A or class B. 
- Class A: The text contains a positive sentiment
- Class B: The next contains a negative sentiment

Classify the text: {text}

Depending on the task, this could work very well. LLMs are generalists and are able to perform most simple text-based tasks. Classifying a text into one of two classes will usually be a simple task, and zero-shot prompting will thus usually work quite well.

Few-shot prompting

This infographic highlights how to perform few-shot prompting:

The follow-up from zero-shot prompting is few-shot prompting. With few-shot prompting, you provide the LLM with a prompt similar to the one above, but you also provide it with examples of the task it will perform. This added context will help the LLM improve at performing the task. Following up on the prompt above, a few-shot prompt could look like:

You are an expert text classifier, and tasked with classifying texts into
class A or class B. 
- Class A: The text contains a positive sentiment
- Class B: The next contains a negative sentiment


{text 1} -> Class A


{text 2} -> class B


Classify the text: {text}

You can see I’ve provided the model some examples wrapped in tags. I have discussed the topic of creating robust LLM prompts in my article on LLM reliability below:

@tool
def get_weather(city):
    # code to retrieve the current weather for a city
    return weather