State of GPT | BRK216HFS

[MUSIC] ANNOUNCER: Please welcome AI researcher and founding member of OpenAI, Andrej Karpathy.

ANDREJ KARPATHY: Hi, everyone. I'm happy to be here to tell you about the state of GPT and more generally about the rapidly growing ecosystem of large language models. I would like to partition the talk into two parts. In the first part, I would like to tell you about how we train GPT Assistants, and then in the second part, we're going to take a look at how we can use these assistants effectively for your applications.

First, let's take a look at the emerging recipe for how to train these assistants and keep in mind that this is all very new and still rapidly evolving, but so far, the recipe looks something like this. Now, this is a complicated slide, I'm going to go through it piece by piece, but roughly speaking, we have four major stages: pretraining, supervised finetuning, reward modeling, and reinforcement learning, and they follow each other serially. Now, in each stage, we have a dataset that powers that stage. We have an algorithm that for our purposes will be an objective over for training the neural network, and then we have a resulting model, and then there are some notes on the bottom.

The first stage we're going to start with is the pretraining stage. Now, this stage is special in this diagram, and this diagram is not to scale because this stage is where all of the computational work basically happens. This is 99 percent of the training compute time and also FLOPs. This is where we are dealing with Internet-scale datasets with thousands of GPUs in the supercomputer and also months of training potentially. The other three stages are finetuning stages that are much more along the lines of a small, few number of GPUs and hours or days.

Let's take a look at the pretraining stage to achieve a base model. First, we are going to gather a large amount of data. Here's an example of what we call a data mixture that comes from this paper that was released by Meta where they released this LLaMA-based model. Now, you can see roughly the datasets that enter into these collections. We have CommonCrawl, which is a web scrape, C4, which is also CommonCrawl, and then some high quality datasets as well. For example, GitHub, Wikipedia, Books, Archives, Stock Exchange and so on. These are all mixed up together, and then they are sampled according to some given proportions, and that forms the training set for the GPT.

Now before we can actually train on this data, we need to go through one more preprocessing step, and that is tokenization. This is basically a translation of the raw text that we scrape from the Internet into sequences of integers because that's the native representation over which GPTs function. Now, this is a lossless translation between pieces of texts and tokens and integers, and there are a number of algorithms for this stage. Typically, for example, you could use something like byte pair encoding, which iteratively merges text chunks and groups them into tokens. Here, I'm showing some example chunks of these tokens, and then this is the raw integer sequence that will actually feed into a transformer.

Now, here I'm showing two examples for hyperparameters that govern this stage. GPT-4, we did not release too much information about how it was trained and so on, I'm using GPT-3's numbers, but GPT-3 is of course a little bit old by now, about three years ago. But LLaMA is a fairly recent model from Meta. These are roughly the orders of magnitude that we're dealing with when we're doing pretraining. The vocabulary size is usually a couple tens of thousands of tokens. The context length is usually something like 2,000, 4,000, or nowadays even 100,000, and this governs the maximum number of integers that the GPT will look at when it's trying to predict the next integer in a sequence. You can see that roughly the number of parameters say, 65 billion for LLaMA. Now, even though LLaMA has only 65B parameters compared to GPT-3's 175 billion parameters, LLaMA is a significantly more powerful model, and intuitively, that's because the model is trained for significantly longer. In this case, 1.4 trillion tokens, instead of 300 billion tokens. You shouldn't judge the power of a model by the number of parameters that it contains. Below, I'm showing some tables of rough hyperparameters that typically go into specifying the transformer neural network, the number of heads, the dimension size, number of layers, and so on, and on the bottom I'm showing some training hyperparameters. For example, to train the 65B model, Meta used 2,000 GPUs, roughly 21 days of training and roughly several million dollars. That's the rough orders of magnitude that you should have in mind for the pre-training stage.

Now, when we're actually pre-training, what happens? Roughly speaking, we are going to take our tokens, and we're going to lay them out into data batches. We have these arrays that will feed into the transformer, and these arrays are $B$, the batch size—and these are all independent examples stacked up in rows—and $B \times T$, $T$ being the maximum context length. In my picture I only have 10 for the context length, so this could be 2,000, 4,000, etc. These are extremely long rows. What we do is we take these documents, and we pack them into rows, and we delimit them with these special end of text tokens, basically telling the transformer where a new document begins. Here, I have a few examples of documents and then I stretch them out into this input.

Now, we're going to feed all of these numbers into the transformer. Let me just focus on a single particular cell, but the same thing will happen at every cell in this diagram. Let's look at the green cell. The green cell is going to take a look at all of the tokens before it, so all of the tokens in yellow, and we're going to feed that entire context into the transformer neural network, and the transformer is going to try to predict the next token in a sequence, in this case in red. Now the transformer, I don't have too much time to, unfortunately, go into the full details of this neural network architecture—it's just a large blob of neural net stuff for our purposes, and it's got several, 10 billion parameters typically or something like that. Of course, as I tune these parameters, you're getting slightly different predicted distributions for every single one of these cells. For example, if our vocabulary size is 50,257 tokens, then we're going to have that many numbers because we need to specify a probability distribution for what comes next. Basically, we have a probability for whatever may follow. Now, in this specific example, for this specific cell, 513 will come next, and so we can use this as a source of supervision to update our transformer's weights. We're applying this basically on every single cell in parallel, and we keep swapping batches, and we're trying to get the transformer to make the correct predictions over what token comes next in a sequence.

Let me show you more concretely what this looks like when you train one of these models. This is actually coming from the New York Times, and they trained a small GPT on Shakespeare. Here's a small snippet of Shakespeare, and they train their GPT on it. Now, in the beginning, at initialization, the GPT starts with completely random weights. You're getting completely random outputs as well. But over time, as you train the GPT longer and longer,

You are getting more and more coherent and consistent samples from the model. And the way you sample from it, of course, is you predict what comes next, you sample from that distribution, and you keep feeding that back into the process, and you can basically sample large sequences. By the end, you see that the transformer has learned about words and where to put spaces and where to put commas, and so on. We're making more and more consistent predictions over time.

These are the plots that you are looking at when you're doing model pretraining. Effectively, we're looking at the loss function over time as you train, and low loss means that our transformer is giving a higher probability to the next correct integer in the sequence.

What are we going to do with the model once we've trained it after a month? Well, the first thing that we noticed — we, the field — is that these models, basically in the process of language modeling, learn very powerful general representations, and it's possible to very efficiently fine-tune them for any arbitrary downstream tasks you might be interested in. As an example, if you're interested in sentiment classification, the approach used to be that you collect a bunch of positives and negatives and then you train some NLP model for that. But the new approach is: ignore sentiment classification, go off and do large language model pretraining, train a large transformer, and then you may only have a few examples, and you can very efficiently fine-tune your model for that task.

This works very well in practice. The reason for this is that basically the transformer is forced to multitask a huge amount of tasks in the language modeling task, because in terms of predicting the next token, it's forced to understand a lot about the structure of the text and all the different concepts therein.

That was GPT-1. Now, around the time of GPT-2, people noticed that actually, even better than fine-tuning, you can actually prompt these models very effectively. These are language models and they want to complete documents, and you can actually trick them into performing tasks by arranging these fake documents. In this example, for instance, we have some passage and then we do QA, QA, QA. This is called a few-shot prompt. Then we do Q, and as the transformer tries to complete the document, it is actually answering our question. This is an example of a prompt-engineering-based model, making it believe that it's imitating a document and getting it to perform a task. This kicked off, I think, the era of — I would say — prompting over fine-tuning, and seeing that this can actually work extremely well on a lot of problems, even without training any neural networks, fine-tuning, or so on.

Now since then, we've seen an entire evolutionary tree of base models that everyone has trained. Not all of these models are available; for example, the GPT-4 base model was never released. The GPT-4 model that you might be interacting with over the API is not a base model, it's an assistant model, and we're going to cover how to get those in a bit. The GPT-3 base model is available via the API under the name Davinci, and the GPT-2 base model is available even as weights on our GitHub repo. But currently, the best available base model probably is the LLaMa series from Meta, although it is not commercially licensed.

Now, one thing to point out is: base models are not assistants. They don't want to make answers to your questions; they want to complete documents. If you tell them to write a poem about bread and cheese, it will answer questions with more questions — it's completing what it thinks is a document. However, you can prompt them in a specific way for base models that is more likely to work. As an example, here's a poem about bread and cheese, and in that case, it will autocomplete correctly. You can even trick base models into being assistants. The way you would do this is you would create a specific few-shot prompt that makes it look like there's some document between the human and assistant and they're exchanging information. Then at the bottom, you put your query at the end, and the base model will condition itself into being a helpful assistant and answer. But this is not very reliable and doesn't work super well in practice, although it can be done.

Instead, we have a different path to make actual GPT assistants, not base model document completers.

That takes us into supervised fine-tuning (SFT). In the supervised fine-tuning stage, we are going to collect small but high-quality datasets. In this case, we're going to ask human contractors to gather data of the form: prompt and ideal response. We're going to collect lots of these, typically tens of thousands or something like that.

Then we're going to still do language modeling on this data. Nothing changed algorithmically; we're swapping out a training set. It used to be internet documents, which has a high quantity locally for Q&A prompt-response data. That is low quantity, high quality. We will still do language modeling, and then after training, we get an SFT model. You can actually deploy these models, and they are actual assistants and they work to some extent.

Let me show you what an example demonstration might look like. Here's something that a human contractor might come up with: here's some random prompt ("Can you write a short introduction about the relevance of the term monopsony or something like that?"). Then the contractor also writes out an ideal response. When they write out these responses, they are following extensive labeling documentation, and they are being asked to be helpful, truthful, and harmless. These labeling instructions here — you probably can't read it, neither can I — but they're long, and this is people following instructions and trying to complete these prompts. That's what the dataset looks like. You can train these models, and this works to some extent.

Now, you can actually continue the pipeline from here on and go into RLHF (reinforcement learning from human feedback), which consists of both reward modeling and reinforcement learning.

Let me cover that, and then I'll come back to why you may want to go through the extra steps and how that compares to SFT models.

In the reward modeling step, what we're going to do is we're now going to shift our data collection to be of the form of comparisons. Here's an example of what our dataset will look like: I have the same identical prompt on the top, which is asking the assistant to write a program or a function that checks if a given string is a palindrome. Then what we do is we take the SFT model which we've already trained and we create multiple completions. In this case, we have three completions that the model has created, and then we ask people to rank these completions. If you stare at this for a while — and by the way, these are very difficult things to do, to compare some of these predictions; this can take people even hours for a single prompt-completion pair — but let's say we decided that one of these is much better than the others and so on, we rank them.

Then we can follow that with something that looks very much like a binary classification on all the possible pairs between these completions.

What we do now is, we lay out our prompt in rows, and the prompt is identical across all three rows here. It's all the same prompt, but the completion of this varies. The yellow tokens are coming from the SFT model. Then what we do is we append another special reward readout token at the end, and we basically only supervise the transformer at this single green token. The transformer will predict some reward for how good that completion is for that prompt, and basically it makes a guess about the quality of each completion. Then once it makes a guess for every one of them, we also have the ground truth.

which is telling us the ranking of them. We can actually enforce that some of these numbers should be much higher than others, and so on. We formulate this into a loss function and we train our model to make reward predictions that are consistent with the ground truth coming from the comparisons from all these contractors. That's how we train our reward model. That allows us to score how good a completion is for a prompt.

Once we have a reward model, we can't deploy this because this is not very useful as an assistant by itself, but it's very useful for the reinforcement learning stage that follows now. Because we have a reward model, we can score the quality of any arbitrary completion for any given prompt. What we do during reinforcement learning is we basically get, again, a large collection of prompts and now we do reinforcement learning with respect to the reward model. Here's what that looks like.

We take a single prompt, we lay it out in rows, and now we use basically the model we'd like to train which was initialized at SFT model to create some completions in yellow, and then we append the reward token again and we read off the reward according to the reward model, which is now kept fixed. It doesn't change any more. Now the reward model tells us the quality of every single completion for all these prompts and so what we can do is we can now just basically apply the same language modeling loss function, but we're currently training on the yellow tokens, and we are weighing the language modeling objective by the rewards indicated by the reward model.

As an example, in the first row, the reward model said that this is a fairly high-scoring completion and so all the tokens that we happen to sample on the first row are going to get reinforced and they're going to get higher probabilities for the future. Conversely, on the second row, the reward model really did not like this completion, -1.2. Therefore, every single token that we sampled in that second row is going to get a slightly higher probability for the future. We do this over and over on many prompts on many batches and basically, we get a policy that creates yellow tokens here. It's basically all the completions here will score high according to the reward model that we trained in the previous stage. That's what the RLHF pipeline is.

Then at the end, you get a model that you could deploy. As an example, ChatGPT is an RLHF model, but some other models that you might come across for example, Vicuna-13B, and so on, these are SFT models. We have base models, SFT models, and RLHF models. That's the state of things there.

Now why would you want to do RLHF? One answer that's not that exciting is that it works better. This comes from the InstructGPT paper. According to these experiments a while ago now, these PPO models are RLHF. We see that they are basically preferred in a lot of comparisons when we give them to humans. Humans prefer basically tokens that come from RLHF models compared to SFT models, compared to base model that is prompted to be an assistant. It just works better.

But you might ask why does it work better? I don't think that there's a single amazing answer that the community has really agreed on, but I will offer one reason potentially. It has to do with the asymmetry between how easy computationally it is to compare versus generate. Let's take an example of generating a haiku. Suppose I ask a model to write a haiku about paper clips. If you're a contractor trying to train data, then imagine being a contractor collecting basically data for the SFT stage, how are you supposed to create a nice haiku for a paper clip? You might not be very good at that, but if I give you a few examples of haikus you might be able to appreciate some of these haikus a lot more than others. Judging which one of these is good is a much easier task. Basically, this asymmetry makes it so that comparisons are a better way to potentially leverage yourself as a human and your judgment to create a slightly better model.

Now, RLHF models are not strictly an improvement on the base models in some cases. In particular, we'd notice for example that they lose some entropy. That means that they give more peaky results. They can output samples with lower variation than the base model. The base model has lots of entropy and will give lots of diverse outputs. For example, one place where I still prefer to use a base model is in the setup where you basically have $N$ things and you want to generate more things like it.

Here is an example that I just cooked up. I want to generate cool Pokémon names. I gave it seven Pokémon names and I asked the base model to complete the document and it gave me a lot more Pokémon names. These are fictitious. I tried to look them up. I don't believe they're actual Pokémons. This is the task that I think the base model would be good at because it still has lots of entropy. It'll give you lots of diverse cool more things that look like whatever you give it before.

Having said all that, these are the assistant models that are probably available to you at this point. There was a team at Berkeley that ranked a lot of the available assistant models and give them basically Elo ratings. Currently, some of the best models, of course, are GPT-4, by far, I would say, followed by Claude, GPT-3.5, and then a number of models, some of these might be available as weights, like Vicuna, Koala, etc. The first three rows here are all RLHF models and all of the other models to my knowledge, are SFT models, I believe.

That's how we train these models on the high level. Now I'm going to switch gears and let's look at how we can best apply the GPT assistant model to your problems.

Now, I would like to work in setting of a concrete example. Let's work with a concrete example here. Let's say that you are working on an article or a blog post, and you're going to write this sentence at the end: "California's population is 53 times that of Alaska." So for some reason, you want to compare the populations of these two states. Think about the rich internal monologue and tool use and how much work actually goes computationally in your brain to generate this one final sentence. Here's maybe what that could look like in your brain.

For this next step, let me blog on my blog, let me compare these two populations. First I'm going to obviously need to get both of these populations. Now, I know that I probably don't know these populations off the top of my head so I'm aware of what I know or don't know of my self-knowledge. I go, I do some tool use and I go to Wikipedia and I look up California's population and Alaska's population. Now, I know that I should divide the two, but again, I know that dividing 39.2 by 0.74 is very unlikely to succeed. That's not the thing that I can do in my head and so therefore, I'm going to rely on the calculator so I'm going to use a calculator, punch it in and see that the output is roughly 53. Then maybe I do some reflection and sanity checks in my brain so does 53 make sense? Well, that's quite a large fraction, but then California is the most populous state, so maybe that looks okay. Then I have all the information I might need, and now I get to the creative portion of writing. I might start to write something like "California has 53x times greater" and then I think to myself, that's actually like really awkward phrasing so let me actually delete that and let me try again. As I'm writing, I have this separate process, almost inspecting what I'm writing and judging whether it looks good or not and then maybe I delete and maybe I reframe it.

and then maybe I'm happy with what comes out. Basically long story short, a ton happens under the hood in terms of your internal monologue when you create sentences like this. But what does a sentence like this look like when we are training a GPT on it? From GPT's perspective, this is just a sequence of tokens.

GPT, when it's reading or generating these tokens, it just goes chunk, chunk, chunk, chunk and each chunk is roughly the same amount of computational work for each token. These transformers are not very shallow networks; they have about 80 layers of reasoning, but 80 is still not like too much. This transformer is going to do its best to imitate, but of course, the process here looks very different from the process that you took.

In particular, in our final artifacts, in the datasets that we create, and then eventually feed to LLMs, all that internal dialogue was completely stripped. And unlike you, the GPT will look at every single token and spend the same amount of compute on every one of them. So, you can't expect it to do too much work per token.

Basically these transformers are just like token simulators; they don't know what they don't know. They just imitate the next token. They don't know what they're good at or not good at. They just try their best to imitate the next token. They don't reflect in the loop. They don't sanity check anything. They don't correct their mistakes along the way. By default, they just sample token sequences. They don't have separate inner monologue streams in their head, right? They're not evaluating what's happening.

Now, they do have some cognitive advantages, I would say, and that is that they do actually have a very large fact-based knowledge across a vast number of areas because they have, say, several 10 billion parameters. That's a lot of storage for a lot of facts. They also, I think, have a relatively large and perfect working memory. Whatever fits into the context window is immediately available to the transformer through its internal self-attention mechanism, so it's a perfect memory. It's got a finite size, but the transformer has a very direct access to it, and so it can losslessly remember anything that is inside its context window.

This is how I would compare those two, and the reason I bring all of this up is because I think to a large extent, prompting is just making up for this cognitive difference between these two architectures — our brains here and LLM brains. You can look at it that way almost.

Here's one thing that people found, for example, works pretty well in practice. Especially if your tasks require reasoning, you can't expect the transformer to do too much reasoning per token. You have to really spread out the reasoning across more and more tokens. For example, you can't give a transformer a very complicated question and expect it to get the answer in a single token. There's just not enough time for it.

"These transformers need tokens to think," I like to say sometimes. Here are some of the things that work well: you may, for example, have a few-shot prompt that shows the transformer that it should show its work when it's answering a question. And if you give a few examples, the transformer will imitate that template, and it will just end up working out better in terms of its evaluation.

Additionally, you can elicit this behavior from the transformer by saying, let's think step-by-step. Because this conditions the transformer into showing its work, and because it snaps into a mode of showing its work, it's going to do less computational work per token. It's more likely to succeed as a result because it's making slower reasoning over time.

Here's another example; this one is called self-consistency. We saw that we had the ability to start writing and then if it didn't work out, I can try again and I can try multiple times and maybe select the one that worked best. In these approaches, you may sample not just once, but you may sample multiple times and then have some process for finding the ones that are good and then keeping just those samples or doing a majority vote or something like that.

Basically, these transformers in the process as they predict the next token, just like you, they can get unlucky and they could sample a not-very-good token and they can go down a blind alley in terms of reasoning. Unlike you, they cannot recover from that. They are stuck with every single token they sample, and so they will continue the sequence, even if they know that this sequence is not going to work out. You have to give them the ability to look back, inspect, or try to basically sample around it.

Here's another technique also: it turns out that actually LLMs do know when they've screwed up. So as an example, say you ask the model to generate a poem that does not rhyme, and it might give you a poem, but it actually rhymes. But it turns out that especially for the bigger models like GPT-4, you can just ask it, "did you meet the assignment?" Actually, GPT-4 knows very well that it did not meet the assignment. It just got unlucky in its sampling. It will tell you, "No, I didn't actually meet the assignment here. Let me try again." But without you prompting it, it doesn't know to revisit, and so on.

You have to make up for that in your prompts, and you have to get it to check. If you don't ask it to check, it's not going to check by itself — it's just a token simulator.

I think more generally, a lot of these techniques fall into the bucket of what I would say recreating our System 2. You might be familiar with System 1 and System 2 thinking for humans. System 1 is a fast, automatic process, and I think it corresponds to an LLM just sampling tokens. System 2 is the slower, deliberate planning part of your brain.

This is a paper actually from just last week, because this space is pretty quickly evolving, called Tree of Thoughts. The authors of this paper proposed maintaining multiple completions for any given prompt and then scoring them along the way and keeping the ones that are going well, if that makes sense. A lot of people are really playing around with prompt engineering to basically bring back some of these abilities that we have in our brains to LLMs.

Now, one thing I would like to note here is that this is not just a prompt. This is actually prompts that are used together with some Python glue code, because you actually have to maintain multiple prompts and you also have to do some tree search algorithm here to figure out which prompts to expand, etc. It's a symbiosis of Python glue code and individual prompts that are called in a while loop or in a bigger algorithm.

I also think there's a really cool parallel here to AlphaGo. AlphaGo has a policy for placing the next stone when it plays Go, and its policy was trained originally by imitating humans. But in addition to this policy, it also does Monte Carlo Tree Search. Basically, it will play out a number of possibilities in its head and evaluate all of them and only keep the ones that work well. I think this is the equivalent of AlphaGo, but for text, if that makes sense.

Just like Tree of Thoughts, I think more generally people are starting to really explore more general techniques of not just simple question-answer prompts, but something that looks a lot more like Python glue code stringing together many prompts. On the right, I have an example from this paper called ReAct where they...

structure the answer to a prompt as a sequence of thought-action-observation, thought-action-observation, and it's a full rollout and a thinking process to answer the query. In these actions, the model is also allowed to tool use.

On the left, I have an example of AutoGPT. Now AutoGPT by the way is a project that I think got a lot of hype recently, but I think I still find it inspirationally interesting. It's a project that allows an LLM to keep the task list and continue to recursively break down tasks. I don't think this currently works very well and I would not advise people to use it in practical applications. I just think it's something to generally take inspiration from in terms of where this is going, I think over time.

That's like giving our model System 2 thinking.

The next thing I find interesting is, this following serve I would say almost psychological quirk of LLMs, is that LLMs don't want to succeed, they want to imitate. You want to succeed, and you should ask for it. What I mean by that is, when transformers are trained, they have training sets and there can be an entire spectrum of performance qualities in their training data. For example, there could be some kind of a prompt for some physics question or something like that, and there could be a student's solution that is completely wrong but there can also be an expert answer that is extremely right. Transformers can't tell the difference between low, they know about low-quality solutions and high-quality solutions, but by default, they want to imitate all of it because they're just trained on language modeling. At test time, you actually have to ask for a good performance.

In this example in this paper, they tried various prompts. Let's think step-by-step was very powerful because it spread out the reasoning over many tokens. But what worked even better is, let's work this out in a step-by-step way to be sure we have the right answer. It's like conditioning on getting the right answer, and this actually makes the transformer work better because the transformer doesn't have to now hedge its probability mass on low-quality solutions, as ridiculous as that sounds. Basically, feel free to ask for a strong solution. Say something like, you are a leading expert on this topic. Pretend you have IQ 120, etc. But don't try to ask for too much IQ because if you ask for IQ 400, you might be out of data distribution, or even worse, you could be in data distribution for something like sci-fi stuff and it will start to take on some sci-fi, or like roleplaying or something like that. You have to find the right amount of IQ. I think it's got some U-shaped curve there.

Next up, as we saw when we are trying to solve problems, we know what we are good at and what we're not good at, and we lean on tools computationally. You want to do the same potentially with your LLMs. In particular, we may want to give them calculators, code interpreters, and so on, the ability to do search, and there's a lot of techniques for doing that. One thing to keep in mind, again, is that these transformers by default may not know what they don't know. You may even want to tell the transformer in a prompt you are not very good at mental arithmetic. Whenever you need to do very large number addition, multiplication, or whatever, instead, use this calculator. Here's how you use the calculator, you use this token combination, etc. You have to actually spell it out because the model by default doesn't know what it's good at or not good at, necessarily, just like you and I might be.

Next up, I think something that is very interesting is we went from a world that was retrieval only all the way, the pendulum has swung to the other extreme where its memory only in LLMs. But actually, there's this entire space in-between of these retrieval-augmented models and this works extremely well in practice. As I mentioned, the context window of a transformer is its working memory. If you can load the working memory with any information that is relevant to the task, the model will work extremely well because it can immediately access all that memory. I think a lot of people are really interested in basically retrieval-augmented generation.

On the bottom, I have an example of LlamaIndex which is one data connector to lots of different types of data. You can index all of that data and you can make it accessible to LLMs. The emerging recipe there is you take relevant documents, you split them up into chunks, you embed all of them, and you basically get embedding vectors that represent that data. You store that in the vector store and then at test time, you make some kind of a query to your vector store and you fetch chunks that might be relevant to your task and you stuff them into the prompt and then you generate. This can work quite well in practice. This is, I think, similar to when you and I solve problems. You can do everything from your memory and transformers have very large and extensive memory, but also it really helps to reference some primary documents. Whenever you find yourself going back to a textbook to find something, or whenever you find yourself going back to documentation of the library to look something up, transformers definitely want to do that too. You have some memory over how some documentation of the library works but it's much better to look it up. The same applies here.

Next, I wanted to briefly talk about constraint prompting. I also find this very interesting. This is basically techniques for forcing a certain template in the outputs of LLMs. Guidance is one example from Microsoft actually. Here we are enforcing that the output from the LLM will be JSON. This will actually guarantee that the output will take on this form because they go in and they mess with the probabilities of all the different tokens that come out of the transformer and they clamp those tokens and then the transformer is only filling in the blanks here, and then you can enforce additional restrictions on what could go into those blanks. This might be really helpful, and I think this constraint sampling is also extremely interesting.

I also want to say a few words about fine tuning. It is the case that you can get really far with prompt engineering, but it's also possible to think about fine tuning your models. Now, fine tuning models means that you are actually going to change the weights of the model. It is becoming a lot more accessible to do this in practice, and that's because of a number of techniques that have been developed and have libraries for very recently.

So for example parameter efficient fine tuning techniques like LoRA, make sure that you're only training small, sparse pieces of your model. So most of the model is kept clamped at the base model and some pieces of it are allowed to change and this still works pretty well empirically and makes it much cheaper to tune only small pieces of your model. It also means that because most of your model is clamped, you can use very low precision inference for computing those parts because you are not going to be updated by gradient descent and so that makes everything a lot more efficient as well.

And in addition, we have a number of open source, high-quality base models. Currently, as I mentioned, I think LLaMA is quite nice, although it is not commercially licensed, I believe right now. Some things to keep in mind is that basically fine tuning is a lot more technically involved. It requires a lot more, I think, technical expertise to do right. It requires human data contractors for datasets and/or synthetic data pipelines that can be pretty complicated. This will definitely slow down your iteration cycle by a lot, and I would say on a high level SFT is achievable because you're continuing...

the language modeling task. It's relatively straightforward, but RLHF, I would say is very much research territory and is even much harder to get to work, and so I would probably not advise that someone just tries to roll their own RLHF implementation. These things are pretty unstable, very difficult to train, not something that is, I think, very beginner-friendly right now, and it's also potentially likely to change pretty rapidly still.

So I think these are my default recommendations right now. I would break up your task into two major parts:

1. Achieve your top performance, and
2. Optimize your performance, in that order.

First, the best performance will currently come from the GPT-4 model. It is the most capable of all by far. Use prompts that are very detailed; they have lots of task content, relevant information, and instructions. Think along the lines of what you would tell a task contractor if they can't email you back, but then also keep in mind that a task contractor is a human and they have an inner monologue, they're very clever, etc. LLMs do not possess those qualities. So make sure to think through the psychology of the LLM almost, and cater prompts to that. Retrieve and add any relevant context and information to these prompts. Basically, refer to a lot of the prompt engineering techniques. Some of them I've highlighted in the slides above, but also this is a very large space and I would just advise you to look for prompt engineering techniques online. There's a lot to cover there.

Experiment with few-shot examples. What this refers to is, you don't just want to tell, you want to show whenever it's possible. So give it examples of everything that helps it really understand what you mean if you can. Experiment with tools and plugins to offload tasks that are difficult for LLMs natively, and then think about not just a single prompt and answer, think about potential chains and reflection, and how you glue them together and how you can potentially make multiple samples and so on.

Finally, if you think you've squeezed out prompt engineering, which I think you should stick with for a while, look at potentially fine-tuning a model to your application, but expect this to be a lot more slower and involved. Then there's an expert, fragile research zone here, and I would say that is RLHF, which currently does work a bit better than SFT if you can get it to work. But again, this is pretty involved, I would say. And to optimize your costs, try to explore lower-capacity models or shorter prompts and so on.

I also wanted to say a few words about the use cases in which I think LLMs are currently well-suited. In particular, note that there's a large number of limitations to LLMs today, and so I would keep that definitely in mind for all of your applications. Models—and this by the way could be an entire talk,

so I don't have time to cover it in full detail— may be biased, they may fabricate or hallucinate information, they may have reasoning errors, they may struggle in entire classes of applications, they have knowledge cut-offs, so they might not know any information above, say, September 2021. They are susceptible to a large range of attacks which are coming out on Twitter daily, including prompt injection, jailbreak attacks, data poisoning attacks, and so on.

So my recommendation right now is to use LLMs in low-stakes applications. Combine them always with human oversight. Use them as a source of inspiration and suggestions, and think of them as co-pilots instead of completely autonomous agents that are just performing a task somewhere. It's just not clear that the models are there right now.

So I wanted to close by saying that GPT-4 is an amazing artifact. I'm very thankful that it exists, and it's beautiful. It has a ton of knowledge across so many areas. It can do math, code, and so on. And in addition, there's this thriving ecosystem of everything else that is being built and incorporated into the ecosystem. Some of these things I've talked about, and all of this power is accessible at your fingertips.

So here's everything that's needed in terms of code to ask GPT-4 a question, to prompt it, and get a response. In this case, I said, "Can you say something to inspire the audience of Microsoft Build 2023?" And I just punched this into Python and verbatim GPT-4 said the following: And by the way, I did not know that they used this trick in the keynote. So I thought I was being clever, but it is really good at this.

It says: "Ladies and gentlemen, innovators and trailblazers of Microsoft Build 2023, welcome to the gathering of brilliant minds like no other. You are the architects of the future, the visionaries molding the digital realm in which humanity thrives. Embrace the limitless possibilities of technology and let your ideas soar as high as your imagination. Together, let's create a more connected, remarkable, and inclusive world for generations to come. Get ready to unleash your creativity, canvas the unknown, and turn dreams into reality. Your journey begins today!"