# Foundations of LLMs Lesson 1 **Learning objectives:** * Have a good mental model of when to train or fine-tune LLMs * Understand high level the key pieces to make it work successfully * Understand why and how to participate in the Neurips LLM Efficiency Challenge ### Introduction

LLM keyphrase usage in arxiv papers from 2020

Many are using LLMs in

LLMs are decoder-only models. Trained to predict the next word in the sequence. ### When to Train or fine-tune?

Slide on resons for finetuning LLMs

* Pre-train/ train the LLM when you want to have full control over the training data. * Finetune the open-source LLM when you have to have more control but want to have cheaper inference. * Use commercial APIs when you have to reduce the time to market. #### Finetuning the model --Types: * **Task specific**: When we have a task specified we can finetune the model to do that task based on a certain prompt or instruction. ![](https://748712899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZcqmeHkGECPaayJZTNEt%2Fuploads%2FdREk25WbTfvzd87EhWNn%2Fimage.png?alt=media\&token=9a05ba22-bea0-44e4-8440-044d6cc45a37) * **Instruction tuning and RLHF** \ instruction tuning: We give text as the instruction and expect the model to follow that instruction. * **Supervised finetuning:** We provide the model with instructions and responses and train the model to behave in a particular way. * **RLHF Reinforcement learning with human feedback:** We have two model techniques. involves training the reward model where we teach that reward model to what humans prefer. We train the LLM with reinforcement learning to align with human preferences. \ ![](https://748712899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZcqmeHkGECPaayJZTNEt%2Fuploads%2FoGaL4xvtOZEkxbkDpNfm%2Fimage.png?alt=media\&token=e90a29d9-236d-49cb-a7dc-6f9f89dc22a2) #### Pretraining vs finetuning Chatgpt started as the code model trained to help with programming Then in further iterations it was trained with RLHF (technique above discussed) for aligning more with human responses.

pretraining vs finetuning slide.

#### Some of the open-source models released which we can finetune today!
#### What do we need for finetuning? 1. What is the goal and what are the evaluation criteria? 2. Choose model architecture and foundation model 3. Create the right dataset. 4. Efficient training and finetuning of the model on data. ### Introduction to Neurips LLM Efficiency Challenge
![](https://748712899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZcqmeHkGECPaayJZTNEt%2Fuploads%2FGjAOhxK4nZJbK5L8VWPt%2Fimage.png?alt=media\&token=fe3e050b-2758-4185-a535-e806d2134390) #### Finetuning LLMs * Quantization ( to fit the model into limited memory ) * PEFT ( instead of training the whole model we train some part or append model with new parameters) ( see images below) * Techniques by model structure * adapters * LORA Low-rank matrix adaptation * QLORA Quantized low-rank matrix adaptation(using lower precision for finetuning the LLM) * Techniques by feeding data in different types * Prompt tuning * Prefix tuning * P-tuning ![](https://748712899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZcqmeHkGECPaayJZTNEt%2Fuploads%2FXez7Svwyayxb5X9KYhH9%2Fimage.png?alt=media\&token=f1cd8af6-535c-4fb0-bb1b-118fdc8e22e5) #### Data curation ![](https://748712899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZcqmeHkGECPaayJZTNEt%2Fuploads%2FR3aXkfbnPt2o7u3XBk2N%2Fimage.png?alt=media\&token=612d3ee1-f47d-4dd2-9ddf-100bc6c65f09) ![](https://748712899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZcqmeHkGECPaayJZTNEt%2Fuploads%2FthioGTwJfL8AN9NjsNQU%2Fimage.png?alt=media\&token=88097245-566f-4a9d-81b7-57f8cf929ebc)![](https://748712899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZcqmeHkGECPaayJZTNEt%2Fuploads%2FZB3YFYaHZLye3Z6L3CEH%2Fimage.png?alt=media\&token=0cbe1bbd-dc3d-4e15-98e6-dd892788cb4e) ![](https://748712899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZcqmeHkGECPaayJZTNEt%2Fuploads%2FgrU38wwYEjiLCRcYOIUQ%2Fimage.png?alt=media\&token=dc5a7252-b01d-4e9b-af06-790667ff5fd2) ### Code to start experimenting {% embed url="" %} {% embed url="" %} Link to youtube recording {% endembed %} {% embed url="" %} Course page link {% endembed %}