LLM-engineer-handbook

🔥 Large Language Models(LLM) have taken the NLP community AI community the Whole World by storm. The LLM space is complicated! This repo provides a curated list to help you navigate; it includes a collection of Large Language Model frameworks and tutorials, covering model training, serving, fine-tuning, and building LLM applications.

Table of Content

• LLM-engineer-handbook
  • Table of Content
  • Applications
  • Pretraining
  • Fine-tuning
  • Serving
  • Prompt Management
  • Datasets
  • Benchmarks
  • Understand LLM
  • Learn LLM
    • Modeling
    • Training
    • Fundamentals
    • Books
    • Applications
      • General
      • Agent
    • Auto-optimization
  • Social Accounts
  • Community
  • Contributing

Applications

Build & Auto-optimize

• AdalFlow - The library to build & auto-optimize LLM applications, from Chatbot, RAG, to Agent. It is AI-first with PyTorch-like design patterns.

• dspy - DSPy: The framework for programming—not prompting—foundation models.

Build

• LlamaIndex — A Python library for augmenting LLM apps with data.
• LangChain — A popular Python/JavaScript library for chaining sequences of language model prompts.
• Haystack - Python framework that allows you to build applications powered by LLMs

Prompt Optimization

• AutoPrompt - A framework for prompt tuning using Intent-based Prompt Calibration
• PromptFify - A library for prompt engineering that simplifies NLP tasks (e.g., NER, classification) using LLMs like GPT.

Others

• LiteLLM - Python SDK, Proxy Server (LLM Gateway) to call 100+ LLM APIs in OpenAI format.

Pretraining

• PyTorch - PyTorch is an open source machine learning library based on the Torch library, used for applications such as computer vision and natural language processing.
• TensorFlow - TensorFlow is an open source machine learning library developed by Google.
• JAX - Google’s library for high-performance computing and automatic differentiation.
• tinygrad - A minimalistic deep learning library with a focus on simplicity and educational use, created by George Hotz.
• micrograd - A simple, lightweight autograd engine for educational purposes, created by Andrej Karpathy.

Fine-tuning

• Transformers - Hugging Face Transformers is a popular library for Natural Language Processing (NLP) tasks, including fine-tuning large language models.
• Unsloth - Finetune Llama 3.2, Mistral, Phi-3.5 & Gemma 2-5x faster with 80% less memory!
• LitGPT - 20+ high-performance LLMs with recipes to pretrain, finetune, and deploy at scale.

Serving

• TorchServe - An open-source model serving library developed by AWS and Facebook specifically for PyTorch models, enabling scalable deployment, model versioning, and A/B testing.

• TensorFlow Serving - A flexible, high-performance serving system for machine learning models, designed for production environments, and optimized for TensorFlow models but also supports other formats.

• Ray Serve - Part of the Ray ecosystem, Ray Serve is a scalable model-serving library that supports deployment of machine learning models across multiple frameworks, with built-in support for Python-based APIs and model pipelines.

• NVIDIA TensorRT-LLM - TensorRT-LLM is NVIDIA's compiler for transformer-based models (LLMs), providing state-of-the-art optimizations on NVIDIA GPUs.

• NVIDIA Triton Inference Server - A high-performance inference server supporting multiple ML/DL frameworks (TensorFlow, PyTorch, ONNX, TensorRT etc.), optimized for NVIDIA GPU deployments, and ideal for both cloud and on-premises serving.

• ollama - A lightweight, extensible framework for building and running large language models on the local machine.

• llama.cpp - A library for running LLMs in pure C/C++. Supported architectures include (LLaMA, Falcon, Mistral, MoEs, phi and more)

• TGI - HuggingFace's text-generation-inference toolkit for deploying and serving LLMs, built on top of Rust, Python and gRPC.

• vllm - An optimized, high-throughput serving engine for large language models, designed to efficiently handle massive-scale inference with reduced latency.

• sglang - SGLang is a fast serving framework for large language models and vision language models.

• LitServe - LitServe is a lightning-fast serving engine for any AI model of any size. Flexible. Easy. Enterprise-scale.

Prompt Management

• Opik - Opik is an open-source platform for evaluating, testing and monitoring LLM applications

Datasets

Use Cases

• Datasets - A vast collection of ready-to-use datasets for machine learning tasks, including NLP, computer vision, and audio, with tools for easy access, filtering, and preprocessing.
• Argilla - A UI tool for curating and reviewing datasets for LLM evaluation or training.
• distilabel - A library for generating synthetic datasets with LLM APIs or models.

Fine-tuning

• LLMDataHub - A quick guide (especially) for trending instruction finetuning datasets
• LLM Datasets - High-quality datasets, tools, and concepts for LLM fine-tuning.

Pretraining

• IBM LLMs Granite 3.0 - Full list of datasets used to train IBM LLMs Granite 3.0

Benchmarks

• lighteval - A library for evaluating local LLMs on major benchmarks and custom tasks.

• evals - OpenAI's open sourced evaluation framework for LLMs and systems built with LLMs.

• ragas - A library for evaluating and optimizing LLM applications, offering a rich set of eval metrics.

Agent

• TravelPlanner - paper A Benchmark for Real-World Planning with Language Agents

Understand LLM

Prompt Engineering

• Brown, Tom B. "Language models are few-shot learners." arXiv preprint arXiv:2005.14165 (2020).

Reasoning & Planning

• Kambhampati, Subbarao, et al. "LLMs can't plan, but can help planning in LLM-modulo frameworks." arXiv preprint arXiv:2402.01817 (2024).

Learn LLM

Modeling

• Llama3 from scratch - llama3 implementation one matrix multiplication at a time with PyTorch.
• Interactive LLM visualization - An interactive visualization of transformers. Visualizer
• 3Blue1Brown transformers visualization - 3Blue1Brown's video on how transformers work.
• Self-Attention explained as directed graph - An X post explaining self-attention as a directed graph by Akshay Pachaar.

Training

• Chip's Blog - Chip Huyen's blog on training LLMs, including the latest research, tutorials, and best practices.
• Lil'Log - Lilian Weng(OpenAI)'s blog on machine learning, deep learning, and AI, with a focus on LLMs and NLP.
• Ahead of AI - Sebastian Raschka's Newsletter, covering end-to-end LLMs understanding.
• Decoding ML - Content on building production GenAI, RecSys and MLOps applications.

Fundamentals

• Intro to LLMs - A 1 hour general-audience introduction to Large Language Models by Andrej Karpathy.
• Building GPT-2 from Scratch - A 4 hour deep dive into building GPT2 from scratch by Andrej Karpathy.

Books

• Build a Large Language Model from Scratch by Sebastian Raschka

• Hands-On Large Language Models: Build, Tune, and Apply LLMs by Jay Alammar , Maarten Grootendorst

• LLM Engineer's Handbook: Master the art of engineering large language models from concept to production by Paul Iusztin , Maxime Labonne

• Generative Deep Learning - Teaching machines to Paint, Write, Compose and Play by David Foster

Applications

General

• AdalFlow documentation - Includes tutorials from building RAG, Agent, to LLM evaluation and fine-tuning.

• CS224N - Stanford course covering NLP fundamentals, LLMs, and PyTorch-based model building, led by Chris Manning and Shikhar Murty.

• LLM-driven Data Engineering - A playlist of 6 lectures by Zach Wilson on how LLMs will impact data pipeline development

• LLM Course by Maxime Labonne - An end-to-end course for AI and ML engineers on open source LLMs.

Agent

1. Lectures

• LLM Agents MOOC - A playlist of 11 lectures by the Berkeley RDI Center on Decentralization & AI, featuring guest speakers like Yuandong Tian, Graham Neubig, Omar Khattab, and others, covering core topics on Large Language Model agents.

1. Projects

• OpenHands - Open source agents for developers by AllHands.
• CAMEL - First LLM multi-agent framework and an open-source community dedicated to finding the scaling law of agents. by CAMEL-AI.
• swarm - Educational framework exploring ergonomic, lightweight multi-agent orchestration. Managed by OpenAI Solution team.
• AutoGen - A programming framework for agentic AI 🤖 by Microsoft.

Auto-optimization

• TextGrad - Automatic ''Differentiation'' via Text -- using large language models to backpropagate textual gradients.

Social Accounts

Name	Social	Expertise
Chip Huyen	LinkedIn	AI Engineering & ML Systems
Damien Benveniste, PhD	LinkedIn	ML Systems & MLOps
Jim Fan	LinkedIn	LLM Agents & Robotics
Li Yin	LinkedIn	LLM Engineering & Author of AdalFlow
Paul Iusztin	LinkedIn	LLM Engineering & LLMOps
Armand Ruiz	LinkedIn	AI Engineering Director at IBM
Alex Razvant	LinkedIn	AI/ML Engineering
Pascal Biese	LinkedIn	LLM Papers Daily
Maxime Labonne	LinkedIn	LLM Fine-Tuning
Sebastian Raschka	LinkedIn	LLMs from Scratch

Community

Name	Social	Scope
AdalFlow	Discord	LLM Engineering, auto-prompts, and AdalFlow discussions&contributions

Contributing

This is an active repository and your contributions are always welcome!

I will keep some pull requests open if I'm not sure if they are not an instant fit for this repo, you could vote for them by adding 👍 to them.

---

If you have any question about this opinionated list, do not hesitate to contact Li Yin