"Professional LLM Inference" — a practical four-week intensive course for those who want to understand how to launch, optimize, and scale large language models in real production systems. The course covers the entire journey: from the fundamental principles of inference and GPU memory management to distributed deployment, edge inference, quantization, profiling, and building high-performance AI services capable of handling large volumes of requests with low latency.
About the Course
The course is dedicated to professional inference of large language models: how LLMs function post-training, how they process user requests, the reasons for latency, how to improve server throughput, and what engineering solutions are used in modern AI companies.
The program combines theory, lab work, live demonstrations, and hands-on practice on real equipment. Sessions are conducted by Dr. Raj Dandekar, MIT PhD, as well as engineers and specialists from Anthropic, NVIDIA, Apple, Microsoft, Amazon, AnyScale, and other tech companies.
What You Will Learn
- Understand the architecture of modern LLM inference systems and the key stages of request processing.
- Optimize the performance of large language models in terms of latency, throughput, and GPU utilization.
- Work with GPU memory, KV cache, attention mechanisms, and hardware platform limitations.
- Apply model quantization for speeding up inference and reducing resource consumption.
- Use popular frameworks and tools: vLLM, SGLang, FlashAttention, TensorRT-LLM, Ray Serve, and Megatron-LM.
- Deploy LLMs on a local computer, server, Raspberry Pi 4, Android device, and NVIDIA Jetson Orin Nano.
- Design production-ready AI services with scalable architecture.
- Prepare for technical interviews that test understanding of LLM inference system design.
Program Structure
The course consists of two independent modules. The first module focuses on inference principles and optimization, the second on industrial deployment and building applied AI systems.
Module 1. LLM Inference Architecture and Optimization
In the first part, you will learn how the inference of large language models is structured, what bottlenecks occur during request processing, and how modern frameworks help increase the efficiency of models.
- Basics of LLM inference and the request lifecycle.
- Prefill, decode, batching, and continuous batching.
- KV cache and memory usage optimization.
- Attention mechanisms and FlashAttention.
- Model quantization and trade-offs between speed, quality, and memory consumption.
- Performance profiling and identifying bottleneck components.
- Practical work with vLLM, SGLang, TensorRT-LLM, and other tools.
Module 2. Production Deployment and Edge Inference
In the second part of the course, you will transition from optimizing individual models to creating full-fledged AI services that can be used in real products and infrastructure.
- Industrial deployment of LLM services.
- Distributed computing and inference scaling.
- Ray Serve and approaches to high-load request handling.
- Edge inference on compact devices.
- Running models on Raspberry Pi 4, Android, and NVIDIA Jetson Orin Nano.
- Comparison of hardware platforms and performance analysis.
- Building reliable production-ready AI applications.
Practice and Lab Work
A large part of the course is built around practical assignments. Each study day is accompanied by lab work in Google Colab, visual materials, demonstrations, and analysis of engineering solutions.
You will not only study theory but also run models, measure their performance, analyze resource usage, compare different configurations, and apply optimization methods in practice.
Equipment and Platforms
- Local computer for basic model launching and testing.
- Google Colab for lab work and experiments.
- Raspberry Pi 4 for studying edge inference limitations.
- Android device for running LLM on a mobile platform.
- NVIDIA Jetson Orin Nano for practice with a compact GPU accelerator.
Final Projects
During the course, you will implement two complete projects that will help reinforce your skills in developing and optimizing LLM inference systems.
Project 1. High-Performance LLM Inference Server
You will go through the entire process from the model's raw weights to an optimized inference service. Along the way, you will set up model launching, conduct profiling, identify bottlenecks, apply acceleration techniques, and prepare the system for handling real requests.
- Loading and preparing the model.
- Setting up the inference server.
- Optimizing latency and throughput.
- Analyzing the performance of each component.
- Preparing the service for production scenarios.
Project 2. AI Assistant for WhatsApp
The second project is dedicated to creating an intelligent AI assistant for WhatsApp. It will process dialogues, generate responses, and improve its behavior using reinforcement learning based on user interactions.
- Integrating LLM with a chat interface.
- Developing the logic for the AI assistant.
- Using dialogues to improve response quality.
- Applying reinforcement learning approaches in an applied scenario.
Tools and Technologies
During the training, you will become familiar with a modern tool stack used to accelerate and scale the inference of large language models.
- vLLM — high-performance inference engine for LLM servicing.
- SGLang — framework for efficient execution of LLM applications.
- FlashAttention — optimized attention mechanisms for accelerating transformer work.
- TensorRT-LLM — NVIDIA tools for optimizing large model inference.
- Ray Serve — solution for scalable AI model serving.
- Megatron-LM — framework for working with large language models and distributed computing.
- Google Colab — environment for lab work and experiments.
Who the Course is For
- ML engineers looking to delve deeper into LLM optimization and deployment.
- Backend and infrastructure engineers working with AI services and high-load systems.
- Data scientists wishing to transition from model experiments to production development.
- AI developers creating chatbots, assistants, and LLM applications.
- Technical specialists preparing for interviews with AI companies.
- Founders and technical leaders who need to understand the cost, limitations, and architecture of LLM inference.
What Results You Will Achieve
After completing the course, you will be able to design and implement LLM inference systems considering requirements for speed, cost, scalability, and reliability.
- Understanding the internal workings of LLMs during inference.
- Skills to optimize models for different hardware platforms.
- Experience with industry-standard tools for LLM serving.
- Ability to conduct profiling and make engineering decisions based on metrics.
- Two practical projects for your portfolio.
- More confident preparation for technical interviews on LLM infrastructure.
Why Study LLM Inference
The development of large language models is not just about training neural networks. In real products, the main cost and complexity often relate to inference: response speed, GPU load, scaling, memory consumption, and service stability.
Specialists who understand how to effectively service LLMs in production are in demand in teams creating AI assistants, corporate chatbots, agent systems, search products, developer tools, and other applications based on generative artificial intelligence.
