WORKSHOP
J On the Beach
Small Models, Big Agents: Fine-Tuning and Serving SLMs on Kubernetes
LLM · Kubernetes · KEDA · Karpenter · AI
Small models are not a compromise. They are often the right tool. In this session we go end to end: fine tuning SLMs and a 70B model on Kubernetes, serving with llama.cpp, and autoscaling with KEDA and Karpenter. You will learn when to use small vs large models, how to optimize cost and performance, and how to run agentic workloads without overprovisioning.
Key Takeaways
- Match the model to the task: which agent steps suit small CPU models, and which genuinely need a larger LLM.
- Deploy efficient SLM inference on CPUs with llama.cpp.
- Build an agent whose routing, tool selection, argument extraction, and self-critique run on small CPU models, handing off to a larger model for open-ended reasoning when that's the right call.
- Train task-specific CPU specialists (classification, scoring, extraction, embedding) for each decision in the pipeline.
- Give agents safe, read-only tools so answers stay grounded in live cluster state.
- Right-size infrastructure with KEDA and Karpenter to balance cost and performance.
- Apply "right tool for the right task" thinking to production agentic AI.
Target Audience
- Platform Engineers
- Kubernetes Engineers
- MLOps Engineers
- AI/ML Engineers
- Cloud Architects
- DevOps Engineers
- Technical Leaders responsible for AI platforms
Requirements for Attendees
Technical Requirements
- Basic understanding of Kubernetes concepts.
- Familiarity with containers and cloud-native applications.
- General knowledge of Generative AI, LLMs, and AI agents.
Equipment Requirements
Laptop required.
Experience Level
Intermediate.