Blog
Announcing my new side-project and third non-fiction, technical book: Elixir Chatbot Alchemy!
Following the same engineering ethos and exploratory learning approach that has proven so popular with readers of Northwind Elixir Traders, Elixir Chatbot Alchemy explores the implementation of a chatbot in Elixir. By utilizing the REST API of a self-hosted instance of Ollama and CPU inference on small(-ish) open-weight LLMs, we explore how LLMs operate, the impact of model choice on the quality of inference, and use Phoenix LiveView to build a variety of features, such as document summarization and a chatbot that can be used on a webpage, and an admin UI with reporting of the use of the chatbot.
Note: this is a super-early low-fanfare release, subject to heavy changes of scope and content.
Current content
- Requirements on the Elixir application (what we want to achieve)
- Choosing an LLM suitable for CPU inference
- Calling Ollama from Elixir
- Understanding Ollama’s HTTP response
- Fundamental knowledge of LLMs
- Short conversations and implementing a chat with completion
- Context truncation and periodic summarization
- Failure modes of completion
- Exceeding the context length
- Context compression through summarization
- GPU-based inference, parallel execution, and scaling
- Summarizing the summaries
- Sidequest: bigger models, better summaries?
- Context stuffing
Planned content
The following sections are envisioned within the scope of the book, but not yet set in stone.
- Implementing an actual chat
- Using LLM system prompts
- Storing chats in SQLite
- Tracking chat state with a GenServer
- Extracting and summarizing text from documents
- Categorizing a document
- Extracting structured JSON data from text
“Maybe” content
The following sections are speculative and any of those might be dropped entirely.
- Phoenix LiveView of a chat on a page
- Implementing an admin UI with reporting
Release schedule
This book is currently a work in progress at a very early stage, and new chapters are being written and released infrequently depending on my workload, as I’m currently focused on Phoenix Product Codex.
For early adopters
An early draft has been released on Leanpub–also note that early adopters benefit from lower prices while the book is in development.