Elixir File Browsing: my new book on developing a REST API client for self-hosted file management

Announcing my new side-project and fourth non-fiction, technical book: Elixir File Browsing.

Following the same engineering ethos and exploratory learning approach that has proven so popular with readers of Northwind Elixir Traders and Phoenix Product Codex, in Elixir File Browsing we deploy File Browser and then iteratively develop ExFileBrowser, a full-featured client for File Browser’s undocumented REST API that we figure out by looking at File Browser’s source code.

Why is this book being written?

1. Because I urgently need simple file-management functionality for my Phoenix LiveView apps with a user-friendly web UI.
2. Because I don’t enjoy the hassle of deploying and babying Minio or Garage to have files stored opaquely.
3. Because I don’t want to have to pay a monthly fee for S3-compatible object storage for my apps.
4. Because writing a fully-featured REST API client in Elixir is a great exercise that hones your Elixir skills.
5. Because the Elixir ecosystem deserves more hands-on guides on end-to-end projects with an exploratory learning approach.
6. Beause Elixir, its ecosystem and community are awesome.

State of development

The book is currently around 55% complete, but I have not yet released it on Leanpub or elsewhere.

Completed content

• Using podman to self-host a File Browser server
• Investigating the undocumented File Browser REST API by looking at its Go codebase
• Jobs To Be Done by the users of our API client
• First draft of the Client module of ExFileBrowser
• Implementing authentication
• Seamless sessions with JWT token renewals
• Cleaning up the Client module with a function pipeline
• Using Ecto schemaless changesets to improve the Client module
• A layered architecture for ExFileBrowser
• Rewriting the Client module for good
• Fleshing out the Request HTTP utility module
• Cleaning up authentication-related functions
• Preparing an executing requests with Req
• Dealing with any public API route
• Further improvements to authentication
• Dispatching a request
• Automatic (re-)authentication and renewal
• Restructuring our context modules
• The Operations module for expert use
• Authentication as an operation
• Uploading a file
• Streaming file contents
• Making sure we don’t corrupt data on upload
• POSTing a local file
• GETting the metadata of a resource
• POSTing a remote file from a URL
• Streaming in and streaming out
• Elixir streams, and Stream vs. Enum
• Transforming the async request’s response body into a stream
• Lots and lots of periodic refactoring
• Downloading files from File Browser
• Enumerables and Collectables
• Separating Req options from other options
• Getting metadata after uploading a file
• Getting the storage space usage from File Browser
• Preventing or allowing uploads and downloads based on File Browser server or local disk space

Draft content

The following sections are in early-draft state:

• Managing File Browser users
• Managing File Browser settings

Planned content

The following sections are planned and will certainly be written:

• Data integrity verification on upload and download
• Downloading entire folders
• Downloading entire folders in archive format
• Uploading entire directories
• Making ExFileBrowser configurable for an application

“Maybe” content

The following sections are speculative and any of those might be dropped entirely.

• Implementing a Phoenix LiveView component to pick resources from the File Browser server
• Proxying requests to the File Browser server with Phoenix
• Storing file metadata on an SQLite database
• Implementing a file taxonomy (tags and categories/subcategories)
• Summarizing supported files with self-hosted LLMs and Ollama
• Reverse-proxying across multiple File Browser servers with Caddy
• Combining File Browser with syncthing for multi-site file availability

Schedule and availability

Target launch of the non-replayed 99%-complete version is by end of August on Leanpub.