Elixir File Browsing: my new book on developing a REST API client for self-hosted file management

Announcing my new side-project and fourth non-fiction, technical book: Elixir File Browsing .

Following the same engineering ethos and exploratory learning approach that has proven so popular with readers of Northwind Elixir Traders and Phoenix Product Codex , in Elixir File Browsing we deploy File Browser and then iteratively develop ExFileBrowser, a full-featured client for File Browser’s undocumented REST API that we figure out by looking at File Browser’s source code .

Why is this book being written?

1. Because I urgently need simple file-management functionality for my Phoenix LiveView apps with a user-friendly web UI.
2. Because I don’t enjoy the hassle of deploying and babying Minio or Garage to have files stored opaquely.
3. Because I don’t want to have to pay a monthly fee for S3-compatible object storage for my apps.
4. Because writing a fully-featured REST API client in Elixir is a great exercise that hones your Elixir skills.
5. Because the Elixir ecosystem deserves more hands-on guides on end-to-end projects with an exploratory learning approach.
6. Beause Elixir, its ecosystem and community are awesome.

State of development

The book is currently around 55% complete, but I have not yet released it on Leanpub or elsewhere.

Completed content

• Using podman to self-host a File Browser server
• Investigating the undocumented File Browser REST API by looking at its Go codebase
• Jobs To Be Done by the users of our API client
• First draft of the Client module of ExFileBrowser
• Implementing authentication
• Seamless sessions with JWT token renewals
• Cleaning up the Client module with a function pipeline
• Using Ecto schemaless changesets to improve the Client module
• A layered architecture for ExFileBrowser
• Rewriting the Client module for good
• Fleshing out the Request HTTP utility module
• Cleaning up authentication-related functions
• Preparing an executing requests with Req
• Dealing with any public API route
• Further improvements to authentication
• Dispatching a request
• Automatic (re-)authentication and renewal
• Restructuring our context modules
• The Operations module for expert use
• Authentication as an operation
• Uploading a file
• Streaming file contents
• Making sure we don’t corrupt data on upload
• POSTing a local file
• GETting the metadata of a resource
• POSTing a remote file from a URL
• Streaming in and streaming out
• Elixir streams, and Stream vs. Enum
• Transforming the async request’s response body into a stream
• Lots and lots of periodic refactoring
• Downloading files from File Browser
• Enumerables and Collectables
• Separating Req options from other options
• Getting metadata after uploading a file
• Getting the storage space usage from File Browser
• Preventing or allowing uploads and downloads based on File Browser server or local disk space

Draft content

The following sections are in early-draft state:

• Managing File Browser users
• Managing File Browser settings

Planned content

The following sections are planned and will certainly be written:

• Data integrity verification on upload and download
• Downloading entire folders
• Downloading entire folders in archive format
• Uploading entire directories
• Making ExFileBrowser configurable for an application

“Maybe” content

The following sections are speculative and any of those might be dropped entirely.

• Implementing a Phoenix LiveView component to pick resources from the File Browser server
• Proxying requests to the File Browser server with Phoenix
• Storing file metadata on an SQLite database
• Implementing a file taxonomy (tags and categories/subcategories)
• Summarizing supported files with self-hosted LLMs and Ollama
• Reverse-proxying across multiple File Browser servers with Caddy
• Combining File Browser with syncthing for multi-site file availability

Schedule and availability

Target launch of the non-replayed 99%-complete version is by end of August on Leanpub .