Elixir File Browsing

Develop an API client in Elixir, and self-host file management for your Phoenix web app

Adding a file-management feature to your web app or API requires making a key decision about where you store these files and how you access them: locally, on an S3-compatible bucket, or in some other way? This book is about some other way that I’ve found serves me well without much complexity. Why though?

Storing files locally comes with a major downside: the files are likely stored within a container or on bare metal on some VPS or wherever you’ve deployed your Elixir app. This means that they are difficult to access. If the app is used by people other than yourself, you shouldn’t be accessing their files to begin with. But what if you have developed an app for your own use? Yes, you can access the files via SFTP (e.g., by using XFCE’s Thunar file manager), but it’s hardly the most convenient way to browse the files, besides through the web app.

The same downside applies to S3-compatible object storage. Managed services like DigitalOcean Spaces (as one example that I have used) do provide you with an online bucket browser UI that parses the “/” characters in the bucket object’s keys and presents the object as a directory tree. So, yes, you can browse the files–but you first have to log on to DigitalOcean, i.e. go through the whole 2-Factor Authentication song-and-dance. Again, this is inconvenient, and also assumes that you’re willing to be charged for some amount per month ($15 or so) for the convenience of storing even a tiny amount of files, or files that total way less than the alloted storage space for that monthly fee. It’s a frivolous option, especially if you take into account not only the monthly invoiced amount, but also that you have to deal with one more invoice for book-keeping, if it’s a business expense.

Things don’t get much better with self-hosted S3-compatible object storage. In fact, they get worse! Let’s take Minio as an example. Many moons ago Minio deprecated flat file storage, which means that files are not anymore accessible directly on the HDD or SSD where you’ve deployed it; instead, they are opaquely stored as hashed blobs and are accessible only through either the S3-compatible API that the Minio server offers, or with its mc command-line client (unfortunately named the same as one of my favorite programs, Midnight Commander) or with any other S3-compatible client, such as aws-cli or rclone. Even worse: in a recent update Minio disabled most of the features of its “web console” that used to also offer a file-browsing UX similar to that of DigitalOcean Spaces. You can deploy the last version of Minio with a fully-functional web console, and that’s what I’ve been doing–which means that I’m stuck on that version. As another example, let’s take a favorite of mine for self-hosted geographically-distributed S3-compatible object storage that I’ve been using in some cases: Garage. Garage is more lightweight than Minio, but offers absolutely no web UI to begin with, and the files are not accessible directly on the filesystem, either.

I’m currently developing 3 web apps and APIs that require a file-management feature to be fully usable by me and other users:

1. Pragmata , a distributor-led Asset Management SaaS for my industrial-equipment trading business, ISATEK . Every piece of equipment that my business delivers gets its own record on Pragmata, and the record is “handed over” to my client after the business transaction concludes. On this record I need to be able to attach documents such as the invoice, the delivery note, the technical specifications, the operations and maintenance manual, regulatory compliance certificates, materials tests, and anything else that will help my client’s maintenance engineers know what they received from ISATEK, how it is to be installed and maintained, which spare parts are available, etc. There is immense duplication of documents across records. For example, multiple units of dust filters with different technical specifications that ISATEK delivers share the same manual (a PDF file) that is specific not to each unit, but to the product family common across multiple units. There is also the need to share some of those documents before a record exists on Pragmata, for example during the quotation and offer phase of the commercial case. I need to be able to share a brochure of a product family or a technical document with my clients long before the offer generates a purchase order, an invoice and delivery note, and eventually a record on Pragmata.

2. Emporoman, a custom mini-ERP that reflects ISATEK’s business process starting with an RFQ (Request For Quotation) and (usually) ending with marking an invoice as settled and the commercial case / business transaction as completed. Here, I need the ability to manage a subset of the documents that Pragmata records include, as well as the ability to share a documents with the client at any phase of the commercial process without filling ISATEK’s internal (self-hosted, of course!) Courier-IMAP server with emails containing attachments that are already stored on ISATEK’s (self-hosted, of course!) Samba file server (and synchronized across devices with the glorious Syncthing).

3. Products, a Product Data Management API that provides the Single Source of Truth for all sales items of ISATEK, each codified with a unique 7-character code, and associated directly with a product family, and indirectly with a supplier’s brand, a product category, and a supplier. This piece of software is an Elixir/Phoenix application, the development of which is currently being documented in Phoenix Product Codex . It includes a content-management feature set that will eventually allow the association of product families with multilingual descriptions and other marketing content for the ISATEK website, as well as the association of product families and codified product variants (and bundles/kits) with documents such as technical manuals, CE certificates, etc. Again, those documents already exist elsewhere, and they must be made available for download on the website, which means that a mechanism for accessing those files is required, similar to S3’s “pre-signed URLs”.

It is a waste of storage space to have each of those applications manage its own subset of all those files, let alone do so locally. It is a waste of time to search for documents at different places, if needed. It is a waste of effort to develop more-or-less the same file-browsing UX/UI for each of those applications. And it’s a waste of time to work around the gradually-enshittified once-user-friendly UI of a paid SaaS for cloud storage, even if my storage space needs will probably never surpass the quota of that SaaS’ free tier.

A few years ago I stumbled upon a piece of open-source software that I’ve been self-hosting for 5 different ventures and going concerns ever since: the unimaginatively-named File Browser . I found File Browser while at the helm of ISATEK’s predecessor business and looking for a way to share files with clients without the aforementioned limitations of a service like Dropbox or attaching multi-megabyte PDFs to emails.

File Browser is truly fantastic. It is trivial to deploy in a non-Docker container, for example in a Debian or Alpine Linux container or VM on a Proxmox VE server. It is a single-binary Go application that offers multi-user authentication, a snappy web app with a simple and understandable UI for folder and file browsing, the ability to generate share links to files and folders an expiration time with preview and download hyperlinks (and, optionally, password-protected too), the ability to download an entire folder in different compressed formats and lastly, the ability to add a custom logo and color scheme for branding purposes. I can give the container or VM as much storage space as I need without paying overpriced tribute to a cloud overlord for a few gigabytes, let alone $15 per month plus the hassle of tracking monthly invoices! I can upload anything I want, from anywhere, and even have its flat-file storage directory synchronized across devices with Syncthing, so that I don’t even have to add folders and files manually through the clean and truly user-friendly web UI. I don’t have to deal with a web UI that’s been finessed by a Product-Led Growth “guru” in the interest of spurring the growth of a stagnating SaaS, with endless attempts to upsell me to features that I don’t need and that would lock me in to that SaaS, and I don’t need to run a proprietary, closed-source client to synchronize data between cloud storage and any PC I want (as long as that PC is either running Windows or macOS, or uses ext4 as its filesystem).

In short, File Browser offers everything my web apps need to get the job done. Most notably for this book, “everything” includes an undocumented HTTP API that is actually used by the filebrowser binary’s bundled Vue-based web UI to realize all features. This means that every single one of my File Browser deployments is a full-fledged solution for my needs, at zero marginal operating cost, and usable both by my clients accessing a document to which they have access during the early phases of my commercial process, as well as programmatically during their exploration of the ISATEK website or their eventual use of Pragmata.

Therefore, this book an exploration of that “other way” of adding file-management features to an Elixir application, i.e. without storing files locally or dealing with using S3-compatible object storage, either self-hosted or managed. There is no Elixir package for interfacing with a Filebrowser deployment’s REST API yet, so we’ll develop our own REST API client, implement file hashing so that we have something equivalent to S3’s “Etag”, see how we can make the package configurable for our Elixir application(s), and how to implement one or more Phoenix LiveView components that can take care of uploading, accessing or downloading files from our self-hosted Filebrowser instance with share URLs with an expiration date, and perhaps even more.

Developing something in Elixir in order to provide file-management functionality to a web app or API is however not the only objective of this book. The ulterior motive is tackling a project that is not exclusive to the File Browser API. This is also (you might say, primarily) a project-based book on using Elixir to develop a client for a REST API that involves authentication. Like my other books before this, this one too is not only about the goal we want to achieve, but also about how we go about achieving it. Even if you might not be interested in deploying File Browser for your own needs and in interacting with its undocumented API, the API provides a great excuse for learning various other Elixir software development and software architecture/engineering skills. The book also covers the reasoning behind the design options and choices made, how we iteratively develop better versions of the code, why and how to use Ecto even though we don’t manage a database, how to gradually refine our code, and much more.