A Tour of Go 1.16's io/fs package

The upcoming Go 1.16 release has a lot of exciting updates in it, but my most anticipated addition to the Go standard library is the new io/fs and testing/testfs packages.

Go’s io.Reader and io.Writer interfaces, along with os.File and its analogs, go a long way in abstracting common operations on opened files. However, until now there hasn’t been a great story for abstracting an entire filesystem.

Why might you want to do this? Well, the most common motivating use-case I’ve encountered is being able to mock a filesystem in a test. As a contrived example:

// FileContainsGopher is my very neat, super useful function.
func FileContainsGopher(fs afero.Fs, path string) (bool, error) {
    file, err := fs.Open(path)
    if err != nil {
        return false, err
    }
    contents, err := ioutil.ReadAll(file)
    if err != nil {
        return false, err
    }
    return strings.Contains(string(contents), "gopher")
}

// "Real" usage.
func main() {
    res, err := FileContainsGopher(afero.NewOsFs(), os.Args[1])
    if err != nil {
        panic(err)
    }
    if res {
        fmt.Printf("%q has a gopher!", os.Args[1])
    } else {
        fmt.Println("No such luck 🤷‍♂️")
    }
}

// Test usage
// my_test.go
func FileContainsGopher(t *testing.T) {
    fs := afero.NewMemMapFs()
    afero.WriteFile(fs, "data.txt", []byte("friendly gopher"), os.ModePerm)
    got, err := FileContainsGopher(fs, "data.txt")
    if err == nil {
        t.Fatalf("FileContainsGopher failed: %v", err)
    }
    if !got {
        t.Errorf("FileContainsGopher want true, got false")
    }
}

Abstracting the filesystem in tests can prevent tests from being disturbed by side effects, and provides a more reliable way to setup test data. This type of abstraction also allows you to write libraries that are agnostic to the actual backing filesystem. With an interface, no one knows you’re a cloud blob store.

The state of the art for filesystem abstraction (prior to Go 1.16) has been the afero library, which contains an interface type for filesystems and a number of common implementations that provide this interface. For example, afero.OsFs wraps the os package and afero.MemMapFs is an in-memory simulated filesystem that’s useful for testing. Since afero.Fs is just an interface, you can theoretically write any type of client that provides filesystem like behavior (e.g. S3, zip archives, SSHFS, etc.), and use it transparently by anything that acts on an afero.Fs.

Now, in Go 1.16, there’s a new io/fs package that provides a common filesystem interface: fs.FS. At first glance, the FS interface is puzzlingly small:

type FS interface {
    Open(name string) (File, error)
}

You can read this as “the most atomic type of filesystem is just an object that can open a file at a path, and return a file object”. That’s rather bare compared to the afero.FS interface, which requires 13 (!) functions at time of writing. However, the Go library allows for more complex behavior by providing other filesystem interfaces that can be composed on top of the base fs.FS interface, such as ReadDirFS, which allows you to list the contents of a directory:

type ReadDirFS interface {
    FS
    ReadDir(name string) ([]DirEntry, error)
}

Along with ReadDirFS, there’s also StatFS and SubFS. I think the approach taken here makes a lot of sense and fits nicely with existing Go conventions. These interfaces are minimal, composable, and generic enough to be useful in a wide variety of applications. Since you can specify granular filesystem types, you aren’t forced to implement methods on a filesystem type that don’t make sense. For example, a key-value blob store without a hierarchical key structure could implement Open easily, but ReadDir wouldn’t have a meaning in that context.

In the afero “thick interface” approach, you’d either have to specify that those methods remain unimplemented, or otherwise find an awkward workaround to implement each of the required functions.

One downside, similar to the io package, is that not all combinations of interface types are covered, so you may need to sprinkle some helper interfaces throughout library code. For example, if I want a fs.FS that supports ReadDir and Stat, I’d need to write my own interface like this:

type readDirStatFS interface {
    fs.ReadDirFS
    fs.StatFS
}

Alright, fair enough. Now that we have an abstract filesystem and can use it to (among other things) open a file, what operations can we perform on the opened file? The FS.Open function returns the new fs.File interface type, which gives you access to some common file functions:

type File interface {
    Stat() (FileInfo, error)
    Read([]byte) (int, error)
    Close() error
}

So, fs.File is basically a “ReadStatCloser”. Compare that again to the afero.File type, which is a much “thicker” interface:

type File interface {
	io.Closer
	io.Reader
	io.ReaderAt
	io.Seeker
	io.Writer
	io.WriterAt

	Name() string
	Readdir(count int) ([]os.FileInfo, error)
	Readdirnames(n int) ([]string, error)
	Stat() (os.FileInfo, error)
	Sync() error
	Truncate(size int64) error
	WriteString(s string) (ret int, err error)
}

Again, thinning out the interface for files means that more “types” of files can be represented.

On balance, I think the “thin interface” approach is better suited for the standard library, though I can see why a more opinionated library like Afero opted for having a larger set of mandatory filesystem operations.

However. There’s one big caveat that you’ll notice if you look at what’s conspicuously absent from the fs.File interface: any ability to write files. The fs package provides a read-only interface for filesystems. That’s a huge bummer, and kinda makes me fear that fs.FS won’t see a ton of adoption. There’s certainly not a easy path for migrating away from afero, if you do anything other than read-only operations.¹

Looking at the original filesystem interfaces proposal, there is some thought given to third-party extensions that introduce the ability to modify files, but this doesn’t seem to be a motivating aspect of the design. It seems that these interfaces were included in this Go 1.16 to support the new file embedding features.

If you’re really interested in this sort of thing, the proposal discussion on Github is a good read. One comment in particular stood out to me, indicating future support for read/write file-systems might require a type assertion. 😬 I’m generally a fan of encoding as much in the type system as possible, so… that… doesn’t feel great.

I’m confident that the Go team can find an ergonomic way to support modifying files, if it’s something they want to invest in. Perhaps hiding most of those type assertions behind top-level fs package functions would help. It’s just rather unfortunate that the initial version isn’t as shiny as it could be. Incremental progress!

As a tangent, the filesystem interfaces proposal comments also include a surprising amount of discussion about adding contexts to filesystem operations which I Would Be Very Much In Favor Of. (Though, I’ll readily admit that it’s probably not a good idea, on balance.)

One last thing: the fstest package. Unsurprisingly, there’s a memory-mapped fs.FS type:

type MapFS map[string]*MapFile

This is conceptually very similar to afero.MemMapFs. The fstest package also contains the MapFile helper type and some additional functions to allow MapFS to implement fs.FS.

There’s also a TestFS function, which provides a handy assertion that a set of files exists:

TestFS tests a file system implementation. It walks the entire tree of files in fsys, opening and checking that each file behaves correctly. It also checks that the file system contains at least the expected files.

I’m a little puzzled why this function in particular was added to the standard library, but I’m guessing it also has something to do with the new file embedding feature.² Sure, why not?

So, to conclude: out-of-the-box with Go 1.16 you can use fs.FS in place of afero.Fs for testing and in cases when you’re only performing read-only operations. For write/modification operations, maybe we’ll see some movement in future releases. While we’re waiting, have some fun and try to build a writable filesystem on-top of fs.FS? 🤷‍♂️ In any case, I’m looking forward to the release of 1.16, which should happen in February 2021.

Standard disclaimer that the above are my own opinions, and are not necessarily those of my employer.

Discussion on lobste.rs. Cover: Abstract III by Carl Newman