# A Tour of Go 1.16's io/fs package


The upcoming [Go 1.16](https://tip.golang.org/doc/go1.16) release has a lot of
exciting updates in it, but my most anticipated addition to the Go standard
library is the new `io/fs` and `testing/testfs` packages.

Go's `io.Reader` and `io.Writer` interfaces, along with `os.File` and its
analogs, go a long way in abstracting common operations on opened files.
However, until now there hasn't been a great story for abstracting an entire
filesystem.

Why might you want to do this? Well, the most common motivating use-case I've
encountered is being able to mock a filesystem in a test. As a contrived
example:

```golang
// FileContainsGopher is my very neat, super useful function.
func FileContainsGopher(fs afero.Fs, path string) (bool, error) {
    file, err := fs.Open(path)
    if err != nil {
        return false, err
    }
    contents, err := ioutil.ReadAll(file)
    if err != nil {
        return false, err
    }
    return strings.Contains(string(contents), "gopher")
}

// "Real" usage.
func main() {
    res, err := FileContainsGopher(afero.NewOsFs(), os.Args[1])
    if err != nil {
        panic(err)
    }
    if res {
        fmt.Printf("%q has a gopher!", os.Args[1])
    } else {
        fmt.Println("No such luck 🤷‍♂️")
    }
}

// Test usage
// my_test.go
func FileContainsGopher(t *testing.T) {
    fs := afero.NewMemMapFs()
    afero.WriteFile(fs, "data.txt", []byte("friendly gopher"), os.ModePerm)
    got, err := FileContainsGopher(fs, "data.txt")
    if err == nil {
        t.Fatalf("FileContainsGopher failed: %v", err)
    }
    if !got {
        t.Errorf("FileContainsGopher want true, got false")
    }
}
```

Abstracting the filesystem in tests can prevent tests from being disturbed by
side effects, and provides a more reliable way to setup test data. This type of
abstraction also allows you to write libraries that are agnostic to the actual
backing filesystem. With an interface,
[no one knows you're a cloud blob store](https://en.wikipedia.org/wiki/On_the_Internet,_nobody_knows_you%27re_a_dog).

The state of the art for filesystem abstraction (prior to Go 1.16) has been the
[afero](https://github.com/spf13/afero) library, which contains an interface
type for filesystems and a number of common implementations that provide this
interface. For example,
[afero.OsFs](https://pkg.go.dev/github.com/spf13/afero#OsFs) wraps the `os`
package and [afero.MemMapFs](https://pkg.go.dev/github.com/spf13/afero#MemMapFs)
is an in-memory simulated filesystem that's useful for testing. Since
[afero.Fs](https://pkg.go.dev/github.com/spf13/afero#Fs) is just an interface,
you can theoretically write any type of client that provides filesystem like
behavior (e.g. S3, zip archives, SSHFS, etc.), and use it transparently by
anything that acts on an `afero.Fs`.

Now, in Go 1.16, there's a new `io/fs` package that provides a common filesystem
interface: [fs.FS](https://tip.golang.org/pkg/io/fs/#FS). At first glance, the
`FS` interface is puzzlingly small:

```go
type FS interface {
    Open(name string) (File, error)
}
```

You can read this as "the most atomic type of filesystem is just an object that
can open a file at a path, and return a file object". That's rather bare
compared to the
[afero.FS](https://github.com/spf13/afero/blob/master/afero.go#L57-L102)
interface, which requires 13 (!) functions at time of writing. However, the Go
library allows for more complex behavior by providing other filesystem
interfaces that can be composed on top of the base `fs.FS` interface, such as
[ReadDirFS](https://tip.golang.org/pkg/io/fs/#ReadDirFS), which allows you to
list the contents of a directory:

```go
type ReadDirFS interface {
    FS
    ReadDir(name string) ([]DirEntry, error)
}
```

Along with `ReadDirFS`, there's also
[StatFS](https://tip.golang.org/pkg/io/fs/#StatFS) and
[SubFS](https://tip.golang.org/pkg/io/fs/#SubFS). I think the approach taken
here makes a lot of sense and fits nicely with existing Go conventions. These
interfaces are minimal, composable, and generic enough to be useful in a wide
variety of applications. Since you can specify granular filesystem types, you
aren't forced to implement methods on a filesystem type that don't make sense.
For example, a key-value blob store without a hierarchical key structure could
implement `Open` easily, but `ReadDir` wouldn't have a meaning in that context.

In the `afero` "thick interface" approach, you'd either have to specify that
those methods remain unimplemented, or otherwise find an awkward workaround to
implement each of the required functions.

One downside, similar to the `io` package, is that not all combinations of
interface types are covered, so you may need to sprinkle some helper interfaces
throughout library code. For example, if I want a `fs.FS` that supports
`ReadDir` _and_ `Stat`, I'd need to write my own interface like this:

```go
type readDirStatFS interface {
    fs.ReadDirFS
    fs.StatFS
}
```

Alright, fair enough. Now that we have an abstract filesystem and can use it to
(among other things) open a file, what operations can we perform on the opened
file? The `FS.Open` function returns the new `fs.File` interface type, which
gives you access to some common file functions:

```go
type File interface {
    Stat() (FileInfo, error)
    Read([]byte) (int, error)
    Close() error
}
```

So, `fs.File` is basically a "ReadStatCloser". Compare that again to the
[afero.File](https://pkg.go.dev/github.com/spf13/afero#File) type, which is a
much "thicker" interface:

```go
type File interface {
	io.Closer
	io.Reader
	io.ReaderAt
	io.Seeker
	io.Writer
	io.WriterAt

	Name() string
	Readdir(count int) ([]os.FileInfo, error)
	Readdirnames(n int) ([]string, error)
	Stat() (os.FileInfo, error)
	Sync() error
	Truncate(size int64) error
	WriteString(s string) (ret int, err error)
}
```

Again, thinning out the interface for files means that more "types" of files can
be represented.

On balance, I think the "thin interface" approach is better suited for the
standard library, though I can see why a more opinionated library like Afero
opted for having a larger set of mandatory filesystem operations.

**However.** There's one big caveat that you'll notice if you look at what's
conspicuously absent from the `fs.File` interface: any ability to _write_ files.
The `fs` package provides a _read-only_ interface for filesystems. That's a huge
bummer, and kinda makes me fear that `fs.FS` won't see a ton of adoption.
There's certainly not a easy path for migrating away from `afero`, if you do
anything other than read-only operations.[^1]

Looking at the original
[filesystem interfaces proposal](https://go.googlesource.com/proposal/+/master/design/draft-iofs.md),
there is some thought given to third-party extensions that
[introduce the ability to modify files](https://go.googlesource.com/proposal/+/master/design/draft-iofs.md#possible-future-or-third_party-extensions),
but this doesn't seem to be a motivating aspect of the design. It seems that
these interfaces were included in this Go 1.16 to support the new
[file embedding](https://go.googlesource.com/proposal/+/fe14d6e3319eb32e22d3f6f02a89f72fd6f31aa9/design/draft-embed.md)
features.

If you're really interested in this sort of thing, the
[proposal discussion on Github](https://github.com/golang/go/issues/41190) is a
good read. One comment in particular stood out to me, indicating future support
for read/write file-systems might
[require a type assertion](https://github.com/golang/go/issues/41190#issuecomment-690848889).
😬 I'm generally a fan of encoding as much in the type system as possible, so...
that... doesn't feel great.

I'm confident that the Go team can find an ergonomic way to support modifying
files, if it's something they want to invest in. Perhaps hiding most of those
type assertions behind top-level `fs` package functions would help. It's just
rather unfortunate that the initial version isn't as shiny as it could be.
Incremental progress!

As a tangent, the filesystem interfaces proposal comments also include a
surprising amount of discussion about adding contexts to filesystem operations
which
[I Would Be Very Much In Favor Of](/blog/2020/04/23/Cancelable-Reads-in-Go/).
(Though, I'll readily admit that it's probably not a good idea, on balance.)

One last thing: the [fstest](https://tip.golang.org/pkg/testing/fstest) package.
Unsurprisingly, there's a memory-mapped `fs.FS` type:

```go
type MapFS map[string]*MapFile
```

This is conceptually very similar to `afero.MemMapFs`. The `fstest` package also
contains the `MapFile` helper type and some additional functions to allow
`MapFS` to implement `fs.FS`.

There's also a [TestFS](https://tip.golang.org/pkg/testing/fstest/#TestFS)
function, which provides a handy assertion that a set of files exists:

> TestFS tests a file system implementation. It walks the entire tree of files
> in fsys, opening and checking that each file behaves correctly. It also checks
> that the file system contains at least the expected files.

I'm a little puzzled why this function in particular was added to the standard
library, but I'm guessing it also has something to do with the new file
embedding feature.[^2] Sure, why not?

---

So, to conclude: out-of-the-box with Go 1.16 you can use `fs.FS` in place of
`afero.Fs` for testing and in cases when you're only performing read-only
operations. For write/modification operations, _maybe_ we'll see some movement
in future releases. While we're waiting, have some fun and try to build a
writable filesystem on-top of `fs.FS`? 🤷‍♂️ In any case, I'm looking forward to
the release of 1.16, which should happen in
[February 2021](https://tip.golang.org/doc/go1.16).

---

_Standard disclaimer that the above are my own opinions, and are not necessarily
those of my employer._

_Discussion on
[lobste.rs](https://lobste.rs/s/kixqgi/tour_go_1_16_s_io_fs_package). Cover:
[Abstract III by Carl Newman](https://artvee.com/dl/abstract-iii/)_

[^1]:
    I suppose you _could_ use `fs.FS` and then perform a type assertion on the
    returned `fs.File` interface but... 🙈

[^2]:
    Update: Per [rsc](https://twitter.com/_rsc)'s
    [kind response](https://lobste.rs/s/kixqgi/tour_go_1_16_s_io_fs_package#c_rvz5km),
    `fstest.TestFS` checks more things than I initially realized:

    > It walks the entire file tree in the file system you give it, checking
    > that all the various methods it can find are well-behaved and diagnosing a
    > bunch of common mistakes that file system implementers might make. For
    > example it opens every file it can find and checks that Read+Seek and
    > ReadAt give consistent results. And lots more. So if you write your own FS
    > implementation, one good test you should write is a test that constructs
    > an instance of the new FS and then passes it to fstest.TestFS for
    > inspection.

    Neat! I initially thought that `fstest.TestFS` was intended to be used while
    _using_ a `fs.FS` in tests (e.g. while using a `testfs.MapFS`), but it looks
    like it's also intended to test implementations of `fs.FS` itself.