Simply, buffered I/O is a way to optimise read and write operations. A buffer is held in memory, and will only actually perform the read or write operation to the underlying resource when necessary. Here's a quick proof of concept in Go. Below, we create a new type of ExampleWriter and define a Write([]byte) (int, error) func on it, so that it satisfies the io.Writer interface. We then use this new type as an argument to the NewWriterSize func, and pass in a buffer size of 8.

package main

type ExampleWriter int

func (ExampleWriter) Write(p []byte) (int, error) {
    log.Println("I am being written to!")
    log.Println("The contents is:", p)
    return len(p), nil
}

func main() {
    buf := bufio.NewWriterSize(new(ExampleWriter), 8)
    buf.Write([]byte("abcd"))
    buf.Write([]byte("efgh"))
}

The outcome of this program? Nothing. You'd maybe expect it to log I am being written to!twice: once for each write operation. However, the buffer doesn't actually write to the underlying resource. The buffer just holds it in memory and only actually performs the write operation when the contents of the buffer exceeds the maximum size, or the Flush func is called on it. You can view this on Go playground #1.

package main

import (
	"bufio"
	"log"
)

type ExampleWriter int

func (ExampleWriter) Write(p []byte) (int, error) {
	log.Println("I am being written to!")
	log.Println("The contents is:", p)
	return len(p), nil
}

func main() {
	buf := bufio.NewWriterSize(new(ExampleWriter), 8)
	buf.Write([]byte("abcd"))
	buf.Write([]byte("efgh"))
	buf.Write([]byte("ijkl"))
	buf.Flush()
}

// I am being written to!
// The contents is: abcdefgh
// I am being written to!
// The contents is: ijkl

As you can see, the third write operation causes the buffer to write to the underlying Writer, and the Flush func causes any unwritten data in the buffer to be sent to the underlying Writer. You can view this code on Go playground #2.

In Practise

In reality, imagine we were getting some real-time data drip-fed to our application. You wouldn't want every new piece of information to be written straight to a file — that would take far too many write operations. Instead, we'd create a buffer to hold information and only write to the file periodically. Let's give that a try:

func main() {
    // Create a waitgroup so that both goroutines complete.
    var wg sync.WaitGroup
    wg.Add(2)

    // Run a goroutine to write to a file 1,000,000 times.
    go func() {
        t := time.Now()
        f, _ := os.Create("unbuf.txt")
        defer f.Close()
        for i := 1; i <= 1_000_000; i++ {
            content := strconv.Itoa(i) + "\n"
            f.Write([]byte(content))
        }
        log.Println("Unbuffered writes exec took:", time.Since(t).Milliseconds())
        wg.Done()
    }()

    // Run a goroutine to write to a buffer 1,000,000 times.
    go func() {
        t := time.Now()
        f, _ := os.Create("buf.txt")
        buf := bufio.NewWriterSize(f, 1024)
        defer f.Close()
        for i := 1; i <= 1_000_000; i++ {
            content := strconv.Itoa(i) + "\n"
            buf.Write([]byte(content))
        }

        // Don't forget to flush anything that is still in the buffer!
        buf.Flush()

        log.Println("Buffered writes exec took:", time.Since(t).Milliseconds())
        wg.Done()
    }()

    // Wait for both goroutines to finish.
    wg.Wait()
}

In the example above, we iterate through numbers 1 to 1,000,000 and write the number to a file. In the first goroutine, we do each write individually. In the second, we write to a buffer and use the buffer to, well, buffer the amount of times we write to the file. We've set the buffer size to 1024, meaning that it'll store 1024 bytes in memory before persisting them to the file.

This means that the first example will write to the file 1,000,000 times! The buffered writer will only write to the file ~1,000 times.

You don't need me to tell you the buffered goroutine will be quicker. But how much quicker?

Buffered writes exec took: 141
Unbuffered writes exec took: 4554

So there we go. Granted, it is quite a low-level language feature, but understanding how buffers work is really useful.