Andrew Venglar

Speeding Up Docker: re-thinking data in 2018

Speeding Up Docker: re-thinking data in 2018

Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.
-- Andrew Tanenbaum

Web development is reaching a turning point -- developers are beginning to see bandwidth expanding to the degree that the amount of available bandwidth exceeds the ability for some tools to utilize it all -- meaning you finally should underestimate the station wagon: its relevance is waning. Because of those tools, one of the nastier challenges we ran into in the early days of Haystack was making file transfers fast.

We explored three approaches:

  1. Let's use Vagrant! The cool kids are doing it!
  • Slow--terribly so. Vagrant is fine for small projects, but the second you throw a large framework at it that generates loads of caches and factories at runtime during development (in our case, Magento 2), you find yourself waiting literally minutes for single page reloads.
  1. Creating a local Docker mount with NFS / osxfs! The cooler kids are doing it!
  • See above. osxfs is certainly faster than NFS, but we're still watching a tortoise outpace a sloth.
  1. Using a remote IDE hosted within Docker
  • Fast, but limiting. Like many developers, my efficiency is dependent on how well I know my IDE. Being shoe-horned into a browser-based IDE is like changing a tire with a wrench. Doable? I guess--but you'll spend the entire time cursing your lack of a tire iron under your breath.

Some of this may sound pedantic, but try waiting 2 minutes every time you want to adjust a font size. It gets old. Quickly. After about two weeks of dealing with this, we asked ourselves a a serious question: Should we even continue with traditional mounting?

But what about all that bandwidth laying around you were on about earlier?

This is where things come full circle: enter Haystack Mounts.

Haystack doesn't create a traditional mount on your system--rather, it uses the power of fswatch to monitor change events to a folder, and syncs them with a specialized Docker container. This container's volume is mounted to your web container.

Great, but I'm a developer, not a system administrator. Why should I care?

Haystack doesn’t create mounts on the remote filesystem, meaning virtually no latency when reading and writing thousands of files at once--maximizing your bandwidth to sync as many files simultaneously as possible. The result is native page load speeds, and the freedom to use your own tools--no matter how massive your framework.

So the best of both worlds?


At Haystack, we believe a happy developer is a one who uses their own tools to do what they love--without feeling hindered by that choice. Leveraging new technologies is paramount to this goal in an industry that moves as quickly as ours.

I like to write code, not wait for it. What about you?