Mac McClain

Why We Built Haystack

Why We Built Haystack

I own a software consulting company, GoSolid. We’re a little different than the average software consulting agency--we have no sales people, we don’t sell adwords, and we don’t build stuff for the sake of billing our customers. I'm a software developer at my core, and there have been many times that I have chosen quality of code for my customers and my team over financial gain for myself.

I'm telling you this because about 6 months ago, we realized that our development and stage environments were getting more complicated, and that parity between our dev, test and production was getting out of hand. For example, a simple change to a timeout setting on the dev environment would break if it was forgotten in production. We were using tools like Vagrant and Packer, but we knew this new shiny thing called Docker was getting more popular. Okay, getting back to why I'm telling you this: we could have said “screw it, who cares,” and just billed our customers--or we could have done the right thing and build real infrastructure and real parity between environments.

After all, dev, test, stage and production parity is the goal, right? Dev on your local machine, push to test, push to stage, and then finally push to production with no errors!

Well, turns out that it isn’t that easy. Our first attempts at Docker and docker-compose led to extremely slow page loads on our development environments. In addition, we had to write a bunch of bash scripts, modify our etc/hosts file, etc.

This wasn’t going to work--we have 6 to 10 active projects at any time. And honestly, no one reads the setup docs anyway, so we burned up valuable dev time dealing with our dev environments. Then, when we were ready to make a staging site on AWS, we had to find some other way to build it out (Bitnami, bless its heart, is not the solution.)

I decided again to choose quality over financial gain, and solve this problem. My goals were simple.

  • A complete development stack for the project with one command.
  • Same single command to create an identical stack on AWS for test or prod.
  • Everything in version control.
  • No bash scripts or setup instructions.
  • Super reliable.

I assembled my team, and we spent a solid two weeks creating a prototype. By solid, I mean 15 hour days with almost no sleep. You know, a typical dev project!

Within two weeks, we had created a Haystack file that could be written in YAML or JSON. It is basically a blueprint for your stack--for example, if you have a LAMP stack then it would include everything needed to build the PHP, NGINX server, and a MySQL DB.

The beautiful part is that it all lives in one file and it tracks in Git, along with the rest of the project. It met all of my goals.

We have been using it for nearly 6 months and I love it. My whole team loves it.

When we work on a project, we all get the exact same stack, down to the PHP and MySQL settings. It’s easy to maintain, and it just works. Our development environments are fast because we bypassed all of Docker’s file system problems, and it takes care of all the URLs, hosts files, memory settings, etc.

I’ve created a lot of tools over the years, but this one is different. I thought about Haystack day and night, and I had to bring Haystack to the world. Everyone should have simple development stacks and easy deployment to test and production. I mean, it is 2018, isn’t it?

We're on a mission to bring Haystack to every developer who cares about their stack. We're rebuilding from the ground up to port over all the things we love, and rework the parts that could use some enhancing. We want to provide some prebuilt stacks for popular software applications like WordPress, Magento, as well as popular stacks like Django, Laravel, Flask, and others. You can also build your own stack pretty easily.

We're bootstrapping this project, meaning we will focus on paying our bills with client-paying work, and work on Haystack at nights, weekends, and early mornings before our kids wake up.

I would love to get your feedback as we go along. Please join us on this journey by signing up for early access to Haystack, and following our story on our blog.

Thank you for reading.