~ From WampServer, to Vagrant, to QEMU

When I first dipped my toe into web development it was with PHP. Building quick and dirty inelegant websites, where the HTML and PHP would blur between one another. At the time, I did have a very firm grip on what HTTP was, where exactly PHP sat in the stack, other than it being in the backend, whatever that was, nor did I understand what a database was, other than a place to query data from. On top of this, I was writing this inelegant PHP code on Windows. If you've ever done PHP development on Windows, then you are most likely aware of WampServer. This is a solution stack for Windows that provides an Apache web server, OpenSSL, MySQL and PHP.

WampServer was exactly what I needed. Something that would allow me to create a proper web application with a web server, a database, and even OpenSSL, locally on my machine. It even provided a user interface to help with the management of this all, no need with mucking around with configuration files. As time went on my understanding of programming grew, as did my knowledge in the realm of web development. I eventually stumbled upon a new growing web framework that made developing in PHP even easier - Laravel.

Laravel wanted to make the entire PHP development process as seamless as possible. At the time, Laravel achieved this with Laravel Homestead, a pre-packaged Vagrant box. Everything you would need would be bundled into a virtual machine, nice and neat, away from your OS. And this was how I was introduced to Vagrant. A means of packaging virtual machines into a portable format, so you could easily create local development environments.

Vagrant worked by using a Vagrantfile to describe the virtual machine you would want to use, how it would be provisioned, the ports you would forward, and the filepaths you would want shared from host to guest. All of these tasks would be handled by the provider, the underlying virtual machine program itself, in my case, VirtualBox. Perhaps the one thing I liked the most about Vagrant, was the ability to provision my machine, clear down if I wanted to and have it back in a clean state for development.

This was perhaps my earliest introduction to Linux in the context of hosting server software, or rather, hosting a local development environment. I was finally getting a glimpse at how all the pieces of my web development stack pieced together, though there was still some mystery behind it all.

It was around the time of Windows 10 being forced onto everyone that I made the switch to Linux. Windows 8 was something I had tolerated, though I wasn't really a fan of the whole pseudo-tablet interface it offered. I had wanted to try out Linux anyway as my daily driver, so I saw this as the perfect excuse. I installed Arch on my desktop, went through the typical cycle of trying to glue together one of those fancy desktop environments with shell scripts, then settled on GNOME, as it just works and has what I consider some sensible defaults.

I acclimated to my new environment, and figured out how to get Vagrant and VirtualBox installed so I could continue my hobbyist hacking as I had done before. I wanted to continue to keep my development environments separate from one another, and from the host OS itself. However, I started to have issues with VirtualBox, nothing major, just paper cuts here and there. But, over time these cuts added up. Furthermore, over the years I had noticed Vagrant becoming slower, or perhaps it was me realising how slow Vagrant had always been.

As someone who enjoys playing video games, and a recent convert to Linux, I was well aware of the derth of support for games. I was also aware of some of the solutions, one of those being GPU passthrough to this thing called QEMU. QEMU is a fast and lightweight machine emulator and virtualizer. This was of course something that interested me, so I went about exploring QEMU and playing with it. When I first started using it, I was offput by it. No slick UI to use, a myriad of flags to pass it, some flags which even took more options. But when I figured out the right incantation of flags to pass it, it worked, and I noticed its speed in comparison to VirtualBox.

At this point when it came to my hobbyist development, I had moved past PHP and started learning Go, and was looking to do some serious development with this for a CI platform I had an idea for. By now, I had a firmer grasp of the software stack I wanted to work with, a better understanding of how everything pieced together. And so I went about developing that CI platform, that would later become Djinn CI. I uninstalled VirtualBox and Vagrant and fully committed to using QEMU, booting up the local machine was as simple as hitting CTRL + R in my terminal, searching for qemu and hitting enter, an elegant solution I know.

QEMU worked fine for what I needed. However there were a few things I missed from Vagrant:

• The ability to spin up a virtual machine based on a box
• The ability to provision machines
• Easy portforwarding (at least something easier than a hostfwd=tcp:127.0.0.1:<src>-:<dst>)

I was tempted to build a full fledged tool to solve these issues. Instead, I found a better solution, a simpler one in my opinion. The solution I found was to implement a simple shell script, called qemu. It looks something like this,

$ qemu debian/stable

if given one argument, then that single argument is treated as the name of the QCOW2 image to use. The image would be relative to the directory specified via the QCOW2PATH environment variable. Here's what mine looks like,

$ ls $QCOW2PATH
alpine  arch  debian  djinn-dev  freebsd  ubuntu

if any leading arguments are given before the image name, then these are passed to the underlying QEMU program as flags,

$ qemu -snapshot debian/stable

if a provision.sh script is detected in the current directory from where qemu is executed then this is checked for the number of CPUs to use and ports to forward, for example,

$ head provision.sh
#!/bin/sh
# cpus: 2
# portfwd: 5432

the amount of memory given to the virtual machine will be half of what memory is available via MemoryAvailable from /proc/meminfo. The provision.sh script will be copied to the machine and executed once booted.

This script will automatically forward port 2222 to 22, and assumes that passwordless root is configured on the virtual machine for SSH access.

I found this script to be a simple rather nice solution to the things that I missed from Vagrant. The QCOW2 image files are simply files kept in a single location, I can use the filesystem to manage these, and refer to them via the aforementioned environment variable. The ability to provision machines is done simply by copying the provision script and executing it to the machine once booted, and portforwarding is hacked ontop of that too. You can find this script here if you're interested in making use of it yourself.

This is a pretty long post, and seems tangential, but I wanted to look back at the different tooling I've used when it comes to programming. I've found that some tools in development helped with my understanding of programming, whilst others, perhaps hindered it. When I first started out I didn't really understand what was happening, I was simply looking for an end result, and would do anything to get there, I'm still guilty of this to a certain degree to this day.