In the last post, I talked about how I setup my Nomad cluster running on bare metal using Void Linux and Ansible. I talked a lot about the basic design, but not how I actually executed on that design. That’s what we’ll get into in this post.

There are 3 main elements to configure in the cluster. First, the network, routing, and data-plane needs to be configured as a foundation that can be used to PXE boot the rest of the fleet, and then managed on an ongoing basis. Second, the Operating System (OS) on the servers needs to be configured to run the services that will make this selection of machines into a Nomad cluster. Finally, Nomad itself needs to be configured with security policies and other runtime tweaks to suit the applications that I want to install into the cluster.

I recently decided to rebuild my development cluster so that I can finally nail down a few services that I’ve wanted to understand better for some years now. To get to the point that I have a working cluster meant setting out some requirements, rebuilding the entire software stack, and then finally making physical changes to the cluster itself. This will be a series of posts detailing the adventure to build a private cloud with actual cloud-like functionality, namely multi-chassis scale-out and the ability to treat arbitrarily many computers as one large resource pool.

Its late at night, you’re on a walk, and you see something wrong. What do you do, who do you call? I walk several miles daily to try to keep my health up given that my career has me sitting at a desk all day. As a result of this, I somewhat regularly encounter situations that need to get called in, and over the years, I’ve become increasingly weary to try. Not because I don’t think its important to do so, but because of how universally bad the response is from the supposedly highly trained incident response groups is.

People who’ve known me for a while know that I’m an alumni of a program called BEST Robotics, and that I still actively volunteer with the program. This is an experiential learning program with a strong focus on industry exposure. Students compete in many different aspects in Spring and Fall, but I am most heavily involved with the flagship BEST Robotics Competition in the fall. This 8 week program sees students presented with a challenge and turned loose for 8 weeks to come up with an answer.

One of the topics of discussion in my circle of friends we keep circling back to is security of personal, work, corporate, and production systems. Its no surprise since many of my circle have backgrounds in highly academic fields and have between us bordering on a century of experience in operating systems at all points on the scale spectrum. From active red-team work to defense of mission critical systems, our discussions tend to span it all, and usually with a fairly blunt and honest look at reality, something I don’t often see in the industry.