Engineering at Domio: My First Server

Hello! I’m Noah, a software engineer here at Domio. I work primarily on our backend systems, but I also get to participate in all aspects of our engineering team, from infrastructure to hiring. I’ve wanted to write down this story for a long time, and this is my chance to do so while also showing what engineering at Domio is all about.

As a startup, our engineering team is small but razor sharp. We have aggressive goals, and more often than not achieving these goals requires clearing unexpected hurdles that you’ve never encountered before. Sometimes none of your coworkers have the answer either — you’re in uncharted waters. Though we can and do pair up with our colleagues on difficult problems, the ability to break through a problem on your own is still crucial. The skills you develop by working solo on your own projects are the same skills you’ll use when it’s your turn to help your coworkers. When you’re both stuck, it comes down to your combined ability to search correctly, make intelligent guesses, and detect when you’re on the right track. This is the story of my greatest personal project, and how it helped me develop these skills.

I’ve known that I wanted to be an engineer since I first saw The Matrix back in 1999. I saw those black screens with the green letters in Neo’s tiny, dimly lit apartment and I dreamed of being just like that. Seventeen years later I graduated from NYU with a computer science degree, and over time my windows turned blacker and blacker. I was now working in a beautiful dark-themed IDE with pretty colors. I thought I must have looked so cool if anyone looked over my shoulder and saw what I was working on. However there was one black screen that still eluded me — the command line.

Throughout my years in school, I always imagined that it was right around the corner. In my next class, they’ll actually teach us how to use git instead of just explaining why the implementation is so interesting. Maybe after we finish talking about the nature of the shell, implementing piping, and the beauty of unix environment manipulation, they’ll actually teach me how to do something with these tools. But it never came. Suddenly I had graduated, and while I had plenty of knowledge of data structures, algorithms, and how computers work, I still couldn’t do much besides change directories, git commit, and some basic file management.

I decided to take matters into my own hands. I rented a server (bare metal — because it sounded cool), and decided I’d dive in and learn it myself. Take away the GUI, and I’d be forced to learn my way around the command line. I didn’t even have a real plan in mind as I paid for the first month, but already it was a dream come true. An entire computer in a rack somewhere that I’d never see in person, but I could command it to do anything I wanted! And since I was paying for it, the urge to get my money’s worth would drive me to make it useful.

First, I just read a bunch of guides. I created a new user, modified my ssh configs, and locked down the firewall. I copy/pasted someone’s bashrc. I set the date and time. I started to learn VIM. Soon, I started coming up with things to do. I could use it to host a website and save myself the hosting costs of a separate service! I installed Apache and read up on how to configure virtual hosts. I ended up hosting multiple services on this server, and I learned how to keep them separate, how to handle various redirects, reverse proxy to services running on local ports so as not to expose them directly, and apply basic authentication (and later digest auth) to keep certain areas private. Many of these services were given their own users and I managed their groups and permissions so that they could work together smoothly.

I set up cron jobs to handle various menial tasks that I realized could be scripted. One of my favorites runs every night, scans my Spotify playlists for new songs, and then adds those to both starred and my music. I got comfortable with the directory structure of a linux system. I wrote service files to automatically start applications after booting, and I set up a mail server and I was admin@<myserver>.net! Now I could automatically email myself logs from my cron scripts and alerts if my services went down, or anyone SSH’d into the server. I knew enough to write my own bashrc, just the way I wanted it. I modified the welcome message and grinned every time I logged in.

It was hard. I didn’t have anyone to go to for help when I got stuck. It was VERY different from debugging my code — it was all about configs and logs instead of functions and breakpoints. I spent days on some problems which seem small now. Setting up a mail server was shockingly hard the first time around — there were multiple packages to install, multiple configuration files to modify in precisely the right ways, DNS records to be set containing identification keys, and then more configuration files and permissioning to enable encryption.

I googled and googled and kept trying new things and taking risks when I was unsure how to proceed. I broke things, I fixed things, I trashed entire projects and started over with new tools. I learned quickly that I could easily get stuck on the same thing twice, and I started recording detailed notes on every new tool I learned to use. I now keep a folder full of those notes which is synced between all my devices, and whenever I solve a new problem or learn a new useful command, I add to that knowledge base. They’re my own personal notes, tailored to my problems, and sometimes I leave snarky messages for future me. Here’s an excerpt from my vim notes:

better vim for mac:

   brew install vim –override-system-vim

edit files that you forgot to open with sudo:

:w !sudo tee %

https://stackoverflow.com/questions/2600783/how-does-the-vim-write-with-sudo-trick-work

if you have trouble with your vimrc (I had python issues), this is what fixed it for me:

$ sudo apt-get install vim-nox-py2

$ sudo update-alternatives –config vim

select vim-nox-py2 as default

-_- aaand 15 minutes later it’s broken again and I can’t fix it.

Solution to YCMDaemon issues: actually read the installation instructions in the readme

 

I’ve upgraded servers multiple times since those early days, and gone through more than a few disasters. I once bricked an entire server and I still have no idea how. One time, I uploaded a lot of data to a new server before realizing that I wanted to merge multiple hard drives into a single logical group. I managed to do the partitioning live, on the server, without losing any data.

This has been the greatest learning experience and adventure of my life. The skills I’ve learned and continue to learn are powerful, and I use them every day on the job at Domio. The command line skills I learned allow me to work more efficiently both locally and on remote servers. My improved understanding of networking and deploying services to a raw linux server help me make valuable contributions to infrastructure discussions. The fastidious note taking behaviors that I picked up out of necessity are the same techniques that I use to write clear specifications and documentation.

More than anything, the challenges I’ve faced have given me the tenacity to keep pushing — trying and failing and trying again when I can’t figure it out and no one can give me the answer. You need this when you’re no longer in school where the teacher has the answer key. At Domio, we’re often working on issues that no one in the room knows how to solve, because they’re original ideas and no one has solved these problems before. We all have this drive, and all engineers should have it. I’m so lucky to be a part of this team, where my skills are growing faster than ever.

While my personal server project has taught me invaluable skills, this is just one example of how to grow as an engineer. Through my work at Domio I’ve learned about measures that need to be taken to maintain integrity and prevent accidents. I’ve seen our infrastructure grow from a single server to 3 different environments with dozens of VMs, firewalls, load balancers, and services. I’ve seen our test suite evolve through two different testing libraries, the implementation of CI/CD, and code coverage analysis. I’ve felt the difference — an extensive test suite makes me feel confident that I’m not overlooking any edge cases as I make changes. I’ve seen 4 engineers passionately debate variable names, directory structures, approaches to agile methodologies, where to store our documentation, and how to structure it. These are experiences I could never have had on my own, and that’s what keeps me hungry for more every day here.

Right now we’re looking for more hungry, passionate engineers. If you can relate to what I’ve written here, please get in contact with us!