# Building a Personal Background Job System
11/15/2021
This weekend I built something that many have tried and most have succeeded to do - Build a background job system. This time, however, it has a bit of a twist in that this is for personal, for fun jobs. This means I don't need globally-distributed, fault-tolerant services. Nor do I need fancy things like exponential backoff for retries or health probes. I simply needed something to run some code and tell me the results (gasp, how original). So with that goal in mind, I started off with a list of ideals to see if I could make something work.
# Core Goals
- Easy to add new jobs - If I have a new script I want to deploy, I don't want to have to spend a few hours configuring it
- Easy to update existing jobs - If a script is broken or needs some tweaking, again, I don't want to have to spend a few hours fixing it.
- Low overhead - Since this is for personal use I don't have the deep wallets of a company to build the ideally engineered system, rather I have a budget of just a few dollars / month.
- Minimized infrastructure - I want to manage as little infrastructure as possible, ideally none.
- Notification system - I need to be able to see the results in an easily consumable method.
# Considered Approaches
# Luigi
Spotify's Luigi (opens new window) was the first option that came to mind. I had seen it released a few years back and the name stuck in my mind. After perusing the README.md it became obvious pretty quickly that it was far more complex than I needed, so I moved on.
- Easy to add new jobs - ⛔️
- Easy to update existing jobs - ⛔️
- Low overhead - ⛔️
- Minimized infrastructure - ⛔️
- Notification system - ⛔️
# Airflow
I know what you're probably thinking - If Luigi was too complex, well Airflow sure as heck will be too. And to that I say, you're right. I actually gave this a bit deeper look because I had used it before so was hopeful that the previous experience would enable me to iterate quickly. I was able to set up the docker-compose.yml file pretty quickly and had a running copy of Airflow! When it came time to start writing jobs however, I remembered there was a significant amount of overhead in the job definition files. Given than alongside the sheer amount of resources required by the docker-compose file, I arrived at the conclusion that this too would likely be a bit too powerful for my needs
- Easy to add new jobs - ⛔️
- Easy to update existing jobs - ✅
- Low overhead - ⛔️
- Minimized infrastructure - ⛔️
- Notification system - ⛔️
# Azure Functions
I finally arrived at a hosted function solution. As I had investigated the other options, I just couldn't get past the ease of using hosted functions and finally realized that this was likely the right path. Azure has a great integration with GitHub actions that enables you to trivially set up CI/CD with the function, meaning any code changes automatically make it to production on merge with main. Not only was it easy to update, but since consumption plan functions bill only on time used and have a generous free tier I was able to run this for $0/month!
So lets review my check list:
- Easy to add new jobs - ✅
- Easy to update existing jobs - ✅
- Low overhead - ✅
- Minimized infrastructure - ✅
- Notification system - ⛔️
# Solving Notifications
Azure Functions fit the bill with virtually everything except for having a simple mobile notification system. While I didn't have a full solution for this yet, I set to work building the background job system on Azure Functions. While I was working on the system, I received a slack update for my App's sales and a 💡 went off - Slack is a ready-made mobile notification service! Since the API to send a slack message is so straightforward, I was able to quickly add logging to slack to record the results of my jobs. Once that was set up, all I had to do was sign into my workspace on my phone and enable notifications for the given channel.
# Conclusion
So far, GitHub CI/CD + Azure Functions + Slack has been working flawlessly (all 2 days). Setup was trivial and I'm hopeful that maintenance will continue to be low so I can spend time writing jobs rather than worrying about them running.