The past sixth months of my life have changed quite a bit. On the bright side, I’ve come to see how much I’ve changed since college, mostly for the better, I think. I’ve also started to understand what I really want and need as a person (“started” is also a very conscious choice of words). Now, this isn’t a grandiose exposé about how I’ve learned so much about myself, instead, just an little story about an event that stuck with me.

Recently, I found myself out at a bar with a friend and a couple which I had met before. The couple is nice enough, just acquaintances. Some alcohol entered the equation and we struck up some conversation. After a bit, we got on the subject of the aforementioned “different” times for me, and how those times started with the ending of a rather long relationship.

One of the most difficult parts of the whole transition was the realization that I have to learn how to make friends again. I’m no longer in college, living in a dorm, seeing people I know every day at class. Now, I’m probably the first to admit that I’m not the most extroverted person in the world, so this prospect is a bit terrifying. Most of the time, I’m rather content to spend time alone. Getting out of my home is typically something I have to really commit to and then push myself to follow through on. Otherwise, I’ll just convince myself there’s a number of really good reasons why I shouldn’t go out and do something.

After I explained this, the strangest thing happened: I was told “Really? You seem to be pretty outgoing!”.

And that was it. I was floored. It’s such a simple thing to say, but it really meant a lot to me. Trying to be outgoing and meet new people was something that took a lot of conscious effort. To know that someone thought that I was outgoing and “normal” really meant a lot.

Looking back on the evening, it seems so silly, so simple. But, you know what, it was nice. It felt like a glimpse into the future. Like I’m doing ok.

I’m doing ok.

I’ve long fought with trying to move my meager web hosting off of HTTP and onto HTTPS servers. A big problem for me was getting certificates that would actually be accepted by default in reader’s browsers. Since the mid-2000’s, the only option I knew about that would do this for free was StartCom.

Sadly, there were still issues with this approach. StartCom required you to maintain your own private SSL key to access their web page. Given that they give you certificates that are valid for two years, I lost that private key every time. This isn’t inherently a problem, but it caused me to repeatedly go through their identification phase which, I believe, still involved a human on the other end clicking a button. This was extremely frustrating because it took something I wanted to update on a Saturday evening all weekend until I got approval.

Lately, I have been seeing lots of chatter about LetsEncrypt, a new certificate authority (CA), an organization which is trusted to distribute certificates, that was

  1. Generally accepted by all user browsers that I cared about
  2. Automatable without user-interaction on the CA side.
  3. Free

As of writing this, I’ve now completely removed all of my old self-signed and StartCom certificates that I had been using and replaced them with LetsEncrypt certificates. I’ve not only cleaned up my server, but I also have a repeatable process for the future to reduce my future efforts.

There are some pretty good instructions available on ReadTheDocs: http://letsencrypt.readthedocs.org/en/latest/index.html

My only complaint (which is very minor since the entire LetsEncrypt software is still in beta) was that I had to finagle a few things to work on Gentoo and with Nginx as my webserver.

I was actually shocked that when first running the letsencrypt-auto tool, it tried to install some necessary packages. Sadly, my installation was a little busted so I had to go through and get these packages installed by hand. After that was done, the tool moved past this step without any extra input from me. I was very pleased.

The Nginx support provided with the letsencrypt-auto tool is still experimental as of writing this, so I had to opt for the Manual plugin instead. This wasn’t terrible either since I’m guessing the way I layout vhosts on disk isn’t going to be considered standard (although I think it is was Gentoo recommends).

For others who might be considering switching, here’s a general outline of my steps:

Workflow

  1. Create a new configuration file with a server running on port 80 without SSL.

    vim /etc/nginx/vhosts/letsencrypt.conf

    This is a bare-bones server (`listen`, `server_name`, and `root` directives). I want to keep these in a separate configuration file because I can easily turn off these servers after I get my new certificates.

  2. Run the `letsencrypt` tool

    ./letsencrypt-auto certonly --manual -d penguinsinabox.com

    This command will provide some instructions as a part of the automation, ultimately instructing you to create a specific file with specific contents under the given domain.

  3. Update the Nginx configuration (first-time only)

    I just need to make sure that my servers are using the appropriate letsencrypt cert and key. This is something that I only need to do once as the `letsencrypt-auto` tool maintains a nice directory structure with symlinks that always point to the newest certs/keys.

  4. Turn off the old servers and restart nginx

    mv /etc/nginx/vhosts/letsencrypt.conf /etc/nginx/vhosts/letsencrypt.conf.off && /etc/init.d/nginx reload

    I can then turn off the servers in one fatal swoop, and reload Nginx to pick up the new certs.

Summary

Quite possibly the best part about the above workflow is that it’s exactly the same steps for certificates for a new domain as it is for certificate renewals! I now have one set of steps to run, regardless of what I’m doing. This is awesome.

My biggest hope is that as the software increases in maturity, my list of 4 steps will become invalidated and I can fully automate the entire process via crontab. Maybe if I take a hiatus from the rest of life/work, I’ll help them myself!

2015 was a rough year. The explanation behind why it was so rough deserves its own post, but, for now, I was extremely happy to have a eleven days of “vacation” on the calendar (despite not being completely disconnected from work things all of those days).

The majority of time was spent in Pennsylvania at my parent’s house. I drove up for the normal Christmas festivities and planned to stay until my sister’s 30th birthday on New Year’s day. After high school, I really didn’t keep in touch with more than a few people from the area (this number is shrinking all the time). As such, I saw a big chunk of time which I knew I wouldn’t have anything to do. To me, more than a few days of “nothing” is more stressful than it is relaxing, so I made a plan to try to catch up on lots of open source work that I had been ignoring.

I often don’t talk about the open source work I do (aside from a tweet), but I thought since I was devoting a big chunk of time (about 3 days), it would be good to actually talk about what I worked on and keep me on track while I work.

Disclaimer: I separate my “dayjob” work from my “open source” work. While I primarily write for open source projects at my dayjob, not all of my open source contributions are directly relevant to my current tasks. As such, I don’t view any of this as “work”. This is a hobby and something I enjoy. I call it “work”, but I don’t think of it like my paying “work”.

Apache Accumulo

Apache Accumulo is (probably?) the open source project I’ve been contributing to longest. As such, it’s very easy for me to fly by, picking up lots of fixes very quickly.

Documentation updates

One of the best things you can do for an open source project is to make sure that commonly asked questions are written down in the official documentation. Accumulo has a user manual which is the official reference material for the project. A user had recently asked how to run multiple TabletServers (Accumulo’s per-node process) over one of the project’s mailing lists. I closed ACCUMULO-4072 after writing a new section in the user manual which covers the considerations in running multiple TabletServers on one host, the configuration changes required, and steps to start and stop the other processes.

Release preparation

Another important thing for an open source software project to focus on is making new releases. Accumulo currently has 3 release lines that we port changes to: 1.6, 1.7 and 1.8. 1.6 and 1.7 are our maintenance release lines, while 1.8 is new development work. We also use JIRA to manage our changes, but this requires some effort when deciding what needs to be fixed before a release is made. I went through the open issues for the next releases on 1.6 and 1.7, 1.6.5 and 1.7.1 respectively, and triaged which issues actually should be completed and which issues should be pushed to the next release.

Low-hanging fixes

As I mentioned earlier, it’s very easy for me to come along and pick up lots of little fixes to Accumulo in one swoop. From my issue triaging, I found 8 issues that I’d easily be able to knock out.

  • ACCUMULO-4081 A simple bugfix backport for a concurrency performance issue.
  • ACCUMULO-4082 A simple bugfix backport for another concurrency performance issue.
  • ACCUMULO-3254 Javadoc improvements to Accumulo table properties.
  • ACCUMULO-4036 Removed verbose/unnecessary logging.
  • ACCUMULO-4064 Include version information on startup.
  • ACCUMULO-4094 Documentation on error handling in the Accumulo BatchWriter.
  • ACCUMULO-3274 Avoid some excessive toString()’ing.
  • ACCUMULO-4056 Update a dependency to avoid shipping a vulnerability.

Apache Yetus

Apache Yetus, in their own words, is a collection of libraries and tools that enable contribution and release processes for software projects. In other, they make it super easy to automate the testing to run over contributions from new developers to a project.

In YETUS-263, I contributed a patch which includes a personality for Apache Accumulo. A Yetus personality defines the tasks that should be run over some set of changes. This lets us define things like:

  • Code style verification
  • The automated tests to run
  • Other static analysis tools (e.g. findbugs)

Hopefully, this personality will help Accumulo get to the point where we can easily wire up automated contribution testing which should lessen the amount of effort the developers need to exert to apply user contributions.

Web hosting

I own the (virtual) machine which this blog runs on. I usually enjoy running my own server, as it lets me explore and learn a bunch of new things along the way. Of course, this often results in me finding things broken for months on end.

Jenkins init.d script

One of the big reasons I run my own machine is that I can set it up to do automated builds for open source projects I regularly contribute to. This helps give back to the community in some cases (more tests being run more frequently) and can also offload my own necessary tests from my work machine.

I find that my Jenkins instance likes to die for some reason every now and again. Sadly, the stdout and stderr for the Java process wasn’t being redirected to files which means that I lost the reason why Jenkins crashed. Redirecting this output should help me tweak the process in the future to prevent it dying again.

Monit updates

In looking at Jenkins being dead, I also had to wonder why my Monit instance didn’t alert me or restart it on its own. Somehow, the alert was turned off. So, that was an easy fix through Monit’s HTTP interface.

Apache Slider

Apache Slider is a YARN application designed to make running application on YARN a bit easier. Slider has the notion of “app-packages” which define how some other application should be run on YARN, the configuration properties exposed, and lots of other features. Some app-packages already provided include Apache Accumulo, Apache HBase and Apache Storm.

Apache Tomcat App-Package

In March 2015, I started work on an app-package for Apache Tomcat. Tomcat lends itself well to Slider’s model because HTTP applications tend to follow the REST model in which instances of Tomcat can be dynamically managed instead of statically managed. By December 2015, I needed to get this code finished and committed.

One big feature I needed to add was the ability for users to specify WAR file(s) when creating a Slider application using the Tomcat app-package. This lets Slider ship a single Tomcat app-package and users can define what web applications to run in their Slider app via configuration only. Thankfully, having HDFS as a storage mechanism and YARN’s resource localization support made this extremely simple. Users add a new configuration property which specifies an HDFS URI to a file, Slider will tell YARN to localize that HDFS to local disk when it creates the YARN container, and then Slider makes sure the WAR file is included in the local Tomcat installation.

Along the way, I also found a few other issues with Slider that bothered me, mostly related to the Slider web-UI.

  • SLIDER-1040 General HTML/CSS formatting issues
  • SLIDER-1041 Application exports aren’t included in export list endpoint

I’m really excited that I finally was able to push this feature in as it was something I did solely on my own time (even though my initial Slider interactions where dayjob related). Tomcat on YARN was something that was commonly asked for my users, but we only ever had a “sorry, we’re working on it” answer to give. But, now that’s all changed!

One missing piece in dynamically deploying web applications on YARN is how clients find and use these HTTP servers. While this is something we could solve at the Slider “level”, it makes more sense to work on this at the YARN “level” instead. It’s a bit of a copout, but it did let me commit the code I have today while (hopefully) contributing as the necessary improvements are made in YARN. Look for this new feature in Apache Slider 0.91!

After it’s all done

I’m really glad I forced myself to write this all down. It’s too easy to trivialize the work you do on your own time. This creates a cycle of negativity for me where I feel bad that I don’t work on my side projects, but I get discouraged from working on them because I don’t feel like I am making “enough” progress. In the end, this was helpful for me and that’s great. I hope others can also benefit from this list, serving as some motivation to write some more code too.

Final Winter Break 2015 Tally:

  • 3 different open source projects.
  • Contributions spanned bug fixes, new features and documentation.
  • Work included “sysadmin” and development work.

I’ve been neglecting this site. I know.

Given that the current year has had enough happenings to fill multiple years already, I’ve been thinking that I should try to get in the habit of writing more regularly. Writing is a skill which I was previously much more adept in. After many years without critiqued practice, I’ve quite quickly fallen off the literary wagon, but that’s a talk for another day.

However, the relevant problem I kept running into was that I never wanted to write because I knew it was going to be a manual pain in the rear to actually deploy the new content I wrote. Well, after being frustrated with this for who knows how long, I finally did something about it.

The Infrastructure

As I’ve previously written, after fumbling around with a couple of different solutions, I found my technology of choice for writing content on the Web in Jekyll.

The other relevant piece of the puzzle is Gitolite which I use for some extra control over my Git repositories. While I do love Github for my programming projects, I’ve found that self-hosting is rather convenient for the more infrastructure-related projects. More importantly, self-hosting is great for automating deployment of updates to this site.

Post-Commit Hooks and Gitolite

Like a vanilla-Git repository, Gitolite provides the means to run a script on the server when changes are pushed. In this case, when I push some edits to the CSS or write a new post, I commit and push my changes which then triggers a script on my server called a hook.

This hook create a fresh clone of the repository (to avoid any issues with trying to fully clean and update an existing repository), generate the site using Jekyll and then copy the results into the configured web-root. The tricky part of this was working around Gitolite.

Gitolite has this interesting approach where it uses specifically named public-keys in Gitolite’s admin repository. Gitolite accepts SSH connections for a single user, identifying the real end-user by the provided SSH key. It would also be a faux paus to allow local filesystem access to Gitolite’s property, so I knew I couldn’t take a shortcut by avoiding SSH completely. So, after forgetting how to properly configure that a few times, realizing I needed to make a brand new key pair to avoid a password on the private key, and then fixing a few Unix file-permissions issues, I finally got the hook working.

Write-Commit-Push-Enjoy

The end result is that, suddenly, my time to deploy some new changes has essentially been reduced to zero. And I couldn’t be happier. I can’t help but think that it might even have a positive impact on the quantity of my writing (hey, I’m writing this the night of).

The reason I find this important enough to write about is that automation has been something often on my mind lately during “normal working hours”. I can’t help but see over and over again the large human cost when machine automation is missing. We make life, both present and future, so unnecessarily difficult simply because we either fail to recognize that there’s even a solution to automate from the start or we consciously decide that the effort required to automate that solution is just too excessive.

From the help of a few great engineers over the past 12 months, I’ve at least come to understand the folly in accepting either of these excuses as valid. It’ll be a cold day in Hell when some series of manual steps for any task is a true one-off. Now I’m not saying I’m an expert in the practice I’m preaching, but at least I’m a bit more cognizant of the problem now. The smile that crossed my face when I pushed some code and immediately saw it deployed did bring a wide grin to my face. That’s motivation enough for tomorrow to try to find something else to automate.

It’s been in the back of my mind for quite some time now to get back on here the start writing some more. Sadly, I typically psych myself out instead and end up with nothing. Luckily, I guess you can call it luck, I’ve had quite a tumultuous past six months.

New job

Back in October of 2013, I joined Hortonworks. Looking back, it still feels surreal, dreamlike even. Since graduating from RPI in 2010, the thought of joining a Silicon Vally software company, a company in the Mecca of software development, appeared to me as this unattainable goal. With the help of a friend and the support of an amazing lady, I took a leap to what I thought was a far-off platform, something I’d maybe get close to. Maybe something I could try again in a few months for. Turns out: I was wrong. It’s been a few months now, and it’s been a great experience. I’ve the opportunity to met more than a handful of people who I only knew by name, profile picture and open source contributions. All and all, it’s been awesome and I’m glad that I had the encouragement from others to try to follow my dreams.

Half a decade

Speaking of people who give me encouragement, I’ve also had the pleasure to spend the last five years with Laura. After a few years of dating at RPI, we’ve been living together for about three and a half years now. And I wouldn’t want it any other way.

I never really considered that I’d find myself in a committed relationship this early in my life already (I suppose that’s a rather “new” way to consider things given that I’m 25), but I’m here. And I kind of like it. I remember back in college when Laura and I were still learning about each other when we were both “learning” how to get back into a long-term relationship. We both had our quirks, but we worked through those, and boy am I glad we did. I know I don’t tell her enough, although she’d probably give me crap if I told her any more often than I presently do, but I’m not sure what I’d do if I didn’t have Laura around. She’s been amazing over the past six months in more detail than I care to publicly go into, and words don’t do it justice to describe how happy I am that she’s in my life.

Never discard anyone who is willing to deal with more tough times to help you follow your dreams. Hold on to that person and don’t let them go.

Death scare

I’ve already said that I’m 25, but I had quite a scare recently that I never expected to have. In short, I had a doctor in the emergency room tell me, “You’re lucky that you came in here today. You saved your own life.” That statement took a few days to sink in (about the duration of my hospital stay), but I am also one that deals with things by, well, not dealing with them. Sitting in the ER, I don’t think I’ve ever had such a high percentage of puns/jokes for total things I said.

After about three or four days of dealing with tremendous leg pain (I had trouble putting any amount of weight on my one leg for that period), I dragged (only half-kidding) over to an urgent care establishment close by where I live. Now, the emergency room doctor told me that I saved my life, but I tend to think that the doctor who saw me there is actually the one who saved my life, as he’s the one who still told me to go up to the hospital and get checked to see if I had a blood clot in my leg, despite having “none” of the risk factors.

Up the road to the hospital I went. After getting sent around the hospital a couple of times (thanks, receptionist), I finally got myself admitted and the ultrasound was administered. Sure enough, the nice lady scanning my leg got a super-serious look on her face and repeatedly told me “I’m really glad you came here”, “I’m really glad you’re here”; that was just the start of it.

Just like that, I was diagnosed with a deep venous thrombosis (DVT), or, in layman’s terms, a blood clot in my leg. To the emergency room waiting area I went. After a few hours in a wheelchair (as I was told that I wasn’t allowed to walk anymore), I was taken back for a CT scan of my chest to make sure that I didn’t have a piece of that clot also in my chest. This, is where things got scary.

About an hour or two after the CT scan, laying on a bed in the emergency room, the doctor came in and delivered me that wonderful news that I had a pulmonary embolism (PE), or, again in layman’s terms, a blood clot in my lungs. For those who might not know it: this is bad. Very bad. Lots of really bad things can happen to the patient if that clot decides to move closer to your heart. Bad things that could have easily put me under the knife that day. Luckily, I had somebody watching over me that day, and the rest of that week, and didn’t have to do any of that.

Fast-forward three days through a bunch of intra-venous fluids and medicines, lots of blood samples, and lots of crappy television, I was released back home, still with leg pain, but also with ample thinned blood and a healthy heart. While the jury is still out on why I developed this clot(s) in the first place, I find myself very lucky how this all fell into place in the end.

And finally, a public service announcement: if you ever find yourself in a situation where you have intense leg pain, especially when you put weight on it, go see a doctor immediately. Do not mess around with this stuff. Sitting in a desk chair for the better part of my day probably didn’t help my situation, but it’s usually not sufficient to cause a blood clot to form, so don’t worry too much my programmer friends.