Over Christmas, I caved and ordered myself an Amazon Alexa device: the Echo Dot. Granted, this was on the heels of a conversation with a friend that there was a way for me to build my own “voice command” support. This intrigued me enough (really, it was just an excuse) to drop $50 and then see if I could build something.

A month prior, I finally put one of my Raspberry Pi’s to work in my apartment. For those who have never seen it, I have a pair of KRK’s Rokit 6 monitors. I’ve been a huge fan of the “rp” series for years (even since a college friend showed me his pair). Once I had sole say over what goes into my living room, obviously these became the centerpiece. To take advantage of this, I picked up a little Chromecast Audio to let me, or anyone else visiting, throw on some tunes without any wires.

rokit

One big problem I had was that I didn’t want to leave these expensive studio monitors on all of the time as they would obviously have no use when I’m working or sleeping. The best solution I had for a while was a lightswitch. While effective, it always generated weird looks when I said “flip the switch”. So I posed the question: could I control power to the monitor via the Raspberry Pi?

After some searching, I remembered my good friend Eric from my days at RPI mentioned the PowerSwitch Tail II. This device has the standard US 120v plug with 5V DC switching. This ultimately means that the GPIO pins on the Raspberry Pi can directly control 120v power flowing to some device. Then, the question becomes, how do I interact with the device?

I believe one of the first libraries I found to interact with the GPIO pins on the Raspberry Pi was a Python library. As such, my next idea was to see if I could build a web-server using Flask. Within the hour, I had a trivial webpage up with unstylized “on” and “off” buttons. Putting this all together, I now had the ability to visit a webpage on my phone (while on my home WiFi) and turn the monitors on and off!

rokit

Now, enter Alexa. The next question was: could I build some kind of service in which I could speak to Alexa and have it invoke that same REST call I was making via the browser on my phone? Alexa has the notion of “skills”. As of this time, an Alexa skill can be built using Amazon’s “Smart Home Skill API” or as a “Custom Skill”. While the “Smart Home API” sounded right up my alley, after spending a day or two trying to make it work, I found that it really wasn’t what I wanted to use. I had something built using AWS Lambda (which was pretty cool, actually) and was able to control my monitors through the Raspberry Pi, but then found out I needed to create an OAuth2 authentication service. What I really wanted was a service that was only for my use. So, I dumped that approach and started looking into the Custom Skills API.

The Custom Skills looked much more like what I wanted to do: write a web-service, accept certain specially formatted requests, and take some action. Amazon only provides a Java implementation as an example, but it appears that the community has developed a number of bindings in other languages. Since I was already in a Python and Flask mode, I was overjoyed to find the flask-ask library. Within an hour, I was speaking to Alexa and controlling the power of the monitors from the comfort of my couch.

One point that I’ve glossed over is security. Obviously, I don’t want some troll on the internet flipped the power on the speakers on and off every second as it will break something. Thankfully, I’ve still been maintaining my Linode server since college (shameless referral link) which is public-facing. I have a collection of services running there sitting behind nginx, secured with certificates from Let’s Encrypt (who are also awesome). So, I can easily deploy my flask-ask application to my Linode server, expose it via HTTPS (as per Amazon’s requirement), but how can I forward those requests back into my apartment to the Raspberry Pi?

After doing a little bit of reading, I realized it will be extremely simple. One of the other things I already had set up on this Linode server was an SSL VPN. Once I started working remotely, I set this up to make sure that, if there was ever any sketchy networks I was on while traveling, I could be a bit more secure. For the purposes of exposing this Raspberry Pi to my Linode server, I realized the VPN would be a secure way for the two nodes to see each other and be networked (most importantly without opening up holes in the firewall to my home network).

From a very high-level, this is what the final architecture looks like. For context, “Hal” is the name I chose to identify my service (thanks, Keith!). Sadly, this appears to be a hard requirement with the Custom skill API.

  1. “Alexa, ask Hal to turn on the speakers”
  2. A request is sent to my Alexa custom skill running on my Linode server
  3. nginx accepts the HTTPS request and forwards it to the local flask-ask server
  4. The flask-ask server sends an HTTP request to the flask app on the Raspberry Pi over the VPN
  5. The flask app on the Raspberry Pi toggles the Powerswitch Tail II

All things considered, I’m actually really happy with how this turned out (aside from the requirements on having to say “ask Hal”). The flask-ask library is awesome. I’d highly recommend it to anyone who wants to try this out on their own. Best as I can tell, it does everything as it should and is just dirt-simple to implement. The setup I already had with nginx, Let’s Encrypt, and the SSL VPN, made this entire adventure so very easy as well.

I’ve done my best to publish all of the code I wrote in addition to the configuration files to my Github account. Feedback is welcome, and I’d enjoy hearing what everyone else has done with their Alexa :)

The past sixth months of my life have changed quite a bit. On the bright side, I’ve come to see how much I’ve changed since college, mostly for the better, I think. I’ve also started to understand what I really want and need as a person (“started” is also a very conscious choice of words). Now, this isn’t a grandiose exposé about how I’ve learned so much about myself, instead, just an little story about an event that stuck with me.

Recently, I found myself out at a bar with a friend and a couple which I had met before. The couple is nice enough, just acquaintances. Some alcohol entered the equation and we struck up some conversation. After a bit, we got on the subject of the aforementioned “different” times for me, and how those times started with the ending of a rather long relationship.

One of the most difficult parts of the whole transition was the realization that I have to learn how to make friends again. I’m no longer in college, living in a dorm, seeing people I know every day at class. Now, I’m probably the first to admit that I’m not the most extroverted person in the world, so this prospect is a bit terrifying. Most of the time, I’m rather content to spend time alone. Getting out of my home is typically something I have to really commit to and then push myself to follow through on. Otherwise, I’ll just convince myself there’s a number of really good reasons why I shouldn’t go out and do something.

After I explained this, the strangest thing happened: I was told “Really? You seem to be pretty outgoing!”.

And that was it. I was floored. It’s such a simple thing to say, but it really meant a lot to me. Trying to be outgoing and meet new people was something that took a lot of conscious effort. To know that someone thought that I was outgoing and “normal” really meant a lot.

Looking back on the evening, it seems so silly, so simple. But, you know what, it was nice. It felt like a glimpse into the future. Like I’m doing ok.

I’m doing ok.

I’ve long fought with trying to move my meager web hosting off of HTTP and onto HTTPS servers. A big problem for me was getting certificates that would actually be accepted by default in reader’s browsers. Since the mid-2000’s, the only option I knew about that would do this for free was StartCom.

Sadly, there were still issues with this approach. StartCom required you to maintain your own private SSL key to access their web page. Given that they give you certificates that are valid for two years, I lost that private key every time. This isn’t inherently a problem, but it caused me to repeatedly go through their identification phase which, I believe, still involved a human on the other end clicking a button. This was extremely frustrating because it took something I wanted to update on a Saturday evening all weekend until I got approval.

Lately, I have been seeing lots of chatter about LetsEncrypt, a new certificate authority (CA), an organization which is trusted to distribute certificates, that was

  1. Generally accepted by all user browsers that I cared about
  2. Automatable without user-interaction on the CA side.
  3. Free

As of writing this, I’ve now completely removed all of my old self-signed and StartCom certificates that I had been using and replaced them with LetsEncrypt certificates. I’ve not only cleaned up my server, but I also have a repeatable process for the future to reduce my future efforts.

There are some pretty good instructions available on ReadTheDocs: http://letsencrypt.readthedocs.org/en/latest/index.html

My only complaint (which is very minor since the entire LetsEncrypt software is still in beta) was that I had to finagle a few things to work on Gentoo and with Nginx as my webserver.

I was actually shocked that when first running the letsencrypt-auto tool, it tried to install some necessary packages. Sadly, my installation was a little busted so I had to go through and get these packages installed by hand. After that was done, the tool moved past this step without any extra input from me. I was very pleased.

The Nginx support provided with the letsencrypt-auto tool is still experimental as of writing this, so I had to opt for the Manual plugin instead. This wasn’t terrible either since I’m guessing the way I layout vhosts on disk isn’t going to be considered standard (although I think it is was Gentoo recommends).

For others who might be considering switching, here’s a general outline of my steps:

Workflow

  1. Create a new configuration file with a server running on port 80 without SSL.

    vim /etc/nginx/vhosts/letsencrypt.conf

    This is a bare-bones server (`listen`, `server_name`, and `root` directives). I want to keep these in a separate configuration file because I can easily turn off these servers after I get my new certificates.

  2. Run the `letsencrypt` tool

    ./letsencrypt-auto certonly --manual -d penguinsinabox.com

    This command will provide some instructions as a part of the automation, ultimately instructing you to create a specific file with specific contents under the given domain.

  3. Update the Nginx configuration (first-time only)

    I just need to make sure that my servers are using the appropriate letsencrypt cert and key. This is something that I only need to do once as the `letsencrypt-auto` tool maintains a nice directory structure with symlinks that always point to the newest certs/keys.

  4. Turn off the old servers and restart nginx

    mv /etc/nginx/vhosts/letsencrypt.conf /etc/nginx/vhosts/letsencrypt.conf.off && /etc/init.d/nginx reload

    I can then turn off the servers in one fatal swoop, and reload Nginx to pick up the new certs.

Summary

Quite possibly the best part about the above workflow is that it’s exactly the same steps for certificates for a new domain as it is for certificate renewals! I now have one set of steps to run, regardless of what I’m doing. This is awesome.

My biggest hope is that as the software increases in maturity, my list of 4 steps will become invalidated and I can fully automate the entire process via crontab. Maybe if I take a hiatus from the rest of life/work, I’ll help them myself!

2015 was a rough year. The explanation behind why it was so rough deserves its own post, but, for now, I was extremely happy to have a eleven days of “vacation” on the calendar (despite not being completely disconnected from work things all of those days).

The majority of time was spent in Pennsylvania at my parent’s house. I drove up for the normal Christmas festivities and planned to stay until my sister’s 30th birthday on New Year’s day. After high school, I really didn’t keep in touch with more than a few people from the area (this number is shrinking all the time). As such, I saw a big chunk of time which I knew I wouldn’t have anything to do. To me, more than a few days of “nothing” is more stressful than it is relaxing, so I made a plan to try to catch up on lots of open source work that I had been ignoring.

I often don’t talk about the open source work I do (aside from a tweet), but I thought since I was devoting a big chunk of time (about 3 days), it would be good to actually talk about what I worked on and keep me on track while I work.

Disclaimer: I separate my “dayjob” work from my “open source” work. While I primarily write for open source projects at my dayjob, not all of my open source contributions are directly relevant to my current tasks. As such, I don’t view any of this as “work”. This is a hobby and something I enjoy. I call it “work”, but I don’t think of it like my paying “work”.

Apache Accumulo

Apache Accumulo is (probably?) the open source project I’ve been contributing to longest. As such, it’s very easy for me to fly by, picking up lots of fixes very quickly.

Documentation updates

One of the best things you can do for an open source project is to make sure that commonly asked questions are written down in the official documentation. Accumulo has a user manual which is the official reference material for the project. A user had recently asked how to run multiple TabletServers (Accumulo’s per-node process) over one of the project’s mailing lists. I closed ACCUMULO-4072 after writing a new section in the user manual which covers the considerations in running multiple TabletServers on one host, the configuration changes required, and steps to start and stop the other processes.

Release preparation

Another important thing for an open source software project to focus on is making new releases. Accumulo currently has 3 release lines that we port changes to: 1.6, 1.7 and 1.8. 1.6 and 1.7 are our maintenance release lines, while 1.8 is new development work. We also use JIRA to manage our changes, but this requires some effort when deciding what needs to be fixed before a release is made. I went through the open issues for the next releases on 1.6 and 1.7, 1.6.5 and 1.7.1 respectively, and triaged which issues actually should be completed and which issues should be pushed to the next release.

Low-hanging fixes

As I mentioned earlier, it’s very easy for me to come along and pick up lots of little fixes to Accumulo in one swoop. From my issue triaging, I found 8 issues that I’d easily be able to knock out.

  • ACCUMULO-4081 A simple bugfix backport for a concurrency performance issue.
  • ACCUMULO-4082 A simple bugfix backport for another concurrency performance issue.
  • ACCUMULO-3254 Javadoc improvements to Accumulo table properties.
  • ACCUMULO-4036 Removed verbose/unnecessary logging.
  • ACCUMULO-4064 Include version information on startup.
  • ACCUMULO-4094 Documentation on error handling in the Accumulo BatchWriter.
  • ACCUMULO-3274 Avoid some excessive toString()’ing.
  • ACCUMULO-4056 Update a dependency to avoid shipping a vulnerability.

Apache Yetus

Apache Yetus, in their own words, is a collection of libraries and tools that enable contribution and release processes for software projects. In other, they make it super easy to automate the testing to run over contributions from new developers to a project.

In YETUS-263, I contributed a patch which includes a personality for Apache Accumulo. A Yetus personality defines the tasks that should be run over some set of changes. This lets us define things like:

  • Code style verification
  • The automated tests to run
  • Other static analysis tools (e.g. findbugs)

Hopefully, this personality will help Accumulo get to the point where we can easily wire up automated contribution testing which should lessen the amount of effort the developers need to exert to apply user contributions.

Web hosting

I own the (virtual) machine which this blog runs on. I usually enjoy running my own server, as it lets me explore and learn a bunch of new things along the way. Of course, this often results in me finding things broken for months on end.

Jenkins init.d script

One of the big reasons I run my own machine is that I can set it up to do automated builds for open source projects I regularly contribute to. This helps give back to the community in some cases (more tests being run more frequently) and can also offload my own necessary tests from my work machine.

I find that my Jenkins instance likes to die for some reason every now and again. Sadly, the stdout and stderr for the Java process wasn’t being redirected to files which means that I lost the reason why Jenkins crashed. Redirecting this output should help me tweak the process in the future to prevent it dying again.

Monit updates

In looking at Jenkins being dead, I also had to wonder why my Monit instance didn’t alert me or restart it on its own. Somehow, the alert was turned off. So, that was an easy fix through Monit’s HTTP interface.

Apache Slider

Apache Slider is a YARN application designed to make running application on YARN a bit easier. Slider has the notion of “app-packages” which define how some other application should be run on YARN, the configuration properties exposed, and lots of other features. Some app-packages already provided include Apache Accumulo, Apache HBase and Apache Storm.

Apache Tomcat App-Package

In March 2015, I started work on an app-package for Apache Tomcat. Tomcat lends itself well to Slider’s model because HTTP applications tend to follow the REST model in which instances of Tomcat can be dynamically managed instead of statically managed. By December 2015, I needed to get this code finished and committed.

One big feature I needed to add was the ability for users to specify WAR file(s) when creating a Slider application using the Tomcat app-package. This lets Slider ship a single Tomcat app-package and users can define what web applications to run in their Slider app via configuration only. Thankfully, having HDFS as a storage mechanism and YARN’s resource localization support made this extremely simple. Users add a new configuration property which specifies an HDFS URI to a file, Slider will tell YARN to localize that HDFS to local disk when it creates the YARN container, and then Slider makes sure the WAR file is included in the local Tomcat installation.

Along the way, I also found a few other issues with Slider that bothered me, mostly related to the Slider web-UI.

  • SLIDER-1040 General HTML/CSS formatting issues
  • SLIDER-1041 Application exports aren’t included in export list endpoint

I’m really excited that I finally was able to push this feature in as it was something I did solely on my own time (even though my initial Slider interactions where dayjob related). Tomcat on YARN was something that was commonly asked for my users, but we only ever had a “sorry, we’re working on it” answer to give. But, now that’s all changed!

One missing piece in dynamically deploying web applications on YARN is how clients find and use these HTTP servers. While this is something we could solve at the Slider “level”, it makes more sense to work on this at the YARN “level” instead. It’s a bit of a copout, but it did let me commit the code I have today while (hopefully) contributing as the necessary improvements are made in YARN. Look for this new feature in Apache Slider 0.91!

After it’s all done

I’m really glad I forced myself to write this all down. It’s too easy to trivialize the work you do on your own time. This creates a cycle of negativity for me where I feel bad that I don’t work on my side projects, but I get discouraged from working on them because I don’t feel like I am making “enough” progress. In the end, this was helpful for me and that’s great. I hope others can also benefit from this list, serving as some motivation to write some more code too.

Final Winter Break 2015 Tally:

  • 3 different open source projects.
  • Contributions spanned bug fixes, new features and documentation.
  • Work included “sysadmin” and development work.

I’ve been neglecting this site. I know.

Given that the current year has had enough happenings to fill multiple years already, I’ve been thinking that I should try to get in the habit of writing more regularly. Writing is a skill which I was previously much more adept in. After many years without critiqued practice, I’ve quite quickly fallen off the literary wagon, but that’s a talk for another day.

However, the relevant problem I kept running into was that I never wanted to write because I knew it was going to be a manual pain in the rear to actually deploy the new content I wrote. Well, after being frustrated with this for who knows how long, I finally did something about it.

The Infrastructure

As I’ve previously written, after fumbling around with a couple of different solutions, I found my technology of choice for writing content on the Web in Jekyll.

The other relevant piece of the puzzle is Gitolite which I use for some extra control over my Git repositories. While I do love Github for my programming projects, I’ve found that self-hosting is rather convenient for the more infrastructure-related projects. More importantly, self-hosting is great for automating deployment of updates to this site.

Post-Commit Hooks and Gitolite

Like a vanilla-Git repository, Gitolite provides the means to run a script on the server when changes are pushed. In this case, when I push some edits to the CSS or write a new post, I commit and push my changes which then triggers a script on my server called a hook.

This hook create a fresh clone of the repository (to avoid any issues with trying to fully clean and update an existing repository), generate the site using Jekyll and then copy the results into the configured web-root. The tricky part of this was working around Gitolite.

Gitolite has this interesting approach where it uses specifically named public-keys in Gitolite’s admin repository. Gitolite accepts SSH connections for a single user, identifying the real end-user by the provided SSH key. It would also be a faux paus to allow local filesystem access to Gitolite’s property, so I knew I couldn’t take a shortcut by avoiding SSH completely. So, after forgetting how to properly configure that a few times, realizing I needed to make a brand new key pair to avoid a password on the private key, and then fixing a few Unix file-permissions issues, I finally got the hook working.

Write-Commit-Push-Enjoy

The end result is that, suddenly, my time to deploy some new changes has essentially been reduced to zero. And I couldn’t be happier. I can’t help but think that it might even have a positive impact on the quantity of my writing (hey, I’m writing this the night of).

The reason I find this important enough to write about is that automation has been something often on my mind lately during “normal working hours”. I can’t help but see over and over again the large human cost when machine automation is missing. We make life, both present and future, so unnecessarily difficult simply because we either fail to recognize that there’s even a solution to automate from the start or we consciously decide that the effort required to automate that solution is just too excessive.

From the help of a few great engineers over the past 12 months, I’ve at least come to understand the folly in accepting either of these excuses as valid. It’ll be a cold day in Hell when some series of manual steps for any task is a true one-off. Now I’m not saying I’m an expert in the practice I’m preaching, but at least I’m a bit more cognizant of the problem now. The smile that crossed my face when I pushed some code and immediately saw it deployed did bring a wide grin to my face. That’s motivation enough for tomorrow to try to find something else to automate.