Accumulating and graphing FM station song history (Noderadio v2.0)

tl;dr

This is the second generation of my (super minimalistic) graphing/visualization project, which visibly shows which FM radio stations have lots of variety (KQRS, KZJK, KQQL), and which don't (KDWB).

Check it out:  https://codepen.io/panurgy/full/YzpGpap?s=kqrs

Introduction

About eight years ago, I built a data-collection system that accumulated and graphed information about the song variety (or lack thereof) across several FM radio stations around Minneapolis MN.  A few years after setting that up, things changed at the various hosting providers, and the entire system fell apart (Heroku Cedar is deprecated, mLab was acquired and shutdown, etc).

A few months ago, my curiosity was rekindled about the variety of songs, so I revisited this project and updated it - using a newer generation of hosting solutions/options.  Some of the code remains the same (and still mentions "use strict" within the functions!), but other parts were rebuilt from scratch.

Getting the data

The first step was figuring out the data acquisition.  All of the radio stations have new websites, which changed (and broke) my song-info collection code (which essentially relied upon "screen scraping" the info from the station's website).  

Rather than writing and deploying server-side code, I used RunKit to create an assortment of individual endpoints, which obtains and returns the "now playing" song information for a specific station. Every station provides their info a bit differently, and fall into three general categories:

• HTML data - the station's website sends a HTML string, which contains DOM elements, and the song info is buried within it. The npm package cheerio works great at parsing out the info.
• JSON data - the station's website returns a JSON string, which is super easy to parse/use. Most of the stations use this format.
• WebSocket - the station's site opens a WebSocket, and uses an "ask/reply" protocol that responds with a JSON string.

All of this information is discoverable using the browser's dev-tools, the biggest challenge is finding the "needle in the haystack" within all the other network requests (advertisements, metrics, trackers, and more trackers).

Here's the list of currently supported stations:

• KQRS - KQ92 (JSON): https://runkit.com/panurgy/kqrs-now-playing
• KTIS - 98.5 (WebSocket): https://runkit.com/panurgy/ktis-now-playing
• KDWB - 101.3 (JSON): https://runkit.com/panurgy/kdwb-now-playing
• KZJK - Jack 104.1 (HTML): https://runkit.com/panurgy/kzjk-now-playing
• KQQL - Kool 108 (JSON): https://runkit.com/panurgy/kqql-now-playing

Storing the data

The next step was finding a place to store the data.  I needed something that could hold a "large quantity" of information, and possessed the ability to index/query upon the song's timestamp. Previously, I used a MongoDB instance (which worked great!), but this time I had issues getting the recent Node/MongoDB drivers working (some native-code dependency wasn't working), so I searched for something else that's "fully cloud" with a REST API.

I decided upon Google Cloud Firestore, because it's collection-based, capable of storing large quantities of data, and provides fast indexing/querying. I also knew that Zapier has an integration that easily connects with Cloud Firestore (disclaimer - I built that integration).

I decided to create a collection for each station, and used the "UNIX/Epoch timestamp" as the document's identifier.  I could have shortened the timestamps from milliseconds down to seconds (and reduced the length by three characters), but decided to leave things as milliseconds, since that works best with JavaScript's Date object. (btw - unix/epoch timestamps are "the best/only" way to store/preserve timestamps in a database/persistence - buy me a beer and I'll share my experiences).

Running the data collection

Once I had those two pieces worked out, the next step was setting up something that ran every few minutes, retrieved the "current song" from the station's website, and placed it into the database. The easiest option was a group of Zaps at Zapier - with a Zap for each station, which looks like this:

• The trigger uses Zapier's Webhook integration, which calls one of the Runkit endpoints that I created
• The filter step discards any data that's missing the song/artist info, which indicates the station is playing advertisements
• The final step saves the data into the correct collection

Since the Zaps run/poll about every five minutes, it's possible that a station might play a short song in between two cycles, but this isn't a production-grade experiment.  Eventually, I might convert/replace the Zaps with Cloudflare's Cron triggers for Workers (assuming I have enough free-time and attention span, which is unlikely).  In the meantime, Zaps are a fantastic way to get things up and running quickly and easily (with observability into how well things are running).

I currently have these five Zaps running, each configured for a specific station:

Rebuilding the front-end

As the Zaps gradually accumulated the historical data, I started working on rebuilding the front-end.  Since the visualization/graphing code is very minimalistic (it's a single static page), and doesn't require any special server, I decided to use a CodePen to host this piece.  Most of the original (and obsolete) front-end code still worked, so I only had to rebuild the parts that interacted with the database - switching things from MongoDB over to Cloud Firestore.  The two biggest challenges were:

• Converting the query - in MongoDB, querying is pretty easy, the database call passes over a "fairly simple" JSON object that contains the search's settings.  In Cloud Firestore, the JSON object is a "bit more complex" and requires a Structured Query.
• Reading the data - in MongoDB, the query returns an array which contains the documents/objects from the database (thus the objects received match the objects in the database).  In Cloud Firestore, the documents aren't "simple JSON", but rather a more detailed format which contains lots of meta-information about each of the document's fields/data-types.  Fortunately, StackOverflow had the answer I needed, to convert those document objects into "plain objects".

Viewing the results

After all that work (and a few days to collect enough historical data), the results show which stations have "lots of variety", and which have "little variety" (and thus, overplay a small set of songs).

For example, KZJK is a Jack FM station, which plays a wide variety (and sometimes random) assortment of music.  During a two-day span, most songs were only played once or twice:

On the flipside, KDWB is a Top 40 station, which plays a small collection of songs more frequently.  During a two-day span, some songs were played 20 times:

Next Steps

There's a long list of enhancements/improvements that I'd like to perform, but it's unlikely that I'll have the time needed to work on things. Such as:

• Switching from Zaps to Clouflare, for faster sampling intervals
• Setup some metrics with Datadog to monitor the data-collection workers
• Possibly setup Sentry.io error logging when things break/fail
• Update and clean up the code, and rearrange it into something more polished

Conclusion

Whenever I'm listening to FM stations, I occasionally encounter a song that seems like it's played "all the time" - but now there's a convenient way to find out whether that's true, or just my perspective (or crappy luck, and listening to a station only when "that song" is playing).

If you'd like to replicate this experiment using your local/favorite FM radio stations, most of the pieces are free to use (Runkit, Cloud Firestore, CodePen).  Zapier's Webhook integration requires a paid plan (but may be available during their introductory trial period?), otherwise you can purchase and cancel anytime. Overall, the biggest challenge is figuring out how to collect/parse the song information from a station's website.

Check it out here:  https://codepen.io/panurgy/full/YzpGpap?s=kqrs

Designing and building the Mini Zapier display

In a previous blog post, I covered the details for a large-scale Illuminated Zapier display, which consisted of 248 tri-colored LEDs that were controlled by NodeJS on a Raspberry Pi.

One of my co-workers mentioned that it would be neat to have a smaller version of the display, which seemed like a great idea for a future project.

The theory/plan:

In May 2017, I started sketching out the possibilities:

When working out the math, it seemed that the quantity of LEDs per arm should be something like 5, 6, or 8.  When I searched around for the available options for LEDs, the best fit appeared to be the Adafruit DotStar 144 LED strip.  That would give me 6 LEDs per arm, and 8 arms per display, which is enough to build 3 full Zapier mini displays (6 x 8 x 3 = 144).

The next part was figuring out how "miniature" I could possibly make things.  Rather than building a custom Pi Hat (that would stack on top of the RasPi), I wanted something that would plug into the RasPi, without adding too much additional (stacked) height. 

The best approach was a Raspberry Pi Zero W, and then connecting it with a custom-designed board that would drive the LEDs. The dilemma was choosing stacked-headers, or right-angle connectors?

The advantage of stacked-headers is that the two rows of pins on the RasPi would line up with the pins on the custom board - whereas the right-angle headers causes things to connect up a little differently (outside rows versus inside rows).

In the end, I went with the (more complicated) option of using right-angle headers/connectors, so that the RasPi and custom board would lay flat together:

After each round of design and planning, I would print out the custom board's layout onto a sheet of paper, punch holes for all of the connectors, and lay out the components, in order to assess the spacing and physical dimensions.  This helped me avoid any pitfalls where the connectors would end up being "too close" and preventing things from being assembled correctly.

I used Autodesk's EAGLE software to design the schematic and layout of the custom board, along with lots of help and learning from some tutorials over at SparkFun. That resulted in the following layout:

When my paranoia was mostly-satisfied that I had all of the custom board's holes and traces/connections arranged correctly, I sent the design off to Osh Park for fabrication.  Their service sent back three copies of the completed board, which matched nicely with the quantity of LEDs that I had from the LED strip.

Some Assembly Required

When all of the parts arrived, the first step involved lots of time with a hacksaw, cutting the board apart, and also cutting up the connectors/headers into the specific sizes:

Then lots of long hours with a soldering iron, getting the connectors installed onto the RasPi and the custom board:

One of the unexpected and most difficult challenges was soldering the LED strip segments.  In my previous project, cutting the LED strip resulted in a full-circle of solder pads on each segment/strip.  Unfortunately, the high-density LED strip only provided a single solder pad in between each LED - thus when cutting apart the strip, I only had a half circle to use for soldering:

Eventually, I was able to solder (and re-solder) enough of the connections to get a few arms working correctly:

And then finally had the finished products:

Lessons Learned

The most difficult part of this project was the assembly/soldering of the LED strips onto the arms - and it's the most brittle part of the project.

When I started this project, there weren't any options available at the time.  Shortly after I started building this project, I discovered the Blinkt product/project, which would likely make the construction less tedious - but would require a larger board, with bigger connectors, and a 3 to 8 demultiplexer (74138 or 74238) along with some custom software/driver tweaks.  There's always a trade-off in these types of projects.

Additional Info

Here are more photos of the construction process.

Here's the GitHub repo to the NodeJS app/code that I use to control the LEDs. 

Tracking workout data, using Zapier, MapMyFitness, and Google Sheets

A few months ago, I joined the Zapier team, and embarked on the mission of making computers perform more work (and thus, reduce the monotony inflicted on us humans).

I'm a huge fitness-data geek, and love all of the stats I can get from apps like MapMyFitness. When I finish a workout, I manually type in the info/stats from that workout over in my daily journal (currently, Evernote), which is time-consuming, and usually error-prone.


Seems like something that a computer could easily perform - and now that's entirely possible!




Using Zapier.com, it's pretty easy to connect up things so that your workout data from MapMyFitness can be fed into Everenote, Google Sheets, or hundreds of other possibilities.



Here's an example that connects things with Google Sheets, which makes it easy to see your workouts in a spreadsheet view.

If you don't already have an account over at Zapier, you can set one up (for free!). Once you're in, you can set up a Zap, which connects data from one source (like MapMyFitness) to a destination (like Google Sheets - which assumes you already have a Google/Gmail account. If not, it's probably time to give up the AOL.com e-mail address you're using).

Let's set up a Google Sheet (spreadsheet) which will receive the workout data from MapMyFitness.

Go to drive.google.com and create a brand new Google Sheet:

In that sheet, setup some column headings, which indicate the fields/data that you will eventually connect up:

At the Zapier web site, go ahead and create a new Zap!

For the Trigger (data source), select MapMyFitness:

For the Action (data destination), select Google Sheets (again - this assumes that you have a Google/Gmail account, and have setup a spreadsheet like the one described earlier in this article):

Step #1 of your Zap should look something like this:

Zap Step #2: Connect your MapMyFitness account:

It will redirect you to the web site for MapMyFitness, where you have to grant access for Zapier to read your Workout data. This does not store any passwords - instead, MapMyFitness returns an "access token" that is used to access your workout information.

Zap Step #3: Connect your Google Sheets account. Same thing as the previous step - we don't store any passwords, just an "access token" provided on your behalf.

Zap Step #4: Filtering.

If you'd like to filter out things (like certain types of workouts), you can select the criteria here. Otherwise, just ignore this and continue to the next step.

Zap Setup #5: Mapping fields

This is the most important step - this is where you select your Google Spreadsheet, and Worksheet within that spreadsheet (since a spreadsheet may contain many worksheets)

Then, you'll need to select the incoming fields from MapMyFitness, and deposit the data into the correct columns.  For each field, use the button "Insert Fields", and select the correct entry. Like this:

Step #6:  Test it out! Go ahead and "Test Zap with this Sample". This will confirm that Zapier is able to receive Workout data from MapMyFitness. If you already have workout data present, you'll see the three most recent workout entries appear like this:

Step #7: Name the Zap, use something that will help you remember what it's doing, like "Track my workouts"

Go ahead and turn on the Zap, and then go for a walk/run/ride/swim or whatever you do with MapMyFitness. As you complete new workouts, they will accumulate within that spread sheet. Like this:

From there, the possibilities are endless - such as functions to display your total mileage for this year!

. 

It's a bomb^H^H^H^H clock!

A month ago, there was a huge amount of discussion over a clock that a kid brought to school, which had him arrested and suspended. I have a clock obsession, but only for really weird/geeky clocks. My current favorite is the Tix Clock available at ThinkGeek.

Since I'm usually a month (or more) behind on my blogging, here's my story.

Back in 1998, my dad happened to have a bunch of dual-color LEDs, consisting of Green and Red (and thus, Yellow when both colors were illuminated) that he no longer needed. When I counted them out, there were just over 60 LEDs present - which made me wonder if I could build a clock?

I had spent too much time building all kinds of fun things using various forms of the 8051 microcontroller, and since I had the EEPROM burner and UV eraser in my home lab, it was the best fit for the project.  (These days, I'd much rather use an Arduino or something).

Since a clock would require 60 individually addressable LEDs, along with two individually addressable colors, that equates to a lot more pins than what's available on the 8051.

Multiplexing to the rescue!

I'm a huge fan of multiplexers (and serial shift registers). In this case, I took the 60 LEDs and divided them into four quadrants. I could use two pins to select a quadrant, and then use four pins to select 16 different possible LEDs within that quadrant, using the 74154 4x16 demultiplexor. Throw in the two pins needed for the color selection, and I could drive all of the necessary data signals with eight little bits!

Since I had a few extra data pins available, I hooked up a few push buttons. Two are to set the minutes forward/backward, and two are to set the hours forward/backward. There's a fifth button, but I don't remember what it does.  Maybe some day I'll find the old assembler code I wrote for this project, and see if there's any mention of its purpose.

Assembly

Any true hacker's project rarely comes in a "polished/presentable" case. So the next question was what kind of packaging to use for housing this project? I needed something round, and a large enough size to house a custom circuit board that I made. I thought of a frisbee, but they weren't deep enough for all of the wiring (done by hand - ugh!).  The next best candidate was an ol' fashioned pie tin!  After stuffing everything inside it, I used a piece of wire mesh to hold everything inside it.

Final result

Here's the "internal view".  The 8051 is the big 40-pin beast in the middle (with electrical tape covering the UV-EEPROM window). The push-buttons are just above the CPU. The 74154 is in the top-right corner. There's a bunch of 7407 inverters spread around the perimeter, which are used to drive the LEDs (common cathode/ground), along with a pair of transistors that supply the red/green anodes on the LEDs. Yes, I used to have a lot of spare time back then.

Action shot: The clock displaying the time 3:01:36 o'clock.

Semi-useless video clip:

I'm really thankful I never brought it into school for "show and tell". What's your favorite geeky clock? Leave a comment below!

Grandma's Marathon 2015

   After all of the grueling fun of running Twin Cities Marathon last fall, I decided to do another full marathon (uff-da!) 

  Here in Minnesota, everyone talks about Grandma's Marathon, so it seems to be the "premiere running event" of the upper Midwest. After having completed the run, it's nice and all, but it doesn't compare to the energy and entertainment of the Twin Cities Marathon. 

  Before the race started, the skies opened up and poured rain on top of the runners, as we waited in a car dealership parking lot. Some of the more "dry-minded" folks hit the pavement, and rolled beneath the various SUVs and "higher clearance" vehicles on the lot. Lots of folks (including me) pulled out the disposable trash bag, and threw it on as a temporary poncho.

  The rain lightened up during the first half of the run, and had pretty much stopped by mile 13. Unfortunately, everything was completely soaked inside and out. By mile 15, my right sock formed a soggy wrinkle beneath the knuckle of my big toe, and a blister ensued. I spent the later miles pushing harder with my left leg, in order to take the pressure off my right foot.

  All of that imbalanced running caught up to me around the 22 mile marker (and Lemon Drop hill), when my left quad started cramping up. I spent the last 4.2 miles alternating between a carefully balanced jogging stance, or just walking and massaging the cranky quad muscle, trying to coax a little more effort out of it.

  In the end, I managed to squeeze out a 4:10 finish, and a couple of free beers (whew!)

Running the Numbers

Over the last few years, I've participated in a few running events - which makes it difficult to mentally keep track them, and my completion times (fortunately, I have an Evernote Premium subscription to help with all of that). 

Some people consistently run the same event/distance every year, so they have no problem remembering their PRs. I like to try out new events/courses, and occasionally mix in a familiar event from a previous year - which further complicates the mental organization of stats.

Fortunately, most of these events have the finishers' information available via their websites, so with a bit of searching, I can eventually recall my stats from the past. 

But this brings up the question: Wouldn't it be great if there was a web interface to search all of those web interfaces simultaneously? 

Swiss Army Node

If you haven't already noticed from my previous blog posts, I love Node.js (and/or io.js -  hopefully they'll become the same thing again). Node.js is great at handling lots of input/output requests asynchronously, efficiently, without needing lots of code - especially if you use Promises to manage the application flow. 

Working from the back to the front

I prefer to start applications "from the back" - making sure that it's feasible to gather the necessary data, finding out what the limits are, and building an API that the front-end can use. 

I started with a list of popular running events in the area, and discovered that three of them used the mtec service to host the results for the past few years, which makes it super easy to get things started: 

• Monster Dash Half Marathon and 10-Mile
• Twin Cities Marathon and 10-Mile
• Grandma's Marathon and Half Marathon

Using Chrome Dev tools, I looked at the request being sent via their search form, and the markup that's returned (yeah, markup) - which can be a bit dangerous if it's not escaped correctly. For example, if a person has an apostrophe in their last name, this returned from their server:

<tr ...>  <td href="#">Firstname O'Lastname</td>  ... </tr>

Ideally, the apostrophe would be &#39; (but not mandatory). But this makes me wonder what would happen if I signed up for a race and entered my name as  <img src="..."/> (which reminds me of Bobby Tables).

Putting the pieces together

In the Node app, here's how I chose to implement the searching:

• request and bluebird: From the back-end, I used the request package to create the HTTP/client connection to the race-results provider. Instead of using a callback function, I prefer to use Promises, so I wrapped the library using bluebird.promisify. 
• cheerio: The race-results provider returns markup, but I'd prefer to use JSON for passing data. The cheerio package makes it super simple to use jQuery style selectors for parsing the text from the markup. 
• bluebird again: In order to query lots of races simultaneously, I make a bunch of HTTP search/requests (which are now promises), and then use bluebird.all to wait until all of the searches have completed. Ideally, there should be a timeout promise in here so that if any of the individual search requests stalls, it won't delay the other searches.

Back to the Front

I've enjoyed using Angular UI Bootstrap on previous projects, so I wanted to branch out and try using Angular Material library. They share a lot of common concepts, such as the Bootstrap Grid versus the Material Design Layout Grid. It's nice to have resources that get a web interface up and working quickly and easily.

The UI is extremely simple, consisting of a single view that contains a few input fields, which are forwarded onto the back-end race-results providers.

To the Cloud

Given the straightforward/simple implementation of this app, it was easy to deploy into a Heroku Dyno. The app spends most of its time sleeping, so the initial request is always very slow (takes about 5 seconds for the Dyno to awaken).

Here's the deployed/running app: http://runcreeper.herokuapp.com

Show me the Code!

The code is available on GitHub at:  https://github.com/panurgy/run-creeper

Handbell Hero!

Overview

It's like Guitar Hero, with 4 to 8 players, using a set of handbells from Groth Music. No musical background is necessary to play!

Background

Every year, HelpSystems has an annual employee luncheon, which also involves some kind of game/entertainment thing that's led by a department.


Our department manager, Matt Bresnan, came up with the idea of putting together a Guitar Hero type of game, but using a set of handbells. I was asked to look into the feasibility of writing this app, and was given the MythBusters directive that If it's worth doing, it's worth overdoing!

Given the tools that I already know, Java was the easy choice because it has MIDI capabilities built into the language (which seems excessive, but extremely helpful). On the front-end, JavaFx was the right fit for this project.

Building and Running the game

The instructions for downloading, building, and running the game can be found at the GitHub repo:  https://github.com/panurgy/handbellhero

The game works best on a Windows PC (using a 720p display or projector), and you'll need the Java 8 SDK (free download) installed (and in your PATH).

After a song completes, you'll need to press Alt-F4 to return to the menu screen (yeah, it's lame - I had it coded up to auto-close the window when the MIDI player/sequencer finished, but occasionally would receive an erroneous/premature "player stopped" event, that would close the game screen)

Limitations

The most technically challenging part of this game is getting the "falling notes" to sync up with the music as it's played. This involves a bit of "lead-in" time ahead of the player, which means the UI has to understand MIDI tempo information.  For some reason, I couldn't get the math to work out, given the Sequence's Division Type, the Sequencer Tempo in MPQ, and the Sequencer Tempo in BPM.

I found numerous posts and Stack Overflow questions related to this topic, but the math never seemed to work out correctly. 

In the end, I just used a lame/dirty hack involving a properties file which specifies the number of "ticks per second" used by each of the MIDI files. Tip - the app defaults to 1700 ticks per second, so if your file happens to be close to that, there's no adjustment needed.

An even bigger limitation

This game currently ignores the MIDI Set Tempo events that occur within the MIDI Sequence.  There is code present to find/read those events, but since the math/calculations never worked out, I just stuck to fixed-tempo songs.

Adding your own MIDI files

If you'd like to use your own MIDI files (using just the key of C Major), you'll need to save them into the "midi" directory.  If you find that the UI starts at the wrong tempo (which is very likely), run the app using the command hh.bat log

Select your song, and let it play for a few seconds, then close the game (press Alt-F4 a few times).  In your Command Prompt window, you'll see stuff like this:

PLAYER TICK = 2641
Ticks 4259.0 to 4341.0
Track entries 8 to 8


PLAYER TICK = 2730
Ticks 4341.0 to 4430.0
Track entries 8 to 8

You'll need to determine how many MIDI Ticks there are in one second. Subtract two of the non-zero "PLAYER TICK" values, such as 2730 - 2641 = 89.  Then multiply that times 20, because the game refreshes the UI 20 times a second.  Thus, this song runs at 1780 MIDI ticks per second.

Edit the file resources\stupidHackTempo.props and add an entry for your song. The song's filenames are case sensitive, and you'll need to use the Unicode sequence "\u0020" for any spaces in the song's filename.

Rerun the game, select your song, and hopefully the UI will line up with your music.

Huge thanks to these people!

This was a team effort, and I'd like to give a shout out to these people:

• Allen Fisher, for doing a superb job putting together the MIDI files.
• Marshall Baltzell, for doing an amazing job with the two image files, especially on the Handbell Hero logo - he was dead-on with the appearance.
• Matt Bresnan, for coming up with the initial game design, and giving me the opportunity to take some time to code up game.
• The creator of the Nightmare Hero font, which adds to the visual effect of the game.
• The audio effect on the selection menu, which adds to the audible effect of the game.
• Apache FilenameUtils - they have libraries that make coding easier (and less cumbersome).

Got feedback?

If you love it, hate it, or have any questions, let me know! Either leave a comment below, or share with the entire world via Twitter @Panurgy