Leap Piano


This is still in-progress, but getting there. My son and his friend came in and tried it – and they thought it was cool, so it must be somewhat. You can now play piano with the Leap – or, that is, a virtual keyboard with almost no control. I’m going to keep working on it, but it’s enough to know that while I’ll get some more accuracy out of it, the truth is it’s not going to get much more accurate using the methodology I’m working with – which isn’t to say it’s out of the realm of the leap to be that accurate – rather I have to really build some tighter JS to handle the math to do it right.

I was really happy to use Peter Coles’ Excellent JS Piano Script, though sadly, I couldn’t get it to work on Safari, and I couldn’t get any Leap stuff to work on Firefox, so I was stuck on Chrome (no disrespect – I just HATE building things that are just single-browser). Having a piano interface (I just snuck in simulating key-strokes) was easy – but using it is a bit of a challenge.

To put the math challenge in perspective, it’s important to understand how the Leap reads vector information. In essence, the vector model for the leap is just a standard XYZ axis-based set of numbers. In it, XYZ go positive towards you and to the right – that is, imagine if zero were the bottom middle of your screen, and the objects to the right, above, and towards you were +. The go negative away from you (Z), to your left (X) and towards only zero below. This is important to get – the Y is baseline, rather than being in the middle of the screen.

This part is actually pretty easy for tracking motion – you just have to compensate for negatives, since most screen and DOM elements will take a 0,0 top-left-corner approach. The folks at leap seem to be 3D model-obsessed, so they think just like a SketchUp team would – using that XYZ. This is fine until you get to rotations, in which case you find yourself dealing with some slightly trickier math – math which is in the API, but not in the JS quite yet.

To think about a piano being played, you have to realize that you’re not just putting your fingers down in space, you’re curling them downwards. And a piano play often hovers – their fingers outstretched – when they’re not playing a particular note. Not a problem except that when they play the black keys – the sharps/flats – they also stretch their fingers a bit. This meant knowing the vector pointer position – that is, (in Leap Gamer JQuery plugin notation)

x = leapData.pointables[i]["direction"][0];
y = leapData.pointables[i]["direction"][1];
z = leapData.pointables[i]["direction"][2];

(where i is the i in a for loop dealing with the pointables)

In order to know where the finger is pointing, you need to know the angle created by the Y and Z coordinates. This is… exactly the kind of math I haven’t done since high school. So, I had to look it up. I’m working on it, and I’ll have more soon – but I wanted to give a quick status update and say, hey, try my air piano!

2 Responses to “Leap Piano”

  1. Jared says:

    Hi. Would love to try it but getting a 404 at the air piano link.

  2. Jared says:

    OH NO! Fail! It was at just jaredwilliam.com – the blog added the blog.! Thanks for the heads up – fixing now!

Leave a Reply