This is still in-progress, but getting there. My son and his friend came in and tried it – and they thought it was cool, so it must be somewhat. You can now play piano with the Leap – or, that is, a virtual keyboard with almost no control. I’m going to keep working on it, but it’s enough to know that while I’ll get some more accuracy out of it, the truth is it’s not going to get much more accurate using the methodology I’m working with – which isn’t to say it’s out of the realm of the leap to be that accurate – rather I have to really build some tighter JS to handle the math to do it right.

I was really happy to use Peter Coles’ Excellent JS Piano Script, though sadly, I couldn’t get it to work on Safari, and I couldn’t get any Leap stuff to work on Firefox, so I was stuck on Chrome (no disrespect – I just HATE building things that are just single-browser). Having a piano interface (I just snuck in simulating key-strokes) was easy – but using it is a bit of a challenge.

To put the math challenge in perspective, it’s important to understand how the Leap reads vector information. In essence, the vector model for the leap is just a standard XYZ axis-based set of numbers. In it, XYZ go positive towards you and to the right – that is, imagine if zero were the bottom middle of your screen, and the objects to the right, above, and towards you were +. The go negative away from you (Z), to your left (X) and towards only zero below. This is important to get – the Y is baseline, rather than being in the middle of the screen.

This part is actually pretty easy for tracking motion – you just have to compensate for negatives, since most screen and DOM elements will take a 0,0 top-left-corner approach. The folks at leap seem to be 3D model-obsessed, so they think just like a SketchUp team would – using that XYZ. This is fine until you get to rotations, in which case you find yourself dealing with some slightly trickier math – math which is in the API, but not in the JS quite yet.

To think about a piano being played, you have to realize that you’re not just putting your fingers down in space, you’re curling them downwards. And a piano play often hovers – their fingers outstretched – when they’re not playing a particular note. Not a problem except that when they play the black keys – the sharps/flats – they also stretch their fingers a bit. This meant knowing the vector pointer position – that is, (in Leap Gamer JQuery plugin notation)

x = leapData.pointables[i]["direction"][0];
y = leapData.pointables[i]["direction"][1];
z = leapData.pointables[i]["direction"][2];

(where i is the i in a for loop dealing with the pointables)

In order to know where the finger is pointing, you need to know the angle created by the Y and Z coordinates. This is… exactly the kind of math I haven’t done since high school. So, I had to look it up. I’m working on it, and I’ll have more soon – but I wanted to give a quick status update and say, hey, try my air piano!

It’s here! I’ve finished my first proof-of-concept.

Check it Out

So, to explain…

As you may know, I recently received my Leap Motion controller – and I’m really excited. As a web developer, it’s particularly exciting to see that the Javascript API allows me to use the Leap on a web application.

The game is not much of a game, but that wasn’t its intent. In this case, I used a very simple variable coming out of the Leap. To understand this, I want to break down – just a little – how the Leap sends data.

if ((typeof(WebSocket) == 'undefined') &&
    (typeof(MozWebSocket) != 'undefined')) {
  WebSocket = MozWebSocket;
}

First, we need to decide who supports the web socket. Here’s a good thread on Stack about who supports the socket. Now that we have a web socket layer, we need to initialize it.

  ws = new WebSocket("ws://localhost:6437/");

This is the most important line in the script, because basically, we have to connect to the local Leap. Note here that that means your Leap is plugged in and the software is running. When you do, much like other custom IO devices, it opens up a local port at 6347 and sends out a bunch of data in a “Frame.” A frame is just a frozen moment in time during which the Leap sends out a bunch of data, or a message. The next code that’s important is the message loop:

  ws.onmessage = function(event) {
    var obj = JSON.parse(event.data);
    // obj["hands"][0]["palmPosition"][0];
}

In my example, I’ve taken the X coordinate of the “palmPosition” variable of hand one. Note that if you want to see a dump of the whole data set, view the str in:

  ws.onmessage = function(event) {
    var obj = JSON.parse(event.data);
    var str = JSON.stringify(obj, undefined, 2);
    // str;
}

This will give you a printout of all the data, though I found that putting a little stopper in there so that you could actually read the data was even more helpful.

In any case, take a look, play, and drop me any questions or comments. I’d love to hear that other Leapers have been able to reach across the web and start using this great device!

Looking for Time Warner Cable on Yelp is NSFW – and for good reason. We hate Time Warner not just for what it is – terrible service by terrible people – but for why it is – the simple fact that there is no other option. It’s like having only one restaurant in town, no kitchen, no supermarket, and also the waiters spit on you when you sit down. Of course, that’s just in general. For the past few nights, my experience has been far worse.

The nightmare started as any nightmare with TimeWarner must – my service went down. This time, it was at 10:30 on a Monday night. Of course, there are a lot of items between me and my internet, so it’s worth noting each. Obviously, there’s the computers – but they’re easy to rule out since none of us had internet. Next is the router – an Apple Airport Extreme. I must admit, this is a major point of failure – not just because it’s not 100% reliable (maybe 95%), but also because the implementation of IPV6 is just confusing and often causes issues just because of setting obscurity. Finally, there’s the cable modem itself – an RCA DCM425 (which we are now paying for).

We are used to mediocrity, of course, with Time Warner, so when the internet went down, I assumed it was just momentary. Daughter and I went out for a late-night snack and I was earnestly surprised when everything wasn’t working around 11:30 after a system reset (I keep the whole mess, including the phone system, backup drives, modems and routers etc. on one power strip for this very reason). This was getting serious as I had work to do. So, I made the usual mistake of calling TWC.

One thing that Time Warner should have going for it is a very slick automated technical advisor who not only takes your information, but also has access to their connection database. So, when I called and gave the relevant information, I was able to be told that yes, Time Warner could see I was off-line. This was actually a very helpful idea. I further respect the fact that when you call TWC, you can basically get to a human in two steps – hit 0 and then, when the automated prompter tries to get your information again, you can hit 0 again and be put straight into the queue.

However, things are not that easy. After the automated system told me I was disconnected (thanks, I guess? Isn’t that like the doctor telling you you’re sick?), I was given the usual runaround (automatedly) about restarting and unplugging – all steps I’d already taken. To their credit, they did have the option after each suggestion of saying that you had already done that. Finally, they recommended I speak to their help dispatch.

The first warning sign that this might not end well was when I was told to wait while I waited for the US Help Center. Hearing that a local service is taking you to a National support site is never comforting. The very pleasant lady I spoke with who, yes, once again took my information, once again verified it, and once again started off with restarting and resetting, finally decided that my modem was kaput. This jibed, actually, with what I thought it might be – the poor thing was blinking like crazy and every so often it would click indignantly, as though expressing technological angst.

I could have, she said, set up an appointment in a few days (between the hours of noon and five) set up an appointment for a technician to come and replace the box and check my connection. This seemed ludicrous – I live in a new-construction high-rise in Manhattan – there were no raccoons eating my wires, and the idea (the idea that no internet user but apparently every internet provider shares) that it would be perfectly reasonable to wait three days for my internet to work was just frustrating. Instead, we agreed that I could go into my local Time Warner and switch out the box.

This I did the next day – and as usual, being downtown, it was pleasant. They have a great lobby and a friendly security guard who looked at me knowingly when the lady in front of me (who clearly imagines her cats in various human situations) starts ranting about her cable service. I, however, am happy to be empowered to solve my problem. The lady at the desk (station 10) isn’t quite as nice and seems to be a practitioner of that religion where when you look at someone when you’re speaking to them, you die immediately. But, she gives me the box and I’m out of there in ten minutes.

Taking the box home triumphantly (I used to live in the less bougie part of town, and THEIR TWC office is essentially a prison-intake mixed with the DMV plus the ever-present threat of a crime), thinking, nice, TWC. This was only supported when I plugged the boxed in, watched the little robot light show, and then, with nary a curse, my internet was back and, much as is the Hess Truck, bette than ever.

This was particularly gratifying since the next morning I had a Skype interview that was mission-critical, and even in the interview pre-mail, I had been told the importance of a high-speed connection. I was like, well, it’s a tech job, so… that’s a good thing. Of course, now that I had my internet back, everything would be lovely. All I had to do was spend some time updating my website that evening, and then Skype party in the morning.

So, around 10:00 that night, I’m working on my updated website which I’m starting to send to prospective clients when the internet stops again. This time, though, it doesn’t do a full-stop. For instance, if I load three pages in tabs, one might load, and the other two won’t at all. Or, I’ll hit reload on a dead page and get the title on top. Then nothing. Emails come in every so often, though I can’t reply to them. The internet is basically stopped.

I look at my new cable modem (which I had to rip out of my system since I had, in a fit of happy relief, zip-tied into my network box just hours before). It’s blinking crazy-town again. Everything is wrong! So I restart, fearing that maybe I’d done something wrong and burned this one out, too. Then I go into crazy mode. By 12:30, I’d done everything from changing out EVERY wire (coax, ethernet, power) on every device to putting an ice-pack on the router (by now it was 1:30, I was tired, and it did seem hot). This was, by the way after calling TWC twice to ask them to check on the box, send that little, “Hey, connect to me” message, etc.

Now it’s 2:30. I can’t go to sleep because I can’t wake up to no internet. I have updated my router 100 times because Apple has 100 different configurations for NAT and tunneling and other words that probably mean something to someone who doesn’t have a website to finish touching up and an interview. In the middle of the night, tired and frustrated, you start to enter into what I describe as voodoo thinking. This is where you think, well, maybe if the router was on its side. Maybe if I unplug the modem for more than 1 minute. Maybe if I start with the system unplugged and then go in a different order. But the thing is, these are robust machines that have seemingly worked for years – this thinking is insane.

Finally around 3:00, I am on a very difficult tech chat (this is on my phone, the only only glimmer of internet I have remaining). I’m asking him what I can do to to get the system back by morning. Is there a way I can get the box replaced by 9? That’s when he says, well, it won’t help, since you have an outage in your area.

Wait, what? Ah, when did it start?

Sir, you know you can check the status of outages before contacting us.

So, he directs me to the Time Warner Help site, and tells me to click on Outages.

Cool. No, sorry, I can’t find the link.

So he gives me the link to the help site.

I’m there – I can’t find the link to outages page…

“Just scroll to the bottom and see the link. Click on it.”

But I can’t find it?

Ten minutes of this idiocy. Then, with the compunction and wit of a 9-year-old, he says, as if we’d not just spent ten minutes telling me I was stupid for not seeing the link.

“Sir, our website no longer has such a link. But I can see that there is an outage in your area.”

“So, when did it start.”

“Oh, around 10:30.”

It’s 3:30AM, I have an interview the next morning that I will end up canceling, work that I will not do. I spend the rest of the day half-asleep and flu-ey. I think of the cat-lady ranting about TWC and realize – not that that will be me someday. But rather, that she’s totally right.

I think it’s time to talk about the way we communicate with our computers. To do that, we need to look at the ways we have in the past. Forgetting about punch-cards and switches, I think it’s fair to look at the technology that most people recognize as input devices. Then, I want to look at what’s next – how do we interact with a computer in ten years. I believe that this is something that’s starting to take shape right now – and I like to call it the Third Wave.

See what I did there? Wave? I know – brilliant – but I’ll move on. Before I get to the third wave, though, let me outline what I feel define the first two.

The First Wave

The first wave was the device we talked to that was attached to the device we wanted to talk to. The first wave encompasses what we once knew as wired devices, though now most of these are wireless. These include keyboards, mice, and other less obvious input devices like scanners, barcode readers, etc. These are external to the thing you’re working with (the screen, the CPU) and they take data from the outside world and bring it into the computer’s world, usually with a bit of translation.

The keyboard is sometimes considered indispensable (I’m sure there are more than a few people over at Blackberry who wish they’d thought that one out) for a reason that’s worth exploring. The thing we do, falsely, is treat written language as sacrosanct. There’s a great comic (which I can’t find right now) that shows a guy holding a scroll and looking at a pile of books saying Sure, those are great, but people will always use scrolls. The thing is, it might be scary to think of the written word as itself an input device, itself an abstraction, but what is more human – reading a story or hearing it? What if the written word were not as sacred – at least when communicating with your computer – as we thought? What if we saw it for what it really was – a secondary input that was required by technical limitations?

What if the computer’s endpoint is a human interaction?

If it is, then just as we don’t communicate with each other using a keyboard, so would we just talk, gesture, and even emote to our computer. If you can think of the computer that way, you can see all of these peripherals as secondary, dated, and eventually redundant. If you can’t, then the first wave may never end.

The Second Wave

If you don’t believe the first wave is doomed to extinction, you might not have been around technology for the last few years. Because the second wave is us talking directly on the device. This is Siri, and the touch-screen, and even the camera. Think of how substantial a difference there is between attaching a flatbed scanner and running it and just taking a picture of a document with your built-in camera. That’s the second wave.

The most important piece of the second-wave, so far, has been the touch-screen. Want to zoom an object - touch the object. Want to open a program? Tap the image for the program on the screen itself. This is momentous because we have taken away yet another layer that stands between us and the thing we’re talking to, the computer. Of course, it could be said that the most momentous aspect of the second wave is the voice-automated assistant – Siri, Vlingo, etc. While not yet pitch-perfect – and certainly limited by a need to process on major boxes rather than on local devices – this is a truly human interaction, speech. And even if you truly believe humans will always communicate through written language, you have to admit that when  you want to set your alarm, it’s easier to ask Siri than it is to open an app, choose the time, and turn the alarm on. In fact, Siri has done something that turns back the clock on technological interaction to the time when you could talk to your butler and just ask them literally to do whatever you want. (Remember that time before Obama destroyed the economy by being elected after it crashed?)

The second wave is here – just tap your phone or tablet and wait for that little beep…

The Third Wave

The third wave, however, doesn’t need a beep. I’m going to start with an interaction that I’ve just said is the crown jewel of the second wave – voice automation. But I’m going to take away the little beep. I’m going to take away the tap and maybe even the tablet. The third wave is when you don’t even need the CPU to talk to the CPU. This is the truest natural state of computing. There may be a screen to express data, just as there may be speakers. But why should you have to interact directly with these objects? We love our tablets and smartphones because they are always with us. But they aren’t always with us. We have to bring them with us. The things that are always with us are:

1. Our Hands, Arms, Bodies
2. Our Voices
3. Our Thoughts

Let’s just say it. Someday, we should be able to communicate with our technology using brain-wave-based functionality. It seems somewhat over-the-top, but let’s not forget, we already have cat ears that we can control with our mind. Until that time, we have to stick with our hands and our voices. The third wave of interaction will be when we can, with no device in our hands and no peripheral at our fingers, communicate with and make demands of our technology.

This leads us to the Leap Motion Controller. For those who are looking for a Minority-Report styled screen interaction – yes. We have that. Now. But not just that – I believe that the next big thing in computers is ubiquitous control without peripherals, and so a built-in Leap (or Kinect) sensor that can track your gestures throughout a room is essentially all you’d need to have a computer with no inputs. Sure, you might point out that the Leap is an input – but it’s a matter of where you are. Are you sitting at your desk? Sure. Are you walking to the desk? Sure. Are you lying in bed? Sure. The difference is that the computer isn’t with you. The tablet isn’t with you.

As a Leap developer, I eagerly await my own third-wave device. But be warned, be enlightened – you heard it here first. This is the future.

Yessss! I was accepted into the LEAP developer program. My Leap Motion controller will be here in a few weeks! The Leap controller is really exciting – it allows you to use 3D interactions with your Mac, PC, or Linux box. I’m hoping to develop some educational apps for it.

That is a lesson that Yahoo’s recent data breach can teach all of us – hopefully even Yahoo. Yahoo committed sins left and right – so trying to walk through them now would be like a confessional in Las Vegas, but the thing that stuck out in my mind was that they stored their user passwords in pain-text, and I want to spend just a second explaining what that means.

Plain-text is the same as saying text. In a storage/content sense, it often refers to non-rich text, as in, just words, no formatting. But in a database/password sense it means that the passwords are literally as you typed them. This is, in one sense, the default. Of course a password would be what you saved it as. But it is not even remotely best-practice. That’s because it means that if anyone ever get their paws on your data, they would now see your passwords clearly. That’s why any self-respecting data manager ensures that passwords are stored encrypted. That means that you enter your password, then they pass it through an encryption algorithm, and it is stored as some other, more complex and obtuse string.

How does that work? Well, there are two parts. The encryption algorithm is like a machine that mushes your original password around into this new string. What’s important to know is that that password, however, will always be mushed in the same way. That is, if you enter your password again, it will end up making it through this machine identically, every time. So, if you’re trying to log into the site and you enter your password, we compare it to the original by looking at how it looks after it’s been through the encryption algorithm.

Yahoo did not do this. Perhaps they just had some bad people at key places, but more likely with a company like that, they just had a vestigial background database and they hadn’t updated their security protocols. Whatever the reason, it didn’t end well for the 44 thousand users who entrusted their data to them. So, if I or some other data architect ever admonishes you, No, we don’t store user emails in plain text, so we couldn’t possibly retrieve a user’s email address, then you smile and say, Of course. And thank your lucky stars that you don’t use Yahoo mail, because really, who does?

The wall socket which runs
Copper down 16 floors
To a substation on 40th Street
Which channels the power,
From Canada,
Like a delta in the Hudson
Drips from the north making it to us.

History is like that,
Some of it even true,
Having flowed downstream
Swimming with
Rocks and Fish
Until it has a special taste,
An ironic freshness.

I want my son to feel,
Not just know,
The pluck of the harpsichord
Indefinite in dynamic, tinny,
Filling up the hall,
In the Medici court.
Just as Cristofori did.

I want him to be there,
Not just look at a picture,
With the men in the lab,
In the 1920s,
Working with vacuum tubes.
I fear the picture, thinking,
Everyone here is dead now.

From the piano to the tubes
To the Casio,
To the iPad he uses for sheet music,
The scope of history is both
Enormous and claustrophobic.
Our apartment stands in Native soil,
But was once a subterranean volcano.

Where does it end?

It makes me think of my son,
Supposed to be reading or
Playing or something else,
Standing suddenly in front of me
Arms open, ready for a hug.
Here, this is for you.

Someday I will be a picture
And the phrase will again be true.
Everyone in this picture
Is dead now.
As, I shudder at the thought,
Will he be.
And what will it matter?

That I was suddenly
If illogically,
Inspired to take him uptown to
the Met
Where they have an original
Cristofori piano,
Molded by his hands.

Will that memory mean,
Nothing
Or will it mean
A small drop of something,
Hundreds of years on.
What will he
What will I

Have created?
Everyone in this picture
Is dead.
But their ghosts,
Lying there with open arms,
Whisper, Here,
This is for you.

We might not remember
Or it might not matter
That the creates are vicious
Bloodthirsty predators
Their black eyes and big noses
And white fur
Make us want to love them.

So seeing them floating
Infinitely abandoned
As their world melts around them
As our world melts around them
Is horrifying
And it reminds us
That we are the predators.

But there was
A last trilobite.
And there was
A last dodo.
And there will be
Most certainly
A last human.

When I am clinging
To the last piece of ice,
I think about the polar bear and the dodo,
And you.
And I am thankful
I have the capacity
To know I am not alone.

As John Cooper said on his FB posting, “This is one of the coolest “flash dances” ever! It’s 2012 and young people in Moscow have put on a “flash mob” happening, dancing to an 83-year-old American song written by a Russian-born American
Jew (Irving Berlin).” I’d add that all those people holding Korean tablets and wearing Australian boots and using Japanese cameras tell an even bigger story. We live in a small world. A small fucking awesome world.