Episode 5: Kissyface

Teenager 1:	00:01	Wow.

Teenager 2:	00:01	Thank you. It's [inaudible 00:00:07].

Teenager 1:	00:06	Oh my gosh.

Speaker 3:	00:06	You're actually way better at this than most people that try to play it.

Teenager 1:	00:10	Really? Because it's like raising my eyebrows.

Mick Ebeling:	00:11	There's nothing quite like the sound of teenage girls playing a silly game on their phones. In this case, it's a demo version of an app called Faceball. It's a game of pinball that lets you control the flippers by moving your eyebrows. Now this is supposed to be a podcast about technology for the sake of humanity. So how exactly is humanity served by playing pinball by wiggling your eyebrows? It's a good question. The answer is going to take us to a very important place, a place where we explore the very nature of how sometimes getting to do something useful has to start with doing something ridiculous.

Mick Ebeling:	00:52	I'm Mick Ebeling and this is Not Impossible.

Mick Ebeling:	01:03	This podcast is brought to you by Avnet, a company dedicated to helping creators of all types find whatever they need to get from idea to product and then get their product to market anywhere in the world. Today we're going to bring you the story of ideas that seem absolutely useless and that just might be changing the way we help people with spinal cord injuries lead a more happy and productive life. And if that sounds impossible, just give us a few minutes because our story starts with a guy who is actually dedicated to creating things with absolutely no redeeming social value. His name is Elliott Spellman, and he didn't seek out the world of technology. He slid into it.

Elliott S.:	01:49	I read science fiction for as long as I think I've been able to read. As my undergrad major, I studied creative writing and English, English literature. And when I was getting ready to graduate, I was also getting into making more visual art. Then I moved to the Bay Area and it's technology everywhere. I wasn't an intentionally resisting that at all. But I think the interest in producing things creatively and coupled with this environment of technology just meant that that's an intersection that I got into a little bit. I made a bunch of projects on my own.

Mick Ebeling:	02:29	And those projects were as intentionally useless as they could get.

Elliott S.:	02:33	The company that I started, if you could call it a company in 2014 is called the RoSham Business. RoSham Business came from the name of our first product, which is the RoSham Bizzers. Those are a pair of scissors that are completely made out of rock and paper, so they are very literal rock-paper scissors and they definitely don't cut anything. They serve no benefit to society whatsoever, but they're really, really fun to make and funny to think about. That was the original idea and we created a series of products that are all these literal versions of figurative things.

Mick Ebeling:	03:13	So for example, they came up with the grain of salt, literally a grain of salt and a little package for when you need to take something with a grain of salt. And you know the old philosophy, 101 question, if a tree falls in the woods and no one's there to hear it, does it really make a sound? So they rigged up a machine to pull a dead tree down in the middle of a forest when there was nobody there and they recorded it. And then they sold the recording of a tree falling in the forest and no one hears it. Like I said, it's totally useless.

Mick Ebeling:	03:51	And he brought his love of the absurd to Stanford where as a part of his thesis project, he and a bunch of other like-minded designers came up with a program for measuring people's conception of themselves as they looked in a mirror. Almost as an afterthought, one of the ways that people controlled the camera angle and the lighting and whatnot was by making different facial expressions. And lo and behold, Elliott's facial expression interface was born. And of course Elliott decided to use it for, well, the goofiest things he could think of.

Elliott S.:	04:26	The first thing that we made that you could use by controlling your face was a game. We tried to start with the simplest games that we possibly could. A famous simple one is this thing called Flappy Bird, also known as the helicopter game, where you're flying forward and you have to hop over obstacles. We just programmed the hop to happen when you raised your eyebrows. And it honestly only took a couple of hours to rig that altogether, just using a computer's webcam. And I guess it just ran from there in increasing versions of complexity.

Mick Ebeling:	05:03	Let's just take a second to ask. Why eyebrows? Simple. Think of all the people in the world who could communicate just with their eyebrows. Imagine Marilyn Monroe whose eyebrows could shoot up to show surprise or shock or delight or just pure sexuality, or the immortal Mr. Spock who could raise one eyebrow and look at Captain Kirk and say, "Fascinating," or Groucho Marx, who could tell a whole joke just by wiggling his eyebrows. They're just one of the most expressive parts of your face, the parts with the most movement. So that's what Elliott used for his games.

Elliott S.:	05:37	The first real game that we tried to make from start to finish was this thing called the Eyebrow Pinball Faceball. Eyebrow Pinball was just funny. It made sense. The mapping is pretty direct between controlling your eyebrows and flipping panels on a pinball board. All those things just made it obvious that that was something we wanted to build. Simple game, simple control.

Mick Ebeling:	06:03	Simple and silly. When you put it on your phone and moved your eyebrows up and down to control the flippers, which by the way also look like eyebrows, the game took pictures of you looking pretty ridiculous. The game hasn't been released yet. Hopefully you'll get a chance to try it out soon. But really that's kind of where this whole story could have ended.

Elliott S.:	06:26	This game was our thesis project in the graduate design program at Stanford. The program has since changed the same to design impact. Because the programs directors would like to encourage more and more a helpful application of the things that people are developing in order to have a positive impact on society. It was a funny position to be in, to make something that was not particularly helpful given the emphasis on helpfulness. I think it was fun to rebel a little bit against this forced helpfulness, which I don't think is a bad thing at all, but it is a little creatively constraining. I guess I'm somebody who believes that helpful and beneficial things don't always come from direct intentionality, that there can be long and securities paths to being helpful.

Elliott S.:	07:21	There's this great professor at Stanford named Dave Jaffe who specifically does assistive technology. And I remember him watching us in our final presentation, coming up to us afterwards, and telling us that there is a strong potential application for this.

Mick Ebeling:	07:35	A strong potential application for this, but what on earth could the application be for a program that just encourages you to make funny faces? You'd have to be someone who thinks pretty far outside of the box to come up with it. Fortunately, Elliott encountered just such a person.

David Putrino:	07:53	I think at first, to be really honest, I just initially hear all of this stuff about playing video games with facial expressions. Then I'm like, "What am I going to do with this?"

Mick Ebeling:	08:08	Regular listeners to this podcast, and thank you very much for that by the way, will recognize that Aussie accent. David Putrino is the head of the Rehabilitation Innovation department at Mount Sinai Hospital in New York and he's also part of the Not Impossible mobile floating brain trust. He's worked with us on a number of projects. And when I sat down with him I was not surprised to hear he'd found a way to do something with this technology, but I was definitely curious as to how this idea came to him.

Mick Ebeling:	08:39	I'm here with Dr. David Putrino who is also known in the Not Impossible world as Not Impossible's chief mad scientist, which I think is a perfect representation of you and how you think and how you approach things, is that you've got initials behind your names and you work for big, slow moving institutions, notable, incredible institutions, but you operate more like a mad crazy entrepreneur, just inventor. I think that's why Not Impossible and David Putrino have gotten along so well over the years.

David Putrino:	09:18	Yeah, no, I think that's fair. I mean, I think it's a sign of the times when the behaviorist mad scientist, which is probably more sane than the general scientific process, which is, "We're just going to keep doing this very slow moving science step by step, even though we know we could bypass all of these steps very, very quickly because this is the way we keep funding established and keep people asking more, but we're not translating fast enough." There is a lot of very traditional research that gets done in my lab, but then there is stuff that we look at and similar to Not Impossible, we see a problem that can be solved and it doesn't matter if there's no funding to solve it. We have the skills as scientists to solve it, so we solve it.

Mick Ebeling:	10:09	All right. Tell me about this guy that you meet. Tell me the process of meeting this young... How old is Elliott?

David Putrino:	10:18	Oh God, I don't even know. I think he's 20, 23, 24. Young enough to put us both to shame at all the things that he's achieving.

Mick Ebeling:	10:26	Speak for yourself.

David Putrino:	10:27	Okay. I'm sorry, I forgot your youth and beauty. I got on a phone call with this guy and a really nice guy and he was chatting about all of this absolutely brilliant stuff that he was doing. He started showing me some demos of his games, just games that all you needed was a phone and your face and a facial expression and you could power these games. I said, "Well, I can see an application here for doing this for people who are locked in, who maybe can't move any other part of their body except for their head and their face but can't afford the typical communication systems that we have for them. And I love this solution because as with all the solutions we try and get going in the lab, all you need is a phone. It doesn't really matter which sort of phone it would be. And then you've got a communication device."

Mick Ebeling:	11:33	Yeah. The thing you have to know about David is that David could see a crumpled piece of paper on the side of the street and he will think about how that could help someone with spinal cord injury or someone with stroke. He's constantly looking for ways to hack or modify or make something so that it affects somebody who has these type of physical limitations and how to empower them so that they don't have those limitations any longer.

David Putrino:	11:57	Yeah. It was immediately obvious to me that this was something that we could use to really change the way that people communicate. It was so simple. It was elegant. It worked with any phone. It was just everything that I look for in an assistive tech.

Mick Ebeling:	12:20	Okay, now let's get technical for a second. Here's what's really behind the Eyebrow Pinball, or should I say, behind facial control technology. Every smartphone has a camera. If you put it in the mode where you're taking a selfie, which let's face it we all do that, there's an algorithm that breaks your face down into what's called a vertex map. It puts little dots all over your face and assigns values to them. These bunches of dots are your eyebrows. This bunch of dots are your eyes, your nose, your lips, your cute little cleft in your chin.

Mick Ebeling:	12:51	Then it notices when those bunches of vertices start to move when you wiggle your eyebrows or pucker your lips. So then it's a simple transition from taking those movements and translating them into something else, say moving the flippers on a pinball machine or something more.

Mick Ebeling:	13:08	You saw this and it doesn't sound like the transition from what he had created to what you saw the purpose or the potential application being, it doesn't seem like there was much transition or translation there.

David Putrino:	13:23	No, not at all. I mean, he had built a control system. We just needed to change the application. This is one of the things I'm really passionate about is the minute you just hop into someone else's world for a second, you always find these applications. I mean this is pure innovation. This is where you say, "Well, I'm out of solutions in my field. Let me just hop over and look at the gaming world." What's hot in the gaming world right now? Oh hey, Snapchat and filters and facially controlled things on your phone. We can use that.

Mick Ebeling:	14:05	And so a little coding, a little experimenting and a little fooling around later, and voila. The first of their phone apps.

David Putrino:	14:14	Before too long we had a basic prototype. You sort of held the phone out in front of yo. You moved your head around. You puckered your lips. Shaking your head no was moving the cursor left to right. Nodding your head yes was up and down, and puckering your lips was click. That was all there was to it.

Mick Ebeling:	14:34	Now it was time to bring it out on the ward.

David Putrino:	14:37	We sat in a meeting with two guys with spinal cord injury who were paralyzed from the neck down. One of them was like a real avid gamer and the other one was really interested in just being able to be independent on Facebook and social media and stuff like that. And they both said to us, "Well, this is cool and all, but really what I want is to be able to control my computer." Elliott and I both looked at each other and went, "That makes sense. Why don't we turn this thing into a mouse?" We had a chat about it. Elliott said, "Yep, I've got this."

Mick Ebeling:	15:20	Okay. Now the task was to take the technology out of the phone and turn it into a full-fledged mouse that would work on your computer. But before we get to that, we have to get technical again because David just alluded to a very important component to facial control technology. You can move things around a lot with your eyebrows, but for clicking the mouse you have to use an expression that as Elliott explained to us, goes by the very complex technical name, Kissy Face.

Elliott S.:	15:49	Dealing with mice and keyboards so often, it's hard to keep this in mind, but a mouse is actually two separate control systems. Your position on a mouse pad controls where the cursor is in space and then your clicks whether it's left, right, or middle control, they're an indicator to the computer that you want to perform a certain action once your cursor's in a space.

Elliott S.:	16:14	Mapping those relatively analog controls to a face is a really fun task, but there's no obvious way to do it when it comes to controlling the position of a cursor on a face and then indicating intent once the cursor has reached the position that the user wants. We've experimented a lot with different clicking mechanisms and what gesture you can make with your face, landing on this Kissy Face pointer, which actually a lot of guys with massive spinal cord injury, they use that as an indicator in day to day practice with their healthcare professionals, as a way to point across the room. It's the most pointer-like thing that a face has on it.

Mick Ebeling:	17:00	And once they had hit on those major emotions, moving the head, raising the eyebrows, making the Kissy Face, Elliott was able to do exactly what the spinal injury patients had asked him to do.

David Putrino:	17:10	He went back to San Francisco and started programming away. Before you knew it, he said, "Okay, now just plug your phone into any computer and now it's a mouse," and that was it.

Mick Ebeling:	17:23	That was it. Now suddenly there was a way to help people who are paralyzed from the neck down. Now they can do things they couldn't dream of doing before. You know when Elliott talks about the absurd? He means like creating a literal rock, paper, scissors. But at Not Impossible, when we talk about the absurd, we mean how absurd is it that just because these people are paralyzed, that they can't use a computer. With all our technology in the world, we can't overcome that? To me that's absurd.

Mick Ebeling:	17:57	But along come David and Elliott and they say, "Let's take this silly little device that's totally useless and find a way for it to be as useful as possible to the people who need it most." I mean, how beautiful is that? And what does it tell us about where ideas come from and the limits that we put on ourselves when we set parameters for a project and what might happen when we throw those parameters out the window and say to ourselves and all the people around us, "Just go for it. Go nuts. Go where the wind will take you and whenever you find there, no matter how useless it seems, bring it back to the lab and we'll see if we can put it to use somehow." What might happen if we made that our basic operational principle? We'll talk more about that question and see what happened when David and Elliott tried their device out on patients at Mount Sinai after the break.

Mick Ebeling:	18:50	You, yes, you are invited to the 2019 Not Impossible awards. Join us on June 1st in downtown Los Angeles to celebrate the inspiring work of people and companies who share in Not Impossible's mission of creating innovative technology to improve the wellbeing of others. For tickets and information go to notimpossible.com/awards.

Mick Ebeling:	19:16	This is a podcast dedicated to technology for the sake of humanity. But it's also about the people behind that technology. If that's you, Avnet can help no matter what you build. Why? Because they are the first company ever to offer true end-to-end solutions for product development in-house. That way creators in any corner of the world can take an idea from prototype all the way through mass production. Are you a startup and established OEM? Do you need help designing your product or organizing your workflow or getting stuff to market? Avnet's got your back.

Mick Ebeling:	19:51	Your world is one that's always changing and that's why Avnet is here to help you reach further. We want to give a special shout-out to Avnet's engineering communities, Hackster and element14 who help creators vet and invent the technology of tomorrow. Want to do more than just listen to this podcast, maybe something like taking on the next Not Impossible challenge? Go to podcastnotimpossible.com to find info for you and for anyone who wants to take their project one step further.

Mick Ebeling:	20:30	So picking back up with our story, this young gun named Elliott comes up with a way to play pinball with your eyebrows and one of the Not Impossible mad scientists who also happens to run a rehab lab at Mount Sinai, figures out how to use it to help patients with severe spinal cord injuries do all sorts of things; surf the web, send emails, anything. They named the system Expressions. David calls it, giving them back the basic human rights of communication. Elliott, being Elliott of course, puts it a little differently.

Elliott S.:	21:01	Computers create a lot of joy. Something like surfing the internet and looking at cat videos, everyone should have that ability. I hope that our technology has and will continue to give people that access. If people could use this controller to surf the internet and watch videos about cats looking at cucumbers, I mean I would consider a job well done.

Mick Ebeling:	21:24	But of course in reality, David and Elliott set their sights a little higher than cats looking at cucumbers. and yet when they decided it was time to try it out on patients, nobody expected what happened next.

Sergio Acevedo:	21:39	I have a spinal cord injury on my C3, C4 spinal cord. That happened in 2006. I was in a bicycle accident. I fell off my bicycle, and I hit my head with a tree. Yeah, broke my neck. Oh, I was paralyzed from my shoulders down.

Mick Ebeling:	21:55	This is Sergio Acevedo. One glance and you know he's a New Yorker. He's just got that look, that New York attitude, right? Sergio is in a wheelchair, and he will be for the rest of his life. He was 31 when he had his accident and from the start, that New York attitude, that, "I can handle this" attitude served him very well. To get back and forth to Mount Sinai he has to take three buses, but he never misses an appointment and he never misses a chance to try something new.

Sergio Acevedo:	22:23	I picked up painting when I was in the nursing home using the mouth stick. The wreck therapist came by and she saw I was able to use the mouth stick with a computer and I was able to write my name and write and sign, write with the mount stick and she decided to put a paintbrush to it. That's how I got started. When I did it, it took up all day and I loved it because on a rainy day there's nothing to do in the nursing home.

Mick Ebeling:	22:55	For seven years, Sergio kept it up, and he got better and better and better. I've seen the paintings he did by holding a paint brush in his mouth, and I got to tell you, they're amazing. All Sergio wanted to do was paint and get out of that damn nursing home. A few years ago he got the second wish. Medicaid came up with a program to pay for a small apartment that he could use for a studio and some nurses to help him out.

Mick Ebeling:	23:22	So you see where this is going, right? Here's Elliott looking for patients to use his new facial recognition software. And here's Sergio, who's paralyzed and loves to paint. Fate brings them together at this hospital, and it was love, love of technology, at first sight.

Elliott S.:	23:39	Sergio is the first patient I think that I had a full sit down conversation with and got his whole story. Sergio has an amazing outlook on life. He's also a painter that uses a mouth stick to make these amazing paintings and illustrations. One of the things that I had been working on is maybe an app you could use to draw using your face, as if your face is controlling a big pencil that's drawing on a screen. We whipped that up and showed it to Sergio and is really, really fun to play around with.

Mick Ebeling:	24:17	And the more they played with it, the better Sergio got.

Elliott S.:	24:22	You should be able to start it by raising your eyebrows. The click is still your pursed lips.

Mick Ebeling:	24:35	Those little pops are actually Sergio clicking the mouse by making the Kissy Face.

Elliott S.:	24:43	And you can change the color too if you want.

David Putrino:	24:44	Where are the colors?

Elliott S.:	24:47	It's this little red box here. It's small.

David Putrino:	24:53	So describe what I'm doing there, Elliott.

Elliott S.:	24:55	Yeah. Sergio, he's driving the cursor around using his facial position.

David Putrino:	25:05	How do I move it? Oh.

Elliott S.:	25:06	It's shaky. He can click and hold down and do paint strokes on the computer that way.

David Putrino:	25:12	How do I change the brushes?

Elliott S.:	25:15	Let's see. I think this one might be spray paint.

David Putrino:	25:19	Right there.

Elliott S.:	25:21	Maybe two more over. Yeah, that there. Yeah. Just give that a whirl. I can undo it if you don't like it.

Mick Ebeling:	25:34	Okay, so don't go now, but up on our website there's a video of what Sergio was doing. I know this is something you should never say on a podcast, but you have to see this to believe it. Before your very eyes, Sergio just moving his head around and making a Kissy Face and doing stuff like that, draws on the computer a perfect, beautiful butterfly. Sergio says, giving him this gift of being able to paint in this way, it's, well, no offense Elliott, but it's a hell of a lot better than watching a video of a cat with a cucumber.

Sergio Acevedo:	26:10	Once you get into the mood, you lose everything. You lose the fact that you're injured. You lose the fact that you lost something. When you stop painting, all comes back to you. Then it's like a refocus on how to deal with the situation, what can make it better. It's very therapeutic.

Mick Ebeling:	26:36	And of course, when he's finished the painting, he was able to sign it.

Sergio Acevedo:	26:42	Ooh. Sergio Acevedo.

Mick Ebeling:	26:44	You did. That is crazy. That is so good. He signed it.

Sergio Acevedo:	26:48	See? Sergio Acevedo.

Elliott S.:	26:49	That was awesome.

Sergio Acevedo:	26:53	That's how I sign all my paintings.

Mick Ebeling:	26:57	That's what Sergio got out of the facial recognition software and that was pretty cool. The coolest part of the story is what I'm about to tell you next. Because very soon a whole lot of people are going to have access to this for free.

David Putrino:	27:16	We're not trying to patent anything. In fact, we are planning to release this app open source and free on the app store. We don't want people with spinal cord injury to pay for it. Because the reality is, there's lots of tools out there that give people with spinal cord injury the same abilities that our app would give them., but the difference here is that these companies are charging them anywhere from 400 to 1,000 or sometimes $10,000. This doesn't always sound like a lot of money, but when you are a 20-something year old kid on Medicaid and you've got all of these other medical bills, $1,000 may as well be a million dollars for how often you're going to come across that spare.

Mick Ebeling:	28:06	What's the model then? Do you have a long-term business model around this or is this just one of those gifts to humanity that you guys are going to do?

David Putrino:	28:13	We do want to try and create some sort of revenue stream that allows us to keep developing this and making it better and stronger and supportive of more facial expressions and all that sort of thing. The way that we think we're going to do it, we're going to bring Elliott's core competency back. We're going to make a few mainstream games for everybody to play. We're going to try and use funding, in-app purchases, all sorts of things from the mainstream games to fund development of the communication tool for people with spinal cord injury, maybe even take away the guilt of playing crappy little Candy Crush-y games for regular, neuro typical people by letting them know they're doing something good while they play.

Mick Ebeling:	29:07	Amazing. Releasing it open source, are you going to develop a community around that so developers can take it and build upon it?

David Putrino:	29:15	We hope so. Yeah. I mean, we're already speaking to a professor at Columbia who thinks he can get this to control one of his robotic arms because he makes robotic arms for people with spinal cord injury. We don't want to slow down that sort of development because again, it turns into, who can make the coolest tool with what we've started with.

Mick Ebeling:	29:40	Then you see that and you see the perspective and the potential of, "Wow, this could be an incredible device that could help people with all types of abilities or disabilities." What's the lesson that someone listening to this would take away? Right? You've already blown us away with what this is doing and what this has the potential to do in terms of disruption. What's the lesson for someone to take away?

David Putrino:	30:02	Yeah. I guess the big lesson is if you're interested in innovation, talk to everyone. What I always find interesting in my scientific career when I pop my scientific hat on is other scientists that I work with, they're always going to the same conferences in their field. I'm like, "Great, you're going to sit in a room with people who agree with you and talk about stuff." I love that. I absolutely love going to certain conferences and hearing about what's new and what's cool in a specific field, but I also make time to go to conferences where I absolutely don't belong. I have no right to be there.

David Putrino:	30:43	I go to visual effects conferences. I go to conferences about the microbiome, and I just chat to people and see what's cool in their field. What is the most leading edge technology you're using right now that you think everyone is going to be using in a year because it's so accessible all of a sudden? Then I think, how can I pull that back to health and what I'm doing. I think that that's the big lesson. If you're interested in innovation, that's the way to do it. Don't pigeon-hole yourself. Don't narrow your field. Step out once in a while and just be somewhere you don't necessarily belong.

Mick Ebeling:	31:24	There you have it. Let me ask you a question. Who is it in your world who is doing something cool that has absolutely no application to what you do? What's the most useless thing going on around you? What if you looked at that absolutely useless thing and said, "Hey, maybe I could use that." Just think of the possibilities.

Mick Ebeling:	31:50	Thanks for listening to Not Impossible. Thanks again to our sponsor, Avnet, a company that provides true end-to-end technology solutions to help people who make, well, anything, anywhere in the world reach further. I want to invite you to check out their communities. Hackster, the world's fastest growing developer community for learning, programming, and building hardware and element14, the biggest designer engineer community on the planet. You can find links to their websites on our website, podcastnotimpossible.com, along with a little video of Elliott playing Face Pinball and that amazing video of Sergio painting the butterfly.

Mick Ebeling:	32:29	Post production of today's podcast happened at Lerman Productions. Our associate producer was Vicki Schairer, our development director, Erin Sullivan, our director of partnerships, Joe Babarsky. Our executive producers are Phil Lerman and me. I'm Mick Ebeling. Until next time, remember, sometimes you just got to take a chance. Leap and the net will appear. I know. It sounds crazy. May be crazy, but it's not impossible.