My company has hosted hackathon competitions for the last few years. Back in early 2007, I immediately saw this type of mixed-space technology as being a game-changer. Not necessarily using a wii-mote to control a PC or draw objects in space on a wall, instead, the concept of multiple touch points inside a virtual or physical space, and controlling that space with something more natural than a mouse or keyboard.
I told myself that this is what I want to learn more about and this is a space that will change the world.
My doppelganger thought the same thing in 1991 while a graduate student in the University of Maryland's Human-Computer Interaction Lab.
Most people use 1-10 fingers and a mouse to interact with a computer and can type anywhere from 1-75 words per minute. The wii-mote added a "z-axis", depth, a gyroscope, motion detection, and additional buttons to what has mainly been a flat interface experience. There were even rumors of a microphone built into the wiimote for voice calls and speech recognition.
Nintendo really dropped the ball on that one. They could have changed the home market quite simply by offering sip phone numbers on a wii and the ability for the wii to wake-up on a call. Mario Google Hangouts? Zelda Snapchat? Pokemon Phone?
Speaking is generally much faster than typing and conveys many different nonverbal signals. The MicroMachines guy could sing Michael Jackson's Bad in 20 seconds. However, it still feels a bit odd and clumsy to me saying "Xbox, play Lego Batman" or speaking to a device while by yourself or in a crowded room.
While doing analytics for a cable company, I remember a story one of the engineers told me about how to get better service with a voice recognition IVR calling system. If the system detected a swear word it would immediately go to a CSR in customer retention. I'm not sure if that was true or not, however it conveys the idea that swearing is a strong signal of positive or negative feelings. Google even detects this, and filters the 7 dirty words. I get tired of listening to reality TV shows, prime-time cable television is now nothing more than a bunch of beep censors.
In 2008 I was offered the Microsoft Kinect among some other Christmas present options from my company. I chose the Kinect though though I didn't even own an XBox, and set to work hacking it on Windows 7. I built some cool apps with C#, Python, OpenCV and OpenNI, speech recognition and physics libraries. The body-tracking features, skeleton detection, depth camera, voice recognition and immersive surround microphones of the Microsoft Kinect make it stand out as one of the most advanced consumer tech products of its time.
The only quirk at the time was you had to hold your hands up in the air like you were under arrest for Kinect to recognize you.
How many thoughts per second can you capture as output to a device?
I was one of 3 people who purchased the now defunct OCZ Neural Impulse Actuator and attempted to get it to translate my thoughts to the screen. The developers were kind enough to send me a couple additional headbands, which still didn't fix the grounding issue I was having.
The OCZ had sensors which detected facial movements, eye tracking and even blinking. You blink about 15 times a minute, and the act of blinking can be
transmitted faster than a keystroke. The latency of a blink is 300-400
milliseconds. A blink is faster than a mouse click. It was a bit tiring playing Unreal Tournament with the OCZ and using your eyes as the fire button.
One of the reasons I think the OCZ NIA failed as a product, other than because of OCZ's financial shenanigans and grounding issues was the fact that many people feel very uncomfortable about something that can "read their thoughts". The creepyness factor is high with these kind of devices.
In 2009 I bought a home projector and screen, which saw a lot of movie time and also some fun hacks with flight sims, the wii-mote, my new Xbox and the Kinect.
In 2010, I won a first-gen iPad and hacked it to work with Synergy, using my laptop to control my desktop PC, iPad and Android phone. Being able to seamlessly control multiple devices, and move between them with the same interfaces is a really amazing productivity tool.
Fastforward to 2016. VR is everywhere. Virtual Reality, Augmented Reality, Simulated Reality, and various flavours of the tech have really come into their own since then. Over the last few years, Microsoft Kinect, Unity, Surface, and now Hololens and Cortana Analytics have changed the way people build and interact with apps.
Reality that hackable culture is now a commodity was when I walked into Bestbuy just before Christmas this year. In the toy section they had the Makey Makey I purchased a few years ago as a Kickstarter. There was a Brain-Sensing Headband for sale for mindful meditation. A lady saw me looking at smart watches and asked if they had one for her boyfriend that he use to could call people with. The Samsung Gear 360 was on sale, to capture the entire virtual reality experience of looking around and moving through a space.
Christmas Day, 2016. I bought a cheap $10 Google Cardboard clone as a stocking stuffer and everyone I showed it to was blown away. It was funny because my 4 year-old keeps trying to touch the objects in the play space with his hands like with the Kinect, however cardboard is really just paper with a button magnet, a couple cheap lenses and some very complex software.
In 2017, I hope to see the technology evolve to the point where it will be possible to have a seamless transition from PC to Mobile to Virtual Reality / Augmented Reality device.
Why can't you point at something on your screen then drag it out to the real world, or point to something in the real world and immediately have a complete copy, including sound and at-scale dimensions, on a screen. Magic trick or not, it is happening. Synergy of the real world.
My kids are addicted to the game Skylanders. As a Portal Master, they can place their plastic figures with encrypted NFC tags on a USB Portal connected to the Xbox, and watch them appear on the screen. If a Skylander villain is trapped, the tinny-sounding speaker on the Portal kicks in to make it sound as though the character is actually being sucked out of your television into a piece of plastic in your living room.
Games like Skylanders, Lego Dimensions, and Disney Infinity are good examples of merchandising that turns a single-dimensional play experience into a tangible reality.
This year the grandparents will be getting plastic clones of their grandchildren, and everyone will become a Skylander.
http://3dvrcentral.com/2016/12/19/sky-kids-coming-to-your-home-1000-winners-augmented-reality-3d-fun-turning-into-toys-pt-4/
This blog will track my links, ideas, and innovation within AR, VR, Unity, Machine Learning and Natural User Interfaces, starting from the genius and simplicity of Cardboard and The Sword of Damocles.
Some topics to get started on.
Microsoft Hololens. HTC Vive. Steam VR. Leap Motion. OSVR and Hacker Dev Kit HDK2. PlayCanvas. Sketchfab. Visor.io.
Google Daydream.
