Georgi Simeonov and the team at Unity published a series of blog posts on a demo called ADAM.
The film is set in a future where human society is transformed by harsh biological realities and civilization has shrunk to a few scattered, encapsulated communities clinging to the memory of greatness.
Adam, as our main character, was the starting point of our visual design process. He was designed to provide a glimpse into the complex backstory of the world, by revealing himself as a human prisoner whose consciousness has been trapped in a cheap mechanical body.
My computer can't even handle playing back the video without a bit of stutter. It's like a message will pop up any second telling me "this isn't something I can even fathom playing back for you in a timely fashion." I can imagine what a $600 graphics card could do with this technology.
The amount of thought and depth that went into this short demo is staggering. What I really find interesting is the concept art and reference sheets. Google Goggles and Machine Vision seems like a great tool for building these otherworldly characters. Take a picture of someone and it will classify, recognize color and text, identify brands and bar codes, and search for related images. We could tweak the Cloud Vision or Microsoft Cognitive Services Computer Vision API technology to generate these reference sheets automatically, and provide additional insight and ideas to the realtime art director.
What if a tiny device sitting in the middle of your living room could model the room with 360 depth video, add ceilings, floors, render light sources, color match, visually classify the contents and determine what would look just a little bit out of place yet matches the color, lighting scheme, and design aesthetics of the space.
The tough problem is no longer modeling a room in realtime - I can do this with a Kinect and Skanect in about 5 minutes.
Well, there's still a few kinks to work out...
The tough problem is making this technology portable, non-intrusive, and insensitive to light sources like the Sun. The ability to use laser technology to capture accurate depth and distance across hundreds of yards of indoor or outdoor terrain. A way of bringing the experience to the individual, rather than bringing the individual to the experience.
What if you could take an experience like watching a hockey game, render it realtime in VR, add binaural audio and throw in a few of your closest friends from around the world.
There's still some room to grow with the APIs though. I'm pretty sure there's plenty more Joy, Sorrow, Anger, Surprise and Headwear at this hockey game.
Partying like it was 1999, I was working with Microsoft Site Server, analyzing web logs in multiple dimensions for the largest telecom company in Canada. Standard dimensions for a web log would include host, referrer, uri, date, time, code, client ip, browser, and the mess that was cookies and session ids, One tool that sticks out for me was Microsoft Site Analyzer. It was the coolest tool I had seen (out of Microsoft, at least) for visually understanding path analysis and links within a web site.
The interactivity of the tool and bouncing, spiderweb physics really made the it fun to use. Though it didn't scale so well once we got up to 50,000 pages of content and 10GB of web logs a day... and I think I was the only one actually using it. Most of the e-marketers were just concerned with how many visits there were to their respective campaign pages, and less concerned with what page links to where or how people got to a page in the site.
Microsoft Data Analyzer was another tool for visualizing multiple dimensions that didn't really take off as well as the Proclarity's and Cognos BI tools of the world.
A game-changing demonstration of interactivity, multi-dimensional data visualization, and time-series animation was with the Gap Minder bubble chart demo and its origins from Ted Talks by Hans Rosling. Hans conveyed an exciting story of the world's population & life expectancy data like a horse race, showing the staggering growth of India and China across the centuries and the effects of war and famine on the world's countries.
Dollar Street, the latest tool from GapMinder, uses 30,000 photos, from 240 families in 46 countries (excluding Canada) and the stories of families, their annual income, and the dimensionality of their lives, including what toys they have and how they brush their teeth.
It is a great example of the power of data combined with multiple dimensions of visual imagery, personal stories, and heart-breaking reality. The story, context and imagery behind the data points is much more important than just the numbers themselves or plots on a graph.
For a more lighthearted example of a different way of visualizing 3-D data, here's Wolfenstein 1-D, Wolfenstein in a single pixel line. Not as good as Snake... or this Deep Reinforcement Model which beats Snake.
In the 90's, John Carmack of id, Wolfenstein and now Oculus Rift fame put together a side-scroller called Dangerous Dave, to clone Nintendo's Super Mario Bros for the PC. At the time it was amazing.
Now we are in 2017, and we have Nintendo releasing Super Mario Run, a mobile side scroller that does the side scrolling for you. And it's amazing? Nostalgia sells, just ask Pokemon Go players.
People tend to avoid eye contact when speaking, as their thoughts and verbal responses are more easily processed when not distracted by the visualization process. This may explain the overwhelming feeling you may get when viewing something the brain cannot easily process, such as a multi-dimensional Virtual Reality video that shouldn't exist in your current reality space.
It may also explain why it's still fun to ignore having to move a character in multiple dimensions and just tap a screen like a slot machine while hurling uncontrollably right and jumping on Koopa Troopas and Goombas.
And why the 8 bit workshop Atari 2600 repl is Amazing.
One "holy shit" arsword moment that stands out for me when seeing a technology hack in action over the last few years was the wii-mote being used as a multi-touch mouse for an augmented reality whiteboard. Johnny Chung Lee posted a video demonstrating the technology and it really grabbed my attention. Microsoft ended up hiring him in 2008 for their XBox Kinect and Applied Sciences team. He was there until 2011 when he joined Google and is now working on Project Tango and 3D vision.
My company has hosted hackathon competitions for the last few years. Back in early 2007, I immediately saw this type of mixed-space technology as being a game-changer. Not necessarily using a wii-mote to control a PC or draw objects in space on a wall, instead, the concept of multiple touch points inside a virtual or physical space, and controlling that space with something more natural than a mouse or keyboard.
I told myself that this is what I want to learn more about and this is a space that will change the world.
Most people use 1-10 fingers and a mouse to interact with a computer and can type anywhere from 1-75 words per minute. The wii-mote added a "z-axis", depth, a gyroscope, motion detection, and additional buttons to what has mainly been a flat interface experience. There were even rumors of a microphone built into the wiimote for voice calls and speech recognition.
Nintendo really dropped the ball on that one. They could have changed the home market quite simply by offering sip phone numbers on a wii and the ability for the wii to wake-up on a call. Mario Google Hangouts? Zelda Snapchat? Pokemon Phone?
Speaking is generally much faster than typing and conveys many different nonverbal signals. The MicroMachines guy could sing Michael Jackson's Bad in 20 seconds. However, it still feels a bit odd and clumsy to me saying "Xbox, play Lego Batman" or speaking to a device while by yourself or in a crowded room.
While doing analytics for a cable company, I remember a story one of the engineers told me about how to get better service with a voice recognition IVR calling system. If the system detected a swear word it would immediately go to a CSR in customer retention. I'm not sure if that was true or not, however it conveys the idea that swearing is a strong signal of positive or negative feelings. Google even detects this, and filters the 7 dirty words. I get tired of listening to reality TV shows, prime-time cable television is now nothing more than a bunch of beep censors.
In 2008 I was offered the Microsoft Kinect among some other Christmas present options from my company. I chose the Kinect though though I didn't even own an XBox, and set to work hacking it on Windows 7. I built some cool apps with C#, Python, OpenCV and OpenNI, speech recognition and physics libraries. The body-tracking features, skeleton detection, depth camera, voice recognition and immersive surround microphones of the Microsoft Kinect make it stand out as one of the most advanced consumer tech products of its time.
The only quirk at the time was you had to hold your hands up in the air like you were under arrest for Kinect to recognize you.
How many thoughts per second can you capture as output to a device?
I was one of 3 people who purchased the now defunct OCZ Neural Impulse Actuator and attempted to get it to translate my thoughts to the screen. The developers were kind enough to send me a couple additional headbands, which still didn't fix the grounding issue I was having.
The OCZ had sensors which detected facial movements, eye tracking and even blinking. You blink about 15 times a minute, and the act of blinking can be
transmitted faster than a keystroke. The latency of a blink is 300-400
milliseconds. A blink is faster than a mouse click. It was a bit tiring playing Unreal Tournament with the OCZ and using your eyes as the fire button.
One of the reasons I think the OCZ NIA failed as a product, other than because of OCZ's financial shenanigans and grounding issues was the fact that many people feel very uncomfortable about something that can "read their thoughts". The creepyness factor is high with these kind of devices.
In 2009 I bought a home projector and screen, which saw a lot of movie time and also some fun hacks with flight sims, the wii-mote, my new Xbox and the Kinect.
In 2010, I won a first-gen iPad and hacked it to work with Synergy, using my laptop to control my desktop PC, iPad and Android phone. Being able to seamlessly control multiple devices, and move between them with the same interfaces is a really amazing productivity tool.
Fastforward to 2016. VR is everywhere. Virtual Reality, Augmented Reality, Simulated Reality, and various flavours of the tech have really come into their own since then. Over the last few years, Microsoft Kinect, Unity, Surface, and now Hololens and Cortana Analytics have changed the way people build and interact with apps.
Reality that hackable culture is now a commodity was when I walked into Bestbuy just before Christmas this year. In the toy section they had the Makey Makey I purchased a few years ago as a Kickstarter. There was a Brain-Sensing Headband for sale for mindful meditation. A lady saw me looking at smart watches and asked if they had one for her boyfriend that he use to could call people with. The Samsung Gear 360 was on sale, to capture the entire virtual reality experience of looking around and moving through a space.
Christmas Day, 2016. I bought a cheap $10 Google Cardboard clone as a stocking stuffer and everyone I showed it to was blown away. It was funny because my 4 year-old keeps trying to touch the objects in the play space with his hands like with the Kinect, however cardboard is really just paper with a button magnet, a couple cheap lenses and some very complex software.
In 2017, I hope to see the technology evolve to the point where it will be possible to have a seamless transition from PC to Mobile to Virtual Reality / Augmented Reality device.
Why can't you point at something on your screen then drag it out to the real world, or point to something in the real world and immediately have a complete copy, including sound and at-scale dimensions, on a screen. Magic trick or not, it is happening. Synergy of the real world.
My kids are addicted to the game Skylanders. As a Portal Master, they can place their plastic figures with encrypted NFC tags on a USB Portal connected to the Xbox, and watch them appear on the screen. If a Skylander villain is trapped, the tinny-sounding speaker on the Portal kicks in to make it sound as though the character is actually being sucked out of your television into a piece of plastic in your living room.
Games like Skylanders, Lego Dimensions, and Disney Infinity are good examples of merchandising that turns a single-dimensional play experience into a tangible reality.
This blog will track my links, ideas, and innovation within AR, VR, Unity, Machine Learning and Natural User Interfaces, starting from the genius and simplicity of Cardboard and The Sword of Damocles.
