One small quality-of-life improvement that could be added: have the software timestamp when you say "here" and then (using a short low-FPS video buffer?) analyze the corresponding frame of video. This would eliminate the pause when you're pointing at something, waiting for the computer to respond.
Alternately a low-latency speech analyzer could look for the word "here" and quickly snap a screenshot. Perhaps project a temporary dot on the surface, so the user knows they can put their hand down now?
Quite the impressive demo! There's huge potential here.
Holographic computing, I think, will be the future. At least for interfaces.
One big difference in direction is that Dynamicland + Folk Computer use projector+computer vision as substrate to try and reinvent/rethink the interactive/dynamic medium, and what end-user programming could be (which is very cool).
In that way I am much less ambitious; this exploration is more about how there is an entire class of existing interactions (the ones showed in the piece) that we perform today on screens, that could be much more interesting + collaborative if overlaid in the real world. And I am also terribly uninspired by the eyewear-based computing vision all major companies have pushed lately.
I mean, merge one of these with a modern anti-consumer "Smart TV", and we're uncomfortably close to 1984's telescreen.
Youtube videos - https://www.youtube.com/results?search_query=sixth+sense+tec...
Reminds me of the projection bracelet scam that Captain Disillusion debunked years ago https://youtu.be/Cw7g8ixbomU?t=75
Most of the videos are shot in real life, during summer daylight (only exception are the 2 person around a table ones, which are shot at night in a room with regular ceiling lighting).
And the projector I used is only 1100 ISO lumens; fancier models go way higher.
If you project mostly white content, it's remarkably readable + visible. Outdoors in direct sunlight would be another story, of course.
The main question I tend to get from hacker-types is "what are the tech specs?". The demos run on a Mac with custom software I built for projection mapping/rendering/etc. (the hand tracking uses Apple's built-in framework, which is very good), a small consumer 4k laser projector, and a basic webcam for vision.
But really, this is primarily a research/design prototype - I think it's much more important to figure out what the user experience should ideally feel like, if the interactions are even plausible & desirable in the first place; and then work backwards to the technology.
I wasn’t sure if any of this was real and that example looked extremely fake to me because I had a hard time believing the device being able to read text from that distance.
(I think I overestimated how much of an angled shot it was the first time I saw it, on top of how easy or hard turning your finger into “actually capture this text”… but you’re totally right that in principle a camera at a non agressive angle can capture that text)
(I like this overall concept a lot! Cooking is the right example, voice interfacing’s 96% use case is basically “my hands are dirty but I must use computing”)
At the very least it's going to have a range limit on that feature.
And there's a projector in there too.
In something smaller than a smart phone.
It's a great concept, but not a very workable one. Kitchens get steamy, tables can be huge, and management will want to use the cameras for user surveillance.
Possible use case: projecting menus, ads, and bills on restaurant tables. With a camera and enough automated layout to work around what's on the table.
It might sell to the people who buy refrigerators with displays on the door. Could be useful as a cooking training aid, like cooking videos but present where you need them.
The light bulb form factor is cute but probably not that useful. Most of the light will be aimed in non-useful directions. Look into ultra short throw projectors.
Who iterate and hone until they have something worth existing in today’s world—often doing do more publicly than corporations’ R&D departments—then let the patent technicalities be the well-paid lawyers’ problem…
Nothing truly new under the sun
gonna hard disagree! you aren't going to have 360° projector in there with wasted light going nowhere. you're going to have 1-4 steerable projectors. the "facade" as a globe bulb means you can place it at any direction you like depending on where the cord needs to go. for ceiling as well you don't want it to have a fixed throw direction. bulb makes perfect sense.
It's not mentioned in the piece, but for the latter I don't see any technical obstacle to having several of these in different rooms in the house, or a couple in the same room to augment coverage.
How many watts does the projector use for how many lumens? I would be curious if these numbers plus size would be a fundamental limit to the tech.
The main issue is probably gonna be that the display sucks compared to actual displays :( but it looks cool though! and the whole VLM thing makes a lot of sense
Sure, it's a little more hands free, and projecting around definitly is nicer than screens. But projecting has a lot of problems too. Shadows, projection surface, ambiant light, heat are difficult to handle, but to me the worst is that it's not persistant. You can't just project 24/7 your family planner on the wall.
I'd really rather have something that prints, or better draws.
You want a recipe => print it, don't beam it.
You want a map of kyoto => tiny robot walks around the page and draws it for you. And you can draw over it if you want.
And it doesn't have to be exclusive, you can do both beam and print. But I think protable printing / drawing has a lot of unexplored potential.
Printing otoh requires hands and prep. That's greater effort. You wouldn't print a recipe if you were just a little uncertain about step 3, you would ask for some kind of just in time modality instead. Voice might be fine for short info but sometimes you'd prefer to have it up and there's beam.
But more importantly I ultimately think that one of the unstated goals is to reduce screen time. And setups like these are only one voice command away from beaming youtube videos all around your house.
If I printed each recipe I followed... gush that would be a lot of paper, and also I bet that paper would get messy very quickly.
For a map, losing the ability to zoom, mark points etc seems like a big loss. Sure you can draw on it (how do you erase your mistakes?), but if everything you draw just ends up lost when you discard the paper, that's not so ideal.
I guess what disturbs me is that I’d rather have one screen in my pocket than tens of screens in my house.
OTOH this would be a great thing for non-private spaces, that are watched by security cameras 24/7 anyway. A school, a library, a clinic, an office, a workshop, a warehouse, a parking garage, maybe even a museum.
Of course, trust and security are going to be major considerations. Maybe not for everyone though: people voluntarily enable various voice assistants, and even pay to order Amazon Echo.
I feel like the demos are disingenuous because they suggest that the lightbulb is responsible for the entire thing.
However, the light bulb concept is limited to indoor usecases - which is not bad - that's where people live most of the time.
The broader goal should be to get rid of dependency on screens, technologies and information. That's when AI can stop enslaving us with HITL etc.
I think screens are time tested and will have some enduring use for now, but I could see people liking this; I've wanted to try a projector screen out but there was a whole rabbit hole of choices that didn't seem to improve upon screens for me for now (and this takes things in a whole other direction with projection)
I also have (I think, haven't counted recently) 12 cameras that are all ballhead so they move around and look for me when I say something to the smart speaker.
But I didn't think of the projector ... there's ~$30 ones ... might be worth buying a few.
Getting this level of response, feedback and accuracy, that's the real magic. The latency is garbage. I'm just using cheap OTS hardware though.
Anyone can do this ... it's not that expensive. Smart bulbs are cheap, wifi cams are cheap, omnidirectional usb microphones are cheap and projectors are cheap too. I'm talking like ~$100 total
“Commander Riker is in his quarters.”