Facebook is developing new algorithms that can track everything you do

Imagine a world where one of your devices can track everything you do in your day-to-day life. It can see, hear and remember everything for you and recall that information when needed. That sort of innovation is brilliant and could do wonders for gadgets like augmented reality (AR) glasses that could do all of the above — see, hear, and remember. But it’s also very creepy. Add Facebook into the mix, and the creepiness level maxes out. For years, Facebook was at the center of various scandals centered around user privacy and security. The latest one comes from a whistleblower who exposed details about what Facebook knows about the toxicity of its apps and how its algorithms might favor harmful content to drive up interactions.

Having that company develop artificial intelligence (AI) that can see and hear the things around you and then remember everything doesn’t sound too reassuring. But the tech is interesting nonetheless. That’s because said AI might target a different kind of product than Facebook’s social networks. That’s the AR glasses that the company is developing for the future.

Facebook’s AR glasses algorithms

Facebook introduced the Ego4D project on Thursday. The AI initiative “aims to solve research challenges around egocentric perception.” Facebook wants to develop algorithms that allow AI to interact with the world as people do. The AI would learn everything from a first-person video point of view. That’s the kind of perspective AI would have of the world if part of AR glasses.

Think of the AR experiences you’ve seen in Marvel movies involving Tony Stark. Iron Man interacts with an intelligent AI that sits on his head. The algorithms can see and hear everything he does. And they can project virtual information that Iron Man can see immediately when interacting with devices. That’s essentially what Facebook wants its algorithms to do, as seen in a demo video that Facebook shared.

Facebook explained it developed five benchmark challenges for developing the kind of smart AI seen in the clip. Facebook wants AI powering AI glasses to remember events in the past, predict what the user might want to do, detect how the wearer is interacting with real-life objects, and recall things said in the past:

• Episodic memory: What happened when? (e.g., “Where did I leave my keys?”)
• Forecasting: What am I likely to do next? (e.g., “Wait, you’ve already added salt to this recipe”)
• Hand and object manipulation: What am I doing? (e.g., “Teach me how to play the drums”)
• Audio-visual diarization: Who said what when? (e.g., “What was the main topic during class?”)
• Social interaction: Who is interacting with whom? (e.g., “Help me better hear the person talking to me at this noisy restaurant”)

The privacy implications are massive

For such a Facebook algorithm to succeed, the AI would need access to everything you do while wearing AR glasses. It would need to record video, including other people, process audio, and store lots and lots of data. So the privacy implications here are incredible. Facebook doesn’t just have to develop this amazing AI experience. It has to convince users to trust it with all the data the AR glasses they’d generate. That’s a topic that Facebook doesn’t cover in the video.

Facebook also doesn’t say anything about whether this algorithm would harvest data for advertising purposes. This is another topic that will immediately come to mind when you hear about an AI that can see, hear, and remember everything you do. To get back to the Iron Man analogy, you never heard Tony Stark’s Jarvis or Friday try to sneak ads in his peripheral view while he was fighting bad guys.

A Facebook spokesperson told The Verge that the privacy safeguards would be developed only later:

We expect that to the extent companies use this dataset and benchmark to develop commercial applications, they will develop safeguards for such applications. For example, before AR glasses can enhance someone’s voice, there could be a protocol in place that they follow to ask someone else’s glasses for permission, or they could limit the range of the device so it can only pick up sounds from the people with whom I am already having a conversation or who are in my immediate vicinity.

That’s to say that, sadly, privacy isn’t part of these advanced Facebook algorithms by design.