‘Underground maps’ segment cities using fashion, AI

Cornell computer scientists have developed a new artificial intelligence framework to automatically draw “underground maps,” which accurately segment cities into areas with similar fashion sense and, thus, interests.

How people dress in an area can tell a lot about what happens there, or is happening at a particular time, and knowing the fashion sense of an area can be a very useful tool for visitors, new residents and even anthropologists.

“The question I’ve been interested in is, can we use millions of images from social media or satellite images to discover something interesting about the world?” said Utkarsh Mall, a doctoral student in the lab of Kavita Bala, professor of computer science and dean of the Cornell Ann S. Bowers College of Computing and Information Science.

Mall is lead author of “Discovering Underground Maps from Fashion,” which he presented at the Winter Conference on Applications of Computer Vision, Jan. 4-8 in Waikoloa, Hawaii.

Co-authors are Bala; Tamara Berg,  research scientist at Facebook; and Kristen Grauman, professor of computer science at the University of Texas, Austin, and a research scientist at Facebook AI Research.

This research builds upon—and actually employs—the Bala group’s previous work that resulted in the AI tool GeoStyle, that can discover geospatial events and forecast fashion trends.

“There’s just so much you learn about human beings by looking at the images they post about themselves,” she said. “You learn about their culture, their style, how they interact with people, and what’s important to them.”

“There’s a lot of individual personality that comes across in how people dress, so analyzing fashion around the world was one of our first goals,” said Bala, whose areas of expertise include computer vision.

Using a fashion recognition algorithm on photographs geolocated from 37 large cities, the researchers were able to detect clothing styles, then typical combinations of those styles within a given radius. The team then used artificial intelligence to detect pockets of a city that were both spatially and stylistically coherent.

The resultant information can be used in several ways:

• to find unique neighborhoods in a city: Based on the fashion sense in a given district, one could determine the most trendy or progressive areas of a city; 
• to find similar neighborhoods across cities: For someone moving, for example, from New York City to Washington state, one could determine “the SoHo of Seattle”; and 
• to find neighborhood analogies: The researchers use the example of Coney Island and its relationship to New York City being similar to Australia’s Bondi Beach and Sydney.

The researchers calculated the accuracy of their method using two human-centered benchmark programs, HoodMaps and OpenStreetMap, as well as polling actual residents of selected cities in the study. In all cases, the Bala group’s underground mapping better captured the sense of a neighborhood than existing methods.

In addition to giving a newcomer to an area some insider’s knowledge of a city, the underground mapping tool could benefit science and research, Bala said.

“The way anthropologists study culture is they go to a location, do interviews with local people and observe,” she said. “An automated tool like this would empower them to do more. It could help them discover new phenomena that they didn’t even know about, and let them drill down deeper within their analysis of why this phenomenon exists.”

Mall said it could also help researchers decades from now.

“We are excited about this idea,” he said, “that some future anthropologist could just run these tools and understand us—take the ‘underground pulse’ of the city—despite not having lived with us.”