After Uvalde, social media monitoring apps struggle to justify surveillance

With the country still reeling from the mass shooting in Uvalde, Texas, officials are scrambling for more ways to stop mass shootings — and facing hard truths about how ineffective many of our existing tools really are. Digital monitoring technology has come under particular scrutiny after reporting revealed that the Uvalde school district had experimented with a service called Social Sentinel, which claims to identify and alert schools to threats based on social media conversations.

It’s an increasingly common service as schools grapple with the chaos of social media, often raising serious privacy and speech concerns along the way. Systems like Social Sentinel promise to give genuine insight into the huge volume of information posted on social media every day, parsing out the signal from the noise so that educators can be informed of threats before harm takes place. For these firms, it can be a lucrative business — but often, they’re mining shallow insights from available data, providing few benefits to outweigh the privacy harms.

For privacy advocates, the lack of evidence for the technology’s effectiveness means that there are no sufficient grounds for the potential violations of privacy that come with its use. Hye Jung Han, a researcher at Human Rights Watch specializing in child rights, told The Verge that using surveillance technology on children could cause unwarranted harm:

“Could you imagine schools using toxic materials to build classrooms, even if it hadn’t met any safety standards? No,” said Han. “Similarly, to use unproven, untested surveillance technologies on children, without first checking whether they are safe to use, exposes children to an unacceptable risk of harm.”

Multiple requests for comment sent to Navigate360 — which acquired Social Sentinel in 2020 — did not receive a response.

The Uvalde school district was confirmed to have purchased monitoring capability from Social Sentinel in 2019–2020, though it is unclear whether the subscription was still active at the time of the shooting. However, even if it had been, the technology would have been unlikely to flag any of the shooter’s posts. There are now numerous reports of concerning activity surrounding the shooter’s online activity: he allegedly made frequent threats to young women and girls via chat apps, sent images of guns to acquaintances, and reportedly discussed carrying out the school shooting in an Instagram chat. But Social Sentinel is only able to monitor public posts and would not have had access to any content shared in private messages.

At the same time, there are significant privacy concerns with the software. In 2019, the Brennan Center for Justice outlined a range of civil and human rights concerns stemming from expanded social media monitoring in K-12 schools, among them the questionable effectiveness of the technology in combination with a tendency to disproportionately impact students from minority communities. In the same year, reporting by Education Week also covered the dramatic expansion of digital surveillance in schools, highlighting the large number of false positives generated by Social Sentinel’s technology. (Alerts were reportedly triggered by tweets about the Mark Wahlberg movie, Shooter and from a student pleased their credit score was “shooting up,” among other things.)

Of all US states, Texas has been the most enthusiastic about the use of digital surveillance for school children. A 2021 investigation by The Dallas Morning News found that no state has more school districts contracting with digital surveillance companies than Texas. But of the Texas districts that did take out these contracts, results were apparently mixed: a number of school districts that had paid for Social Sentinel told the Morning News that they had declined to renew contracts, describing a service that provided few actionable alerts or flagged mostly irrelevant information.

But while Social Sentinel advertises an ability to monitor a broad range of platforms, there’s some suggestion that its surveillance capabilities are dictated more by the accessibility of data sources than by their importance. A client presentation from the company shared by the EFF lists a range of social media sources for monitoring, including Instagram, YouTube, Vimeo, Flickr, Tumblr, WordPress, and even Meetup.

Data obtained by BuzzFeed News confirmed this through data obtained under the Freedom of Information Act, which showed the company skewed heavily towards Twitter monitoring. Of the 1,206 Social Sentinel alerts provided to BuzzFeed, 98 percent (1,180) related to tweets — even though Instagram, YouTube, and even Facebook are more widely used by younger demographics. But the conventions of Twitter — where the vast majority of posts are publicly visible, even unintentionally — mean that it is comparatively easier to monitor, providing a wealth of social media data on tap that can be assimilated by companies looking to boost their surveillance credentials.

In large part, the success of social media monitoring programs comes down to how few options school officials have. As more and more of young people’s social lives extend into the digital realm, school staff are aware that they need at least some insight into their students’ online activities to carry out their duty of care towards them. Tackling cyberbullying is recognized as a big challenge for teachers and was recently noted by the UN as a major concern of parents whose children use the internet. Mental health advocates have long highlighted the negative effects of social media on self-esteem, while other online harms like revenge porn are also sadly a part of young people’s lives online.

Into this landscape has come a range of companies promising to safeguard the wellbeing of children and teens through a range of monitoring services that will keep tabs on their digital lives: Social Sentinel is a major player, with Gaggle and Securly often cited as main alternatives.

Though the efficacy of services like Social Sentinel is contested, investors have backed social media monitoring companies to the tune of tens of millions of dollars, betting on the longevity of digital surveillance as a feature of the educational landscape. Despite critics’ objections, the perception of social monitoring as a high-tech, relatively low-cost safety solution has proved to be attractive for school districts across the country. Until schools can be made safe by other means — like desperately needed gun control legislation — it seems likely that the expansion of surveillance technology will continue.