Apple scheme to detect child abuse creates serious privacy and security risks, say scientists

Pressure from intelligence agencies

Apple’s summer announcement is the first time a major IT player appears to have been ready to give in to such government pressure in the west. Pressure from intelligence agencies and repressive governments to block, subvert or legally prohibit effective cryptography in digital communications has been incessant for over 40 years. But, faced with increasingly effective and ever more widely used cryptographic systems, these actors have shifted to attacks on endpoints and infrastructure instead, using methods including legally authorised hacking. 

“The proposal to pre-emptively scan all user devices for targeted content is far more insidious than earlier proposals for key escrow and exceptional access”

“The move highlights a decisive shift in the latest battle by intelligence agencies to subvert modern and effective cryptography,” Abelson and colleagues say today. “Instead of having targeted capabilities, such as to wiretap communications with a warrant and to perform forensics on seized devices, the agencies’ direction of travel is the bulk scanning of everyone’s private data, all the time, without warrant or suspicion.”

The principle of CSS is that high-quality cryptography would be permitted, but material matching government-supplied and loaded templates would be flagged and secretly exported.

“Technically, CSS allows end-to-end encryption, but this is moot if the message has already been scanned for targeted content,” they note. “In reality, CSS is bulk intercept, albeit automated and distributed. As CSS gives government agencies access to private content, it must be treated like wiretapping.

“Once capabilities are built, reasons will be found to make use of them,” they add.

Bulk surveillance

The authors criticise not only Apple’s incompetence in applying basic security principles, but also its culpable naivety in suggesting that such a system once deployed would not immediately be repurposed. Even if deployed initially to scan for illegal and publicly condemned child sex material, “there would be enormous pressure to expand its scope” – and no way to rein back the privacy- and safety-destroying tool they had created.

The “promise of a technologically limited surveillance system is in many ways illusory”, they caution. Once introduced, as the targeted terms or images would be secret, and secretly managed, how would Apple or any user prevent other materials being added to the list, including information that was lawful but displeased the government of the day in a powerful state?

Apple has already yielded to such pressures, such as by moving the iCloud data of its Chinese users to datacentres under the control of a Chinese state-owned company, and more recently by removing the Navalny voting app from its Russian app store.

The security experts also highlight the fatal error of placing powerful systems like CSS onto client devices, thus exposing them to repurposing, gaming, misdirection and deception by every class of bad actor, from a powerful nation-state to criminal drugs and murder gangs, to cyber smart teenagers trying to set each other up.

Flawed technology

As proposed by Apple, the first CSS system would use “perceptual hashing” to match images being copied to iCloud to a library of government-supplied image “fingerprints”.

Perceptual hashing does not test for an exact bit-for-bit match but for image similarity. 

Apple’s latest version of perceptual hashing, called NeuralHash, was launched in August and promoted as a way of securely and reliably detecting abuse images. Critics quickly demonstrated that the system produced false positives and could be reverse-engineered and then exploited.

Researchers took barely two weeks to reverse-engineer the version of NeuralHash algorithm built into iOS 14. It led to immediate breaches, including engineered evasion and engineered false positives. The system’s reputation plummeted when one team showed that it matched two utterly unlike real-world images. Apple withdrew NeuralHash a month later.

Another “perceptual hash” technique, Microsoft’s PhotoDNA, has also been reverse engineered to rebuild target photos, yielding sometimes recognisable, very low-resolution target images. 

Machine learning techniques would be even more vulnerable, as the model and the training engine would necessarily be exposed on vast numbers of devices, Adversaries could seek to “poison” learning algorithms with specially configured datasets.

For these reasons, the experts agree “we find no design space for solutions that provide substantial benefits to law enforcement without unduly risking the privacy and security of law-abiding citizens”.

As a “phase change” in information technology, client-side scanning “would gravely undermine [protection], making us all less safe and less secure” they say, concluding that “it is a dangerous technology”.