Apple Expands Its On-Device Nudity Detection to Combat CSAM

In December, Apple announced that it was killing a controversial iCloud photo-scanning tool the company had devised to combat child sexual abuse material (CSAM) in what it said was a privacy-preserving way. Apple then said that its anti-CSAM efforts would instead center around its “Communication Safety” features for children, initially announced in August 2021. And at the company’s Worldwide Developers Conference in Cupertino today, Apple debuted expansions to the mechanism, including an additional feature tailored to adults.

Communication Safety scans messages locally on young users’ devices to flag content that children are receiving or sending in messages on iOS that contain nudity. Apple announced today that the feature is also expanding to FaceTime video messages, Contact Posters in the Phone app, the Photos picker tool where users choose photos or videos to send, and AirDrop. The feature’s on-device processing means that Apple never sees the content being flagged, but beginning this fall, Communication Safety will be turned on by default for all child accounts—kids under 13—in a Family Sharing plan. Parents can elect to disable the feature if they choose.

“Communication Safety is a feature where we really want to give the child a moment to pause and hopefully get disrupted out of what is effectively a grooming conversation, or at least might be,” says Apple’s head of user privacy, Erik Neuenschwander. “So it’s meant to be high-friction. It’s meant to be that there is an answer which we think is likely right in that child’s situation, which is not to move forward, and we really want to make sure they’re educated.”

Apple said in December that it planned to make an application programming interface (API) available so third-party developers could easily integrate Communication Safety into their apps and use it to detect child sexual abuse material, or CSAM. The API, known as the Sensitive Content Analysis framework, is available now for developers. Platforms like Discord have already said that they plan to incorporate it into their iOS apps.