Must the person in the image sign up?
Apple has named nonconsensual AI imagery in its app rules. I want the person depicted to be able to seek help without joining the service.
“Published contact information so users can easily reach you” is the last bullet in Apple’s rule for apps that let people contribute content.[2] It is a modest requirement. Reading yesterday’s update about nonconsensual intimate imagery, I wanted to know how far that word users extends.
Someone whose likeness has been used in a sexual image may never have installed the app that produced it. I want that person to be able to ask for help without becoming a customer first. That is my test for a useful reporting system, not a new requirement Apple explicitly announced.
On October 16, Apple said its updated App Review Guidelines require apps to proactively prevent and remove nonconsensual intimate imagery, including AI-generated imagery.[1] The current section 1.2 expressly includes AI content creation apps and requires filtering, reporting with timely responses, blocking abusive users, and published contact information.[2] MacRumors independently reports the change and notes that sexually explicit content was already prohibited; this is an expansion of the moderation rules, not Apple’s first prohibition on pornography.[3]
I welcome the explicit inclusion of generated images. “The camera never took that picture” is a poor answer to someone being made the subject of it. My concern is what the person can get done after the policy has correctly named the harm.
Reaching someone who can act
Apple says developers who lack adequate safeguards or fail to remedy violations after notice risk removal from the App Store.[1] Section 1.2 describes a compliance process, including removal until improvements are demonstrated, with immediate removal available for egregious or repeated behavior.[2] That is a lever over developers. It does not tell me how an individual report will go, and I would not mistake a revised rule for evidence that the affected person can now obtain a prompt remedy.
I would start with a public reporting page that works without an account, an acknowledgment with a reference number, and a way to explain that the problem concerns the person depicted. Those are my proposed features. I have not audited Apple’s store or the apps in it for their presence.
I would also want the reporting page to explain what evidence is necessary before asking for it. A person should be able to identify the relevant image or post without automatically being required to upload more photographs of themselves. If a reviewer needs additional information, explain why, who can see it, and how long it will be kept. An abuse report should not quietly become another collection of sensitive pictures.
There are limits to what a developer can fix. Removing a post on its service does not retrieve every copy someone may have saved elsewhere. Disabling an abusive account does not erase an image already sent in a message. I would rather receive a precise account of what was removed and what remains beyond the service’s control than a reassuring “resolved” with no explanation.
Prevention matters too. A reporting form cannot be the whole safety plan for a feature whose foreseeable misuse is putting another person’s face into sexual material. I want developers to examine what the feature makes easy before publishing it, rather than making the depicted person discover and document the problem afterward. I am not prescribing a detector that can reliably infer consent from pixels. I don’t have one to offer.
Consent without a general suspicion of sex
I am wary of allowing this argument to slide into a claim that all adult expression is the same problem. Consensual adult work, fictional erotic art, and a sexual fabrication targeting a real person raise different questions. My objection to the last one is about using that person’s identity without permission. Discomfort with sex is a much broader and less useful standard.
Apple’s own standard is broader: section 1.1.4 prohibits overtly sexual or pornographic material, while section 1.2 permits incidental mature content from web-based services under specified conditions.[2] Supporting the explicit protection against nonconsensual imagery does not require me to endorse every boundary of Apple’s adult-content policy. Nor would I describe this update as permission for a consenting adult-content app to enter the store.
For queer artists and communities, I want enforcement to distinguish an allegation of abuse from an objection to the people or relationships a work depicts. That is a standard I would apply, not a finding that this update has already removed queer work. A complaint saying “I dislike seeing these people together” should not acquire the force of a consent complaint simply because both arrive through the same form.
There is a serious difficulty here. A malicious reporter could falsely claim that a consensual image is unauthorized; an abuser could falsely insist that permission exists. Requiring an account might discourage some frivolous complaints, but I would not make joining the service the price of being heard. The better answer, in my view, is a reachable review process that can ask relevant questions, protect the reporter’s information, and give creators a meaningful route to challenge mistakes. I would accept more work in that process before accepting automatic removal as the final answer to every disputed report.
The update also gives hidden functionality its own rule, 2.5.19. Apple describes apps that conceal harmful capabilities during review and reveal them after approval; 9to5Mac confirms that the underlying prohibition already existed in section 2.3.1.[1] [4] I support holding developers responsible for the app people actually receive. But that responsibility also makes me reluctant to award a clean bill of health based on an approval badge. This announcement supplies rules and possible consequences, not a completed audit of every approved app.
I would judge the change by a less visible encounter than a successful App Review submission: someone opening a reporting page about an image of themselves. They should find a person or process able to act, learn what will happen to the information they submit, and get an answer that means something. They should not first have to press “Create account.”
Sources
[2] App Review Guidelines - Apple Developer
[3] New App Store Rules Target AI Deepfakes and Nudify Apps - MacRumors
[4] Apple’s new App Store rules crack down on AI nudification and ‘bait-and-switch’ apps - 9to5Mac