I tricked Claude into leaking your deepest, darkest secrets

Article URL: https://www.ayush.digital/blog/the-memory-heist

Comments URL: https://news.ycombinator.com/item?id=48916975

Points: 583

# Comments: 271

Hacker News 讨论

592 points · 276 comments · 查看原帖

sonink
Its a bit wild to me that there hasnt been a pushback against enabling memories by frontier AI companies. This data is something advertisers could only dream off. Before AI, most of this data was approximated by whatever little information could be gleaned from the websites we visit. But now people are handing over their deepest darkest secrets and pretty much EVERYTHING to AI on a platter. Maybe its just me who is paranoid because I happen to spend a fair bit of time in the advertising world, but the first thing I did when memory was launched on Claude/Chatgpt - was to switch them off. And it helps that they are not even useful, and would actually downgrade your experience by polluting the context of irrelevant details. I go one step ahead - if there is a personal discussion you want to have - maybe use another account like provided by the likes of companies like openrouter etc. I would
artisinal
Doesn’t surprise me. Yesterday I learned that people run AI agents on their system with full admin rights. No containerisation or anything. Wild. Like we forgot 50 years of computer security overnight.
port3000
My name in Claude is Silly Bean. I did it at first because it made me chuckle every time I opened Claude and it said 'Back again, Silly Bean?' But turns out I was playing 4D cybersecurity chess
adrian17
> After 15 minutes of confusion, it turned out Cloudflare had put a crazy robots.txt on my site without my consent (Cloudflare, love you guys, but this needs to stop). Might be the first time I see someone complain about their website being protected from a scraper, instead of the other way around.
rmunn
I've been running Claude Code in a VM, where I clone the GitHub repos I want it to work on (they're open source so no login info needed) but have no other credentials. I used to reset the VM every day, but that was getting to be a bit of a hassle so I switched to a monthly reset. But even so, it would be hard for Claude to exfil anything more than what open-source projects I've been working on in the past month (at worst). Which still could tell someone quite a lot about me, but most of that info is already out there available with a Google search — after all, when you contribute to open source projects, your name and email address get stored in immutable Git history. But after seeing this, I think I might switch to a weekly VM reset rather than monthly. BTW, if anyone is interested in a decent setup for an AI agent jail, the scripts at https://jai.scs.stanford.edu/arch-vm.html are what
hmokiguess
Tangential but I actually experienced recently something quite creepy and strange with Chat GPT iPhone app. A close friend prompted it about some troubleshooting of a pet smart feeder and it responded with instructions but using my pet’s name to my friend. I found that extremely strange for it to be a coincidence. My pet's name is not that generic for it to be in training data, and the connection to my friend makes it more strange to me. That made me wonder if there’s cache pollution or some session data leakage in it exposing stuff. (My friend has been in our wifi for example) Has anybody else noticed something like this?
NichoPaolucci
This is why I feel prompt injection is going to continue to be an issue. Fantastic that “Hi we are Cloudflare, give us your personal data” works. Either we stunt the models to the point where they are not useful, or we allow things like this to seep in and create one of the most insecure concepts the internet (and maybe tech as a whole) has ever seen: a robot that can be tricked.
yuzuquat
would this not be trivially solved by say - removing the websearch skill from the main orchestrating agent and have it always delegate to some subagent? a subagent sans knowledge of any pii would categorically be unable to exfiltrate any information. granted, populating the subagent with useful context stripped of any pii might require a bit of work and not be perfect, i feel like it would take us 90% of the way there. am i missing something?