It’s an LLM. All it’s supposed to do is generate output that fits with the given input. There isn’t some logic in it that makes sending an email an error, or anything that enforces detecting any kind of error. You can wrap it in other tools for that, but there isn’t any indication that’s the case.

More than that, the initial post says the user found out about the email and then queried the LLM, so it didn’t detect anything. There isn’t any bug here, this is how LLMs operate.

Your response contradicts what you said before. If you understand that an LLM doesn’t know things, how is it interesting that it said that “it crossed the line”? Saying they crossed a line is a common response to being told they shouldn’t do something, so it’s natural that an LLM would respond that, no matter what it did before.

It doesn’t know anything, it just parroted a response that fits with what the user wrote. That’s what LLMs do

For me the line is the cost of the AI tool: if it requires running on a datacenter and training with massive amounts of data obtained without consent of the creators, it is too costly for memes.

Other applications might have extra criteria, for example, I’ve seen nature enthusiasts point out that “enhancing” photos of animals might modify the photo enough that the animal loses its defining features