That's true also, but different from what I was trying to get at. When someone comes up to a photographer and demands that you do not take pictures of them, their kids, their dog, etc. and/or demands to see your photos they are acting as though it's their right not to be photographed when in fact the opposite is true. Legally, the photographer is not prohibited from taking pictures in public. Harassing someone though by interfering with them, cursing at them, threatening, assaulting, damaging their gear (e.g. hollywood celebs) is in fact prohibited. The point is people tend to think they have rights they don't actually have and then get irate when they think you're violating those perceived rights.
Social norms are a different matter. People do feel uncomfortable being photographed and if I noticed someone specifically photographing me whether with a long lens or a smartphone I would feel uncomfortable. If they kept it up I might ask them to stop but would more likely just walk away. If they followed me and continued photographing me specifically now they might be crossing a legal line in the sense of harassing me.
So, yes I get your point but I was specifically talking about rights and the law, not social norms.