Who is the "public" anyway?

Yes, I have another law-related post, but at least this one is not too technical.

I was talking to someone recently about an internship I did at the Public Defender's office. As I was talking, I starting thinking about the name of the office. Why are criminal defense attorneys for the poor people called "public defenders"? If you think about it, aren't the prosecutors really the ones defending the interests of the public? They're prosecuting criminals as a method of improving the community, by getting criminals off the streets, addicts into rehab, etc. Prosecutors typically bring suit on behalf of the city, the state, or "the people," and if the city, the state or the "people" don't constitute the public, then I don't know who does.