Switch to your combat log when going into towns and cities and you won't see what people say about you.