Drunks judging other drunks -- how ironic is that?
One thing to keep in mind is that perhaps these other men don't know that you were raised to be a gentleman, and that you behave as one. Perhaps they are projecting their faults onto you. You assume that they know something of your character, but as we all know, occasionally character defects come up.
Also, you didn't say if the woman you had coffee with was a newcomer or someone with some time. If she was a newcomer, I can understand the concern that you might be "taking advantage", no matter how well you comported yourself. A lot of women who come into the rooms are extremely vulnerable due to a lack of self esteem, or perhaps they were in a violent relationship, or they lost their husband and/or family as a consequence of their drinking.
This is not to say that men aren't vulnerable for all the same reasons, and female predators abound in the rooms as well. That's why it is generally suggested that women stick with the women, and men with the men.
I see no harm in people of opposite sex having a cup of coffee together, provided they are both at a stable stage of sobriety. I have seen many loving relationships blossom between people in the rooms, but I have also seen broken hearts and shattered self esteem from those who entered into relationships before they even knew themselves.