Your point is well made. The question is really what is food and what isn't. To a vegetarian everything other than vegetarian food is not food. To a non-vegetarian everything other than (possibly) human meat is food. To the Aghoras, everything on this earth including human shit and flesh from rotting human corpses is food.
Now to understand the psyche of the vegetarian who baulks at sitting at the same table as a non-vegetarian eating meat, I suggest that the non-vegetarian sits at the same table as a cannibal eating human flesh or better still an Aghora who is eating human shit!