If the intention of those (non-Hindus) who enter any Hindu temple is not to defile it, then, for heaven's sake, allow them inside and don't make a mountain out of a molehill! Let them admire the architecture, offer prayers or just to relish the atmosphere.
Also, it is my humble opinion that a temple's sanctity will NEVER be affected by allowing women inside it.
Our lives would be much better if we could get rid of our prejudices and archaic laws/beliefs and, at the same time, maintain certain traditions that are time-tested and will always hold good, irrespective of the era.
