Do you go to temples ?? If yes, why do you go to temples ?? You can pray before a photo at home. Right ??
Rituals and all are ok. But what is required, is devotion and love ... Bhakthi towards whichever god you believe in. Going to a temple may help increase that devotion. Why deny anyone that right?? What we dont want is people who come to just "see" the temple or sneer at the rituals and worship that happens inside.