There is nothing religous about this claim, Buddy. It's all politics.
As per Vaishnavism and Sivaism, every bits and peaces are as pure as a religious monument. But you can't sell that to public to gain power. What do you think the caste system is? It is just a political move. People just grasp the political aspect and forget about the spiritual aspect. The real spiritual seaker won't mind about monument getting destroyed. Because he knows that he is not seeking monument but God. The Guys who cry about monuments are bascially driving for political gain