First, please show me if you have some non-Muslim source.
Secondly, Buddhists did not come here from outside India. They were born in India. Buddhism was considered to be one of the branches of Hinduism. Buddha was even considered to be an incarnation of Vishnu. Even today, Hindus have absolutely no hatred or ill feelings towards Buddhists.
Thirdly, if you say that so many Buddhist temples were destroyed, it is also true that so many Buddhist temples were built here. What about that?
Finally, B R Ambedkar himself blamed Muslims for the decline of Buddhism in India.