I agree with you. but have you ever thought why this happens. This is because, brahmins always try to protect their identity and try to force others to identify themselves the way brahmins want. offcource there are some economic reasons for it.
If they let people define their identity, they will be free some such accusaitons. :-) Let sikhs, jains buddhist ligayats, vaishnavas, shaivas identify themselves the way they want insted of bundling them in a fictitions religion "Hindu". Let people have freedom of work, worship and speech no one will blame brahmins. :-) Do not try to behave liv "hindu" religion is their private property.