You are correct. There was not "hindu" religion before muslims invaded the subcontinent. The religion itself is a term used by westerners.
I would call it "way of life". There were major groups who followed certain ways of life. There were people who believed in vedic rituals and sacrifices. There were some people who believed in "buddist" way of life. and some believed in "Jainist" way of life. and there were many others who had different life styles.
Even though brahmins were mainly following the vedic way fo life there were some who followed others too.
After seeing the success of buddhism some of its tradisions and practices were absorbed by other religions. The vedic people also started folliwing ahinsa etc. They also liked the way buddist people's idea of shunya or tathata. The three ways to achieve shunyata were mahayana, hinyana and vajrayana(tantric version)they Started worshiping "pradnya" and to achieve shunata started following tantrikism as this was the easiest route. In doing that either some of them lost their actual rituals described in vedas and started worshiping "Shiva" and "Vishnu". When patanjali a brahmin ruler started ruling india he wanted brahmins to rule the continent and re-wrote manusmrity and thus came to existance the caste system. Now in doing that brahmins kept their status as supreme and to happen that they wrote so many hipocritic scripts that people still believe are written by gods.
Just my observation from what I have read so far and analysed so far.
