What do you mean the speaker can not be impeached easily? It only takes a 14 day's notice and a SIMPLE MAJORITY of the members of the current Lok Sabha to remove the speaker (or deputy speaker0 from the office. You can be sure that if the Congrss patrey moved such motion them the BJP will have no qualms in supporting it.
