Basically the problem is Fradulent cheque Unauthorized entry in to the system. Delay in cheque encashment
Option-1 if the people is having credit or debit card.
All the people are need to have to credit or debit card to check their money status online during the transaction Preferable credit card if the cheque is fail to encash then amount will be detected from their credit card automatically.
Option-2 If they dont have card
Whoever buying shares need to register with the firm with completed details about their finance including bank account no so their financial status can be regulary checked with bank and they can rated by the agency about their financial range in which they can trade.
Option-3
Member who is buying the shares are encouraged to take credit card ( Share agencies can partnership with bank for the same- now-a-days life time free card is available )
Only authorized people are allowed to do the cheque entry if the cheques are found to be wrong then whoever entered is need to pay the loss that occured due to that transaction . You should not fix any percentage simply whatever amount your paying for the loss needs to be pay the defaulters about double the times.
If the default is occuring for more than '3' occasion from the same person, he needs to be blacklisted from teh share market.
Basically you need to validate your customer by checking his financial health,give rating based on that his range can be fixed,credit card(as a safety measure).