You are right. It is not the Sex which is involved. But once you are attached to anyone your thoughts will take you there and you will want to meet him/her again and again.
Half the troubles are there because we do not control our senses. Once we do that Life will be beautiful in our own eyes.
When we behave like Animals, our childeren also becomes Animals. All the illicit population and so called mistresses are the root cause for so called confusion/depression and greed. We can contribute by atleast controlling our senses.
Why repent later. Wise man thinks before the problem comes and a fool repents after the mess. Choice is ours.