It is ok to pay the price and then have anything we want. Every enjoyment has its own price. We get particular joys because we have done something nice to others in our previous lives. But if in this life we do not do good things but spend our good karma then in next life we will suffer.
It is like stealing something from the shop. So what will be the result? Jail!
So let's understand that this life is very short and to spend it for miserable sex which gives a second pleasure is a great loss. Where as we could have unlimited pleasure when we restrict ourselves JUST ONE life time out of billions which we already lived…