Oh gosh but I know that feeling
One suggestion is to have two save files. One where you keep them all and the other where you harden your heart and work at becoming a horse trainer/trader. That way, when you feel bad, you can go back to the other save where you kept them.
Of course this could result in many, many save files as you find that the next horse you adopt to train up is equally as adorable!