There are no technicalities or marriages that "count" more than others. You are perfectly within your rights to have a second wedding and to make it as big (or small) as you like, and to invite whomever you wish.
The people who love you won't care that it's not your first wedding.
People who love you just want to see you happy. Go with the choice that you don't think you'd regret 40 years from now. And don't let people bully you.
My view is that one has to focus on the admirable or worthwhile aspects of a person in order to avoid developing a runaway attitude problem which can lead to dislike or hatred. It is often easier to dislike than to find positive qualities.
I was lucky to find a father of two, with two failed relationships behind him, after two similar trainwrecks of my own. So I feel you are not beyond hope at all; while the innocence of youth may not return, yet there is much more to be savoured.