Do I have a child abandonment case..?

I have a 3 year old who will be 4 in Nov. Since he was about 6 months old his father nor his father's family has had ANYTHING to do with him. They all have my home, cell, and email. And still nothing. I went after child support about 2 years ago, and the did a DNA and it was confirmed that he was the father. Yet in 2 years I have got probably gotten 400 dollars(the settled amount was 80 a week). So, I meet the most wonderful man, and we have been together going on 3 years and married almost 2. My son calls him daddy and know NO different. And he really wants to adopt my son. My sons "donor" I will call him also has a IV drug habit...horrible one, and 3 domestic violence charges(one before me, one from me, and once recently), and a hand full of other charges. He will do any drug he can get his hands on, from Meth, to ANY pills, Pot, and if all else fails he will drink himself stupid. He violated one of his 3 probation in 3 different counties, and is currently serving a 1 year stint in Rehab(which somehow his family manged to get the courts to let him do that instead of 3 years in jail). HELP ME!!! What can I do to get this man out of my child's life for GOOD, and let the man who has been there adopt him and give him the life he deserves...thanks!!