My husband is inferfile now but has a son with another woman but won't do a sperm donor...am I alone?

My husband has a 5yr old whom I absolutely adore, but with another woman from a previous relationship. Now 2 yrs after our marriage we learn he has extremely low sperm count with low mobility and morphology. We have had 5 IUI's performed and prior to my husband I have had a miscarriage with twins. Since then I have corrected the cause for myself. We now know we have a less than 3% chance of conceiving on our own. I am 25 and husband is 27 but we are ready for a larger family, both being raised in large families ourselves with both biological and adopted siblings but cannot afford adoption now. He says he cannot bear to raise another man's child and have to wait until 35 until we try donors but yet who am I....I am raising another woman's child and share EVERYTHING...how does that make me feel?? I love my stepson with all of my heart but want to have children permanently at home too... is he in the wrong or am I?????