Hi Shannon, sorry this got long!
I would really encourage you to not get too disappointed before you take the second phase of the test, and before you get quantitative (number, actual counts) results from a lab and talk them over with a doctor. There's a lot of things that can affect sperm count, since it takes about 100 days for sperm to form start-to-finish. For example, did he have a fever in the last 100 days? Spermies don't like high temps (that's why testes hang down like they do), so a fever could lower his count temporariliy.
I would really encourage you to get a sperm analysis from your doc, and see what the numbers are. Then the doc can talk to you about different things you could do to help the numbers like not drinking, smoking, good nutrition, vitamins, changing style of undies and such. They will probably want to test the count again after a while to see if these things helped. Do you have insurance? You could check if they cover it as a diagnostic test. If not, shop around. Where I live, we have a lab that doesn't accept insurance payments, only cash from patients (you can get reimbursed from your insurance for the tests). As a result, they're a lot cheaper than other labs. And see if you could make payments if you want to test ASAP.
All hope is not lost. Sperm counts can vary over time. And I would encourage you to to deal with these issues that come up as problems you have as a couple, instead of his problems or your problems. He sounds like he's willing to work on TTC with help, which is great. I told my DH that I thought we should get a SA before we start and he won't do it until we try for a while and don't have any luck. Things are always worse when

is here, too.
Hang in there, we're hangin' with you! Take care.