What would happen if i didn't let the non custodial parent of my child take her with him?

i have a 3 year old daughter with an ex husband who is currently living in another country with his other family. He has seen my daughter since he left to iraq in march of 07 approximately 2 times. the first time is when she turned 1 and the second time is when she was 2 1/2 and for 20 minutes when he came to our hometown on leave. he is currently not in the military anymore but his wife is, so she is currently stationed out of the country. he calls twice a week (he explains, because of the hour difference, its incovenient to call everyday) and hes sent her one birthday gift and thats it. no christmas, and no other birthdays. ive come so close to going to the judge and accusing him of abandonment. he stopped paying child support for awhile, and he even threatened several times to sign his rights over. NOW, he's paying, (when he wants to) and he claimed to be upset at the time he said he wanted to sign over. He chose to leave, and live somewhere else, even though we have joint custody. Yet, he still bitches about having her spend a week with him, when he gets here for a few weeks, even though he only gets every other weekend AND hes only going to BE in the states for a total of 2 weeks EACH year for about 3 years he says. I personally dont want to let her go, due to confusion and distress of not knowing who he is. Not sure what the guidelines are in tx for noncustodial parents and joint custody.any help? just wanna know whats exactly the worst that could happen, if i, the custodial parent, refused to give her up for his time?