"No Balls"

Can anyone shed any light on the phrase "No Balls"? When used in a sentence such as " you won't do that because you have not got any balls".
What is the true meaning of the phrase, what is its origin?
My understanding is that is means "a lack of courage", though some of my colleagues claim the remark can be construed as sexist. I disagree. Can anyone help?

Your understanding is correct--"a lack of courage".
Just a guess but I think the origin is quite sexist.
In the US women are discriminated against in jobs, pay and status. They are second class citizens to males.
The phrase is a reference to a woman's lack of a scrotum and, therefore, her lack of status since she does not have the ability to produce testosterone, that manly of all manly body chemicals.

We can fix this. How about -- "You don't have the ovaries to do that." Or, if someone is cowardly, call him or her "t*tless." What do you think?