"I couldn't give a fig"

"I couldn't give a fig" - where does this come from, and why figs?

Replies