Tough bananas

My dad says, "tough bananas" all the time in place of "too bad". What is the origin of "tough bananas"? I've seen it elsewhere online but its not explained here.