Lost your marbles

What is the origin behind the phrase: Have you lost your marbles?

If you are considered to be losing your marbles then you are going mad or people think you're nuts.

Just a guess.
There is an Italian or Latin word--marmor--which means shining,sparkling or gleaming.
Maybe if you lose your shine--your sparkle--you are becoming undone.

Seems like it may also belong in the category of a few other phrases such as "one brick short of a load" etc etc ad nauseam.