Origin of "passing the baton"

Does anyone know the origin of the phrase:

"passing the baton"

I'd guess it comes from the relay race. When your turn comes to an end, you pass the baton (or responsibility) to the next person in your team.