Origin of fence phrase

I am searching for the origin of the phrase that goes something like: (he) played both sides of the fence.
I am thinking possibly Chaucer,,,any one have a lead?