Stood here like a lemon

Does anyone know the origins of the phrase (left me)stood here like a lemon, parents say it and they are from Hull....I was wondering if it was to do with lemons left rotting in the dock or market ?