Ungodly hour

Does anyone know the origin of the phrase, "ungodly hour?" I have used it many times and considered suddenly that it probably has an interesting origin.

Guessing that at a certain time, godly people who had to rise up early the next morning and earn an honest living were asleep. And the bad people -- midnight creepers, etc. -- were up and stirring at an ungodly hour.