Dwindle away to nothing.
The earliest known use of peter as a verb meaning dwindle relates to the mining industry in the USA in the mid 19th century, and it is reasonable to accept that that is where it originated. Thoughts of US mining at that date bring to mind images of the California Gold Rush, which is sometimes suggested as the source of this phrase. The earliest uses of the word in that context come from later, for example, this piece from the Wisconsin newspaper the Milwaukee Daily Gazette, December 1845, which pre-dates the California rush (although there was an earlier Georgia Gold Rush in 1829). The story concerns an old prospector who is comparing his dwindling life circumstances with his diminishing finds of the mineral galena (lead sulphide):
"When my mineral petered why they all Petered me. Now it is dig, dig, dig, drill, drill for nothing. My luck is clean gone - tapered down to nothing."
There are several other records of the use of 'petering' to refer to dwindling mining reserves in 1840s USA, although none of these explicitly uses the phrase 'peter out'. For such a reference we have to wait for a figurative usage in the American lawyer and writer Henry Hiram Riley's collection of articles - Puddleford and its People, 1854:
"He hoped this 'spectable meeting warn't going to Peter-out."
While the root source of 'peter out' is fairly certainly mining, there's no clear understanding of why the word 'peter' was chosen in this context. As always, when an etymology is uncertain, people like to guess.
'Peter' has many meanings, both as a noun and a verb, and so the speculations are wide-ranging. They include a suggestion of a link to Saint Peter and to the story that his faith in Jesus faded when he denied him before his crucifixion. There could also be a link between St. Peter being called the rock, in Matthew 16:18:
And I say also unto thee, That thou art Peter, and upon this rock I will build my church; and the gates of hell shall not prevail against it.
That may have made an association between Peter and rock, so that a dwindling seam of rock or mineral could be said to have petered out.
Another suggestion is an allusion to the French word péter (to break wind - literally to explode, but also used figuratively to mean fizzle), as in the phrase péter dans la main, meaning 'to come to nothing'.
Of all of the proposed derivations of the word 'peter' in the idiom 'peter out', the one that best stands up to scrutiny is the link to saltpetre (potassium nitrate). This mineral was a constituent of the gunpowder that was used as an explosive in mining and was also used to make fuses. Saltpetre is at least associated with something that miners would have known something about, i.e. mining, as opposed to theology or French quotations.
See other phrases that were coined in the USA.