We have left undone those things which we ought to have done

"...my poor grandmother once incidentally observing, in the course of an instruction upon the use of the Prayer-book, that it was highly improbable that I should ever do much that I ought not to do, but that she felt convinced beyond a doubt that I should leave undone pretty well everything that I ought to do."

Now, Smokey Stover indicated that grandmother uses here the language of the Common Prayer Book; and indeed that's what it written there:

"We have offended against thy holy laws. We have left undone those things which we ought to have done; And we have done those things which we ought not to have done."

My question is: What is the origin of the last excerpt? It must be the Bible. But where in the Bible?

Thank you.