Its all down hill from here

Does this mean that it's going to get harder or better?
I thought that it menat its going to get worse, but I am not sure.

My opinion: it could mean either depending on the context. If the speaker is finishing a difficult task, "down hill" would mean the hard part is finished. If the person has gone through a pleasant experience and says "It's all down hill from now" it means he's had a peak experience and nothing can match it.