Downhill putts are harder than uphill putts. They tend to pick up momentum. So you tend to hit it easier, but then it tends to break more! Plus you're likely to leave the ball short of the hole.
So what do you do? Well, practice more (!) and keep in mind the softer you hit the ball the more it will react to that "hill" right before the hole.
Enjoy the game again,
P.S. Lefties. Get ready to have your own spot on Mirror Image Golf Swing!