[AJW]

I’m going to say about 18 months to 2 years is a reasonable goal for an average Joe.
How sporty you are has a huge effect on how quickly you can learn new physical skills. The sense of proprioception is more highly developed in some people then others.
Sports men tend to hold low handicaps if they play golf.

The five years of playing is probably going to hurt more then help you will have taught yourself a lot of things that will need to be unlearnt.
The horrible short game is probably going to help. As others have said most of your shots are taken inside 100 yards so if you can go from horrible to bad you can probably shave five strokes off your average round.