You're not getting 4:1 on your money preflop. You're getting (20+X)
4+X), where X is whatever you'll have to call off postflop to get to showdown. If X is, say, 11 (which it was, if you only consider the flop), then it's actualy 31:15 ~= 2:1 and you obviously should fold.
More realistically, X is probably the rest of his stack, so 38.