Small pot, big laydown?

[MRBAA]

Playing 1-5 stud (1-10 on river) on the commuter train -- all the players have played together many times, 8 handed. $1 bring in, one call, one fold, I make it $3 to go with (A4)A, (one other ace is out) all fold to a fairly loose, fairly erratic player who doesn't notice that I raised and tries to raise to $2 with a king in the door. It's pointed out that I raised and he just calls. All others fold and we are head up, with $9 in the pot. On fourth he pairs his door and I pair my side card, so the hands are as follows:

(xx)KK
(A4)A4

He says "let's get this hand over right here" and leads out $5 and I...