Player fails to protect his action

[Mano]

Single table home tourney. I am fairly short stacked in middle position and push first in, there is a quick succession of folds, the small and big blind fold, the chips are pushed my way and I gather them up and toss in my hand, at which point the button looks up and says "I haven't folded", and indeed he has two cards which were shielded by his hands. It is a friendly home game, with many novices (the villian here is not a total newb, but I am fairly sure this is an honest mistake on his part), so I say whatever, pull my cards back and put the chips back in and let him call and play the hand out (he has about half as many chips as I do, and ends up winning a race with pocket twos, but that really doesn't matter). I am pretty sure I would be in the right here if I had put up a stink, and said there was significant action past him and he failed to protect his action so I should get the pot. Anyone disagree?