Should NBA refs "let them play" in the final seconds?

[VarlosZ]

There seems to be an unwritten rule that, in the closing seconds of a contested game, only blatant, egregious fouls will actually be called. When that unwritten rule is ignored and the refs call the last few seconds the same as the rest of the game, fans and players of the offended team always invoke the unwritten rule and say it's ridiculous for the ref to make that call in that situation. The other side, of course, says that a foul's a foul, and quit you're whining.

Who's right?

While I wish the refs would be more consistent in these situations so players didn't start to expect special treatment, I'm definitely in the camp that says a foul with 20 minutes left is also a foul with 2 seconds left. Going the other way leaves the door open for too much subjectivity about how much leeway to give, and which situations are important enough to warrant a suspension of the normal rules. Still, I'd be interested in hearing a good argument for backing off on the foul calls.