50NL: How often should I go broke here?

[RunDownHouse]

I must look like a huge nit, I've been card dead, there's a bunch of loose players to my left and someone raising pf reasonably to my right, so I can't even steal that often. Villain is on another table I'm at, and so far he's shaping up to be something like 15/10 or 20/15, something like that. Doesn't get out of line. Stacks are about $50.

Villain limps UTG to me on button, I raise A[img]/images/graemlins/diamond.gif[/img] J[img]/images/graemlins/club.gif[/img]

Villain calls, pot is ~$6

Flop J[img]/images/graemlins/heart.gif[/img] 6[img]/images/graemlins/spade.gif[/img] 3[img]/images/graemlins/spade.gif[/img]
Villain checks, I bet $4, Villain calls.

Pot $10, Turn A[img]/images/graemlins/heart.gif[/img]
Villain checks, I bet $10, villain c/r's AI.

I can't decide if there are so many draw/worse two pairs out there that I should be ok calling, or if I should go more with my stat read and decide this is almost always a set.