Should I have pushed set harder on flop and turn

[hammerva]

Full Tilt Poker
No Limit Holdem Ring game
Blinds: $0.10/$0.25
8 players
Converter

Stack sizes:
UTG: $21.15
UTG+1: $20.20
MP1: $33.45
MP2: $20.35
hammerva: $12.05
Button: $31.10
SB: $9.25
BB: $37

Pre-flop: (8 players) hammerva is CO with A[img]/images/graemlins/diamond.gif[/img] 6[img]/images/graemlins/diamond.gif[/img]
UTG calls, 3 folds, hammerva calls, Button calls, <font color="#cc0000">SB raises to $0.5</font>, BB calls, UTG calls, hammerva calls, Button calls.

Flop: 6[img]/images/graemlins/spade.gif[/img] 6[img]/images/graemlins/club.gif[/img] 4[img]/images/graemlins/club.gif[/img] ($2.5, 5 players)
SB checks, BB checks, <font color="#cc0000">UTG bets $1.25</font>, <font color="#cc0000">hammerva raises to $3</font>, 3 folds, UTG calls.

Turn: 3[img]/images/graemlins/diamond.gif[/img] ($8.5, 2 players)
UTG checks, <font color="#cc0000">hammerva bets $4.5</font>, UTG calls.

River: 5[img]/images/graemlins/heart.gif[/img] ($17.5, 2 players)
UTG checks, hammerva checks.

Results:
Final pot: $17.5

According to PokerTracker UTG guy was very aggressive with a VI$P over 50 and an aggression factor of 1.4. From watching his play after this, he definitely never met 2 suited cards he didn't like. I have a feeling the opinion here will be "go all in on the flop (or turn)" but with a flush draw like that isn't that extremely risky. I want to chalk this hand as "Not much I can do" but I am pretty sure I did something wrong. Looking at the turn, my 55% of the pot bet looks really bad now.

Thoughts. [img]/images/graemlins/frown.gif[/img]