Do I suck at yatzee? (Kind of long and most likely tricky)

[zinna]

A couple of weeks ago I was in Croatia for holiday and on my way home to the hotel I pass a casino with a big sign saying poker so I go in and ask if they offer any poker and they point towards a couple of video poker machines and Im on my way leaving when I seeing a couple of guys playing some game with money on their table so I go over and check it out. I approach the table and I can see that they are playing yatzee and Im thinking what heck I can play some yatzee if it isnt high stakes.

I ask what stakes they are playing and they explain in crappy english that they are playing for 2 kuna (about 0.4$) per point and you pay the diffrens between your score and the winners score and every one does that so there is only one winner. There are 4 players in the game if I join.

I join the game and order a beer and we are all playing and talking. After maybe 20 minutes or so the following happens: I and 2 of the other players are about even in score maybe 10 points diffrens or so and the last is probably trailing 40 points or so.

At this point I have completed 1 pair, 2 pair, small straight, full house, 3 of a kind, chance (not sure of the english term but its the one where you take the sum of your 5 dices and add up) ones (2 points), twos (8 points), threes (9 points) and fours (8 points). On my first roll I get 2 sixes but I cant get any more and Im now facing the decision on what I should erase (probably not the correct term but you will hopefully understand anyway) and I tell the guys who is writing down the points to remove my yatzee. He looks at me as Im the biggest retard he has ever met and ask me If I really want to erase my yatzee and I say: Yes remove my yatzee. He looks at me and shakes his head and removes it and the other guys start laughing and they say something in their home language that I dont understand and laugh even more.

Now the question: Is removing yatzee a really big mistake at this point? What should I have done otherwise? How many points am I winning/losing by removing my yatzee?

I understand that this isnt a standard question and that it might be very hard to answer but I would really like an answer since this has been bugging me for the last weeks.

Yes I know my english suck but Im from Europe so please have this in mind.