Ruling: Button not moved

[DeuceKicker]

You post the BB and play the hand out.

The next hand you throw out the small blind and turn away to order food or whatever. When you turn back you see that everyone has folded to a late-middle position player who has raised. The guy to your right (who should be the button this hand but has posted the small blind) is considering and the dealer tells you that you need two more chips to complete the big blind. You throw the extra chips out then immediately grab them back and say that the button has not been moved, and that you were big blind last hand.

The guy who raised says there has been too much action to declare a misdeal.

Ruling?

I didn't want to hold up the game, so I just tossed the guy the two chips, but I was curious so I asked the floor about it later. His answer made me wonder how the next hand/s would be handled.