[Siegmund]

I would guess, and it's only a guess, that your thumb and forefinger don't apply perfectly equal amounts of force to opposite ends of the magnet, and impart a net acceleration to it.

You can test the theory by spinning ONE magnet on a flat surface, with the same hand motion. Guess: whether one or two magnets, clockwise or counterclockwise, the magnets move the direction your thumb pushed them. (Additional guess: you are right-handed.)