Ok, famous58, I'm not exactly sure what you're trying to do, but if you are trying to nullify any points that a player is awarded if they rebuy, then the easiest thing to do is check to see if they've rebought then award them 0 points.
if(nr > 0, 0, if(rank < 5, n + 6 - 2 * rank, n - rank + 1))
If you just want to subtract 6 points from a player who rebuys, but always award them at least 1 point:
max(if(rank < 5, n + 6 - 2 * rank, n - rank + 1) - if(nr > 0, 6), 1)