That actually looks close to what I *think* you're trying to do. A couple of things, though:
if (r>n*(a*0,01),0,1)*
OK, I see what you're trying to do here, which is multiply the entire formula by 1 of the player falls into the top a% ranked players, or 0 otherwise. In other words, only award points to the top a% of players. This looks correct except you need to replace that comma (,) with a period (.):
if (r>n*(a*0,01),0,1)* should be if (r>n*(a*0.01),0,1)*
Other than that it looks OK. But where do the values for "a", "P1", and "P2" come from? If these are values you want to be able to easily change each tournament, you need to add them to the formula, like this:
assign("a", 10);
assign("P1", 100);
assign("P2", 50);
if (r>n*(a*0.01),0,1)* roundUpToNearest (((10*sqrt(n))/(sqrt(r)) + if (r < 9,P1,0) + if (r < 3,P2,0)),10)