The answer is no. Elephants, squirrels and whales can have sperms of similar length. Indeed, the largest recently discovered spermatozoon is that of Drosophila bifurca, a gnat.
With the "tail" unrolled, the tangled spermatozoon exceeds 5 centimeters in length (20 times larger than the gnat itself). It is also the most fruitful: it has a 100 chance in 100 of meeting the egg because it occupies, all alone, the whole genital apparatus of the female.
Size doesn't matter. For comparison, the human spermatozoon measures 0.06 millimeters, and that of the hippopotamus 0.033 millimeters.