The top Segre classes of tautological bundles of punctual Hilbert schemes of surfaces are numbers which generalize the degrees of secant varieties. They can be assembled in a generating function and The Lehn conjecture gives a complete description of this function, as obtained from a rational function by a change of variables. We establish geometric vanishings of these top Segre classes in certain ranges for K3 surfaces (which had been first obtained by Marian-Oprea-Pandharipande by different methods) and also for K3 surfaces blown-up at one point. We show how all the Segre numbers (hence the whole generating series) are formally determined by these vanishings and we thus reduce the Lehn conjecture to showing that the Lehn function also has these vanishing properties. Marian-Oprea-Pandharipande and Szenes-Vergne in turn used the present results to complete the proof of the Lehn conjecture.