High harmonic generation (HHG) has developed in recent years as a promising tool for ultrafast materials spectroscopy. At the forefront of these advancements, several works proposed using HHG as an all-optical probe for topology of quantum matter by identifying its signatures in the emission spectra. However, it remains unclear if such spectral signatures are indeed a robust and general approach for probing topology. To address this point, we perform a fully ab initio study of HHG from prototypical two-dimensional topological insulators in the Kane-Mele quantum spin-Hall and anomalous-Hall phases. We analyze the spectra and previously proposed topological signatures by comparing HHG from the topological and trivial phases and across the phase transition. We demonstrate and provide detailed microscopic explanations of why, in these systems, none of the observables proposed thus far uniquely and universally probes material topology. Specifically, we find that the (i) HHG helicity, (ii) anomalous HHG ellipticity, (iii) HHG elliptical dichroism, and (iv) temporal delays in HHG emission are all unreliable signatures of topological phases. Our results suggest that extreme care must be taken when interpreting HHG spectra for topological signatures and that contributions from the crystal symmetries and chemical nature might be dominant over those from topology. They hint that a truly universal topological signature in nonlinear optics is unlikely and raise important questions regarding possible utilization and detection of topology in out-of-equilibrium systems.