Abstract: Transition plays an important role in the aerodynamic and aerothermal design of hypersonic vehicles. Generally, crossflow instability plays the dominant role in inducing transition in the three-dimensional boundary layer. However, in the noisy environment the influence of the second mode of the instability cannot be ignored. In order to obtain a better understanding of the development of crossflow in the hypersonic boundary layer, the measurement of instability waves in the boundary layer of a 7° half-angle sharp cone model with 6° angle of attack was carried out using focused laser differential interferometer (FLDI) and high frequency pressure fluctuation sensor (PCB) in Mach 6 Ludwieg tube tunnel. The experimental results show that the high-frequency unstable waves are detected at different circumferential positions. Through power spectrum and bi-spectrum analysis, the variation of the unstable wave along the generatrix and the nonlinear interaction between the high-frequency unstable wave and the low-frequency signal (20–40 kHz) are obtained.