We investigate the long-time asymptotics of the fluctuation SPDE in the Kuramoto synchronization model. We mainly establish the linear behavior for large time and weak disorder of the quenched limit fluctuations of the empirical measure of the oscillators around its McKean-Vlasov limit. This is carried out through a precise spectral analysis of the unbounded evolution operator, using arguments of perturbations of self-adjoint operators and analytic semigroups. We state in particular a Jordan decomposition of the evolution operator which is the key point in order to show that the fluctuations of the disordered Kuramoto model are not self-averaging.