Quality numerical simulations of the dynamics of a given many-body electronic structure system is an important research area in material analysis and nano-optics, etc. Quantities such as the time-dependent dipole moment are essential for further study. There are two components in such simulations, i.e., the ground state calculation, and the following dynamic simulations with the ground state as an initial state. These two components can be obtained by solving Kohn-Sham and time-dependent Kohn-Sham equations, respectively. In this paper, based on the finite element method, a unified numerical framework is proposed for the whole simulation. For the ground state calculation, the classical self-consistent field iteration method is employed for the linearization of the equation, in which the derived generalized eigenvalue problem is solved by the locally optimal blocked preconditioned conjugate gradient method, and we also design an effective preconditioner based on the multigrid method for the acceleration of the iteration. For the simulation of the dynamics, an implicit midpoint scheme is used for the temporal discretization, while the linear finite element method is used for the spatial discretization. A predictor-corrector method is used for the linearization of the equation, and an algebraic multigrid solver is developed for the derived complexvalued system in order to accelerate the simulation. In particular, an h-adaptive finite element method is developed for further improving the efficiency, in which two residual type a posteriori error indicators are designed for the Kohn-Sham and time-dependent Kohn- Sham equations, respectively. A variety of numerical experiments verify the effectiveness of our method.