We develop a nonequilibrium statistical mechanical description of the evolution of point vortex systems governed by either the Euler, single-layer quasigeostrophic or two-layer quasigeostrophic equations. Our approach is based on a recently proposed optimal closure procedure for deriving reduced models of Hamiltonian systems. In this theory the statistical evolution is kept within a parametric family of distributions based on the resolved variables chosen to describe the macrostate of the system. The approximate evolution is matched as closely as possible to the true evolution by minimizing the mean-squared residual in the Liouville equation, a metric which quantifies the information loss rate due to model reduction. The point vortex approximation of the fluid dynamics allows the optimal closure, which is formulated on phase space, to be transferred to physical space resulting in an exact mean-field theory for the continuum limit. The near-equilibrium linearization of this theory is used to model the intrinsic relaxation rates for isolated coherent vorticity structures. The equilibration of initially disturbed vorticity fields is captured by a reduced model that has few resolved variables and no adjustable parameters. For the Euler and single-layer quasigeostrophic equations, the theory is used to model the axisymmetrization of a deformed vorticity patch. In particular, the reduced model exhibits how the rate of symmetrization depends upon the energy and the Rossby deformation radius. For the two-layer equations the study focuses on the relaxation of baroclinic perturbations of stable barotropic structures and the transfer of available potential energy to kinetic energy. The model predicts the dependence of the barotropization rate on the energy and the internal Rossby deformation radius. Both axisymmetrization and barotropization are prominent features of the coherent vortex structures observed in direct numerical simulations of two-dimensional and quasigeostrophic turbulence. The reduced model is tested against the evolution of an ensemble of point vortex systems to validate its predictions. Therefore, the reduced model furnishes a mathematical theory of these fluid dynamical phenomena.