Using the hard thermal loop (HTL) perturbation theory, we investigate the collective modes of gluon in an anisotropic thermal medium in the presence of a constant background magnetic field.
The momentum space anisotropy of the medium has been incorporated into the distribution function via the generalized $`$Romatschke- Strickland' form. The magnetic modification arises from the quark loop contribution where the lowest Landau level approximation has been considered.
We examine two special cases: i) spheroidal anisotropy with an anisotropy vector orthogonal to the external magnetic field, and ii) ellipsoidal anisotropy with two mutually orthogonal vectors describing aniostropies along and orthogonal to the field direction. We use the general structure of gluon self-energy that consists of six independent basis tensors. It is found that the strong background magnetic field has a significant impact on the growth rate of the unstable modes. This could have important effects on the equlibration of magnetized quark-gluon plasma.