The primer applied with an edger or brush is drying smooth but when I apply the primer with the roller, it is drying with an “orange peel” look.
A paint roller produces its own texture. This texture effect can be lessened by using a shorter tighter nap and thinning the primer a little. A splash of water in water based primer or paint thinner (mineral spirits) in oil base primer can help by allowing the primer to flow out on the surface.
Paint additives can also be used. For acrylic (water based) primers use Floetrol. For alkyd (oil base) primers use Penetrol. Both are available at your local independent paint store.