Brushing wainscoting is very time consuming. I recommend rolling but use a mohair roller, this roller has a very fine weave, plus your choice of paint will affect the final outcome. Oil base paint has better flowing characteristics, but it smells bad and takes a long time to cure, while modern acrylic paints are more handy to work with.
In order to reduce the "roller texture" consider conditioning your paint for better flow on the surface. Use a product like Floetrol for water based paints or Penetrol for oil based paints. These products will slow down the initial drying so the paint flows on the surface and reduce the stickiness for easier application.
Another way to apply the paint is to do all the cutting in, a section at a time, use a roller to quickly apply the paint to the main surface then lay it off with a brush from the top to the bottom.
The "lay it off with a brush" part is using a good brush to give the paint its final look. Right after rolling run a brush along the surface. Carefully hold the bush so the bristles lay flat on the surface. This isn't like cutting in with the brush.