Sound like your siding is is good shape, overall. A primer isn't necessary in your situation. If you have any areas that are peeling, then you will need to spot prime these areas before painting.
Pay special attention to the lip of the siding. This area is masonite sidings weak spot.