The loose latex needs to come off. There isn't a primer that can glue it to the underlying oil based paint. Adding another layer on top of the latex paint would simply be a thicker layer that will peel.
Removing this layer will be difficult and time consuming. Try using hot water and a scouring pad. This should work well. For loose yet stubborn spots use steel wool and water or a paint scraper.
It is possible that the underlying oil base paint has lead. This is always a possibility with a older home. Using hot water will lesson the risk. Don't worry if some latex paint won't come off. This can be primed over.
Once the loose paint is removed prime with a good oil based primer. This will seal up the old paint. Now you can paint with a good acrylic paint.