You need to remove the latex paint, lightly sand with 150 grit sand paper then prime with an oil based primer.
Hopefully removing the paint won't be to much work. If it is a little difficult, try using a mild paint stripper and water. After all loose paint is removed, sand the door with the sandpaper. Now wipe down with either a damp clean rag or tack cloth. It is now that your door is ready for priming and painting.
Once the latex is removed and the door is primed you can repaint with your latex. This time it will stick.