Definitely some kind of contamination interfering with the primer. Try a tinted shellac primer, might take 2 coats. Also, if your walls are smooth, no texture, sand the affected areas with 150 grit paper or medium sanding sponge. This will help get things to stick.
I'm also wondering if the Mr. Clean eraser left a residue. Do you noticed this problem mostly where the eraser was used?