How long did the paint dry before peeling and how did it first happen? Your right that there shouldn't be any problems. I have used a flat first coat many times without problems.
First stop peeling the paint, I assume you are picking at it. It does take time for a semi-gloss to cure and get fully adhered to the surface. Maybe your paint needs more time to cure. What brand are you using?