I remove plaster from a brick wall in my old house (1921) and there was no smell. When I washed it with water to get the remaining dust, there was a strong ammonia smell. Plaster of Paris was made with ammonia, so I think that wetting it has somehow activated the smell.
I don’t think it’s just the paint. I think it’s a combination of paint and humidity/moisture that makes it smell.