Sounds like you did a great job with the prep. I know what you did is a lot of work. If this is mold the only thing I can think of the use of an oil base paint as the finish. Oil base paints allow more mold growth than acrylic paints, the oils act as food for mold. Also a mold inhibitor could have been added to the primer and paint. A mildewcide paint additive like Add-2 or MX-3 will help for a time.
I'm also wondering if this isn't mold you are seeing. Local auto traffic can cause a similar look from tire wear and exhaust. It will also leave stains that are hard to remove. Will this stuff mostly wash off with some soap and water?
If it is mold then yes it needs taking care of. You don't need to totally strip everything. A good washing and repainting might be all that is needed. Also a lot of work. If you decide to repaint with an acrylic paint you will need to prime so the new paint can stick to the oil base paint. Also use a mold resistant acrylic paint and add extra mildewcide for more protection.