This is weird, paint doesn't normally act like this. I suppose it could be possible for small 'flakes' or particles to come off a surface and land elsewhere. If this is happening then it is a major failure of the paint film.
Have you talked to your neighbors about this problem? Are they noticing the same thing?