Using the same white for the trim and walls won't supply enough contrast to be noticeable, even with a sheen difference. It is possible that some off white colors might look "dirty" with the trim painted white. But these colors would look a little dirty anyways.
Personally, I like contrast between the wall and trim colors and would go a little darker than "a fairly white shade of white".
Test your color combinations by applying paint samples to the wall and adjacent trim. Make the sample(s) on the wall fairly large (3×3). Applying a sample next to a door and base trim will be helpful.
Try to narrow your walls colors to 3 and have samples of "bright white" and "high hide white" for the trim. Bright white has mostly titanium white added and high hide white has a little color added for coverage. Once everything is done it is hard to tell the difference.
I have painted a lot of rooms with bright white trim and off white walls and never noticed any problems. The key is finding the perfect wall color that fits your life style and decor.