You know I wonder about that. A brief bout of googing suggests that the Colosseum in Rome was faced in travertine, which is a stone of white to yellow to brown coloration. So the PM coloration might actually be relatively accurate for the original building. I'd guess that provincial arenas were more likely gray stone or concrete, as you suggest. Interesting to think about.
By the same token those white statues on the PM colosseum should probably be naturally colored; my understanding is that ancient statues were not left as raw white marble but were painted to look as lifelike as possible. So one could just replace them with normal klickies!