No, the 2 on the left belong to the mansions (victorian). The yellow and the tan pieces can be mixed, I think.
The right one belongs to smaller buildings. It will fit medieval houses for example. But since they're not the right height, I don't think they will fit with the mansion-steck pieces.