I think you will find the the only dice that are swappable are those belonging to a regular general or a regular train command. The dummy dice does not belong to either of these and is therefore not swappable.
The bigger problem is that if you have an army with 5 dice, its immediately obvious because there isn't the need for a 5th dummy dice.
5 vs 4 dice does also skew the weather, but not as much as 3 vs 4 or 2 vs 3.
Interestingly, if commands are shattered they stop rolling dice which means the weather becomes more changeable at the end of the battle.