Mordor and Isengard are the only two factions with the same builder model and skin.
Would it be unrealistic (in the context of Middle Earth) to make the Mordor builder a troll instead of an orc? I imagine they would have worked with the orcs when making buildings, with the orcs doing the thinking and the trolls the heavy lifting.
A potential model for a troll builder can be found in BFME1. The Evil campaign featured a 'Near Harad' mission for Mordor in which you have to transport tributes to the Haradrim. The unit for this was a troll carrying the tribute in a cart behind it.