The solution for me was: make the worker invisible and make the movementspeed incredible fast. It will solve the problem. If u dont want to have the worker invisible, it will work too. But why do u wish to see the worker?
Because it makes it more realistic. Having buildings pop out of the ground randomely when there has always been a worker doesn't really sound appealing. Or do buildings pop out of the ground?
I made the changes in my own "patch" year ago or so.. if u want i can check if i still have the files... its not popping out of the ground.. it just starts asap...without a random delay.