Upgrade your pirate hall to gain the ability to create new types of buildings. Spend gems to build the second and the third Builder's hut. This will help provide enough workers to build your island. Do not spend gems on anything else until you have at least three builders.
