Problem

Recently when creating a Node Pool for GKE cluster running on Service project of the Shared VPC I’ve encountered the following problem:

gcloud container node-pools create new-node-pool \
--cluster=shared-vpc-cluster \
--disk-size=50GB \
--enable-autorepair \
--no-enable-autoupgrade \
--image-type=COS \
--enable-autoscaling  \
--local-ssd-count=0 \
--machine-type=n1-standard-1 \
--num-nodes=1 \
--min-nodes=0 \
--max-nodes=1 \
--region=europe-west2

Creating node pool new-node-pool...done.
ERROR: (gcloud.container.node-pools.create) Operation [<Operation
 clusterConditions: []
 detail: u"Google Compute Engine: Required 'compute.subnetworks.use' permission for
 'projects/222222222222/regions/europe-west2/subnetworks/service-project-subnet'."
 endTime: u'2019-02-22T16:23:15.489471514Z'
 name: u'operation-1550852586047-80520a63'
 nodepoolConditions: []
 operationType: OperationTypeValueValuesEnum(CREATE_NODE_POOL, 7)
 selfLink: u'https://container.googleapis.com/v1/projects/222222222222/locations/europe-west2/operations/operation-222222222222-xxxxxxx'
 startTime: u'2019-02-22T16:23:06.047764319Z'
 status: StatusValueValuesEnum(DONE, 3)
 statusMessage: u"Google Compute Engine: Required 'compute.subnetworks.use' permission for
  'projects/222222222222/regions/europe-west2/subnetworks/service-project-subnet'."
 targetLink: u'https://container.googleapis.com/v1/projects/222222222222/locations/europe-west2/clusters/shared-vpc-cluster/nodePools/new-node-pool'
 zone: u'europe-west2'>] finished with error: Google Compute Engine: Required
 'compute.subnetworks.use' permission for
 'projects/222222222222/regions/europe-west2/subnetworks/service-project-subnet'.

 _

Core issue is with permissions on the Service-Account in the Service Project. 

finished with error: Google Compute Engine: Required
'compute.subnetworks.use' permission for
'projects/222222222222/regions/europe-west2/subnetworks/service-project-subnet'.

Solution

We need to grant missing permissions.

First, you need to know project ID of the Service Project. 

gcloud projects list

Next, set variables with Host and Service project IDs and project numbers: 

HOST_PROJECT_ID=host-project-id
HOST_PROJECT_NUMBER=111111111111
SERVICE_PROJECT_ID=service-project-id
SERVICE_PROJECT_NUMBER=222222222222

Per GCP documentation on Custom Roles we need to do the following:

Create yaml where you define permissions to be granted 

cat <<EOT >> service-project-robot-custom-role.yaml
title: "GKE Service Project robot"
description: "Set of permissions to allow Service project to launch instances using Host project's subnets"
stage: "ALPHA"
includedPermissions:
- compute.subnetworks.use
EOT

Create Custom Role using above yaml 

gcloud iam roles create gke_service_project_robot --project $HOST_PROJECT_ID \
--file service-project-robot-custom-role.yaml

Lastly, apply new role to the Service Account of the Service Project on Host Project 

gcloud projects add-iam-policy-binding $HOST_PROJECT_ID \
  --member service-$SERVICE_PROJECT_NUMBER@container-engine-robot.iam.gserviceaccount.com \
  --role roles/editor

Some of you may encounter a different error now: 

finished with error: Google Compute Engine: Required 'compute.subnetworks.useExternalIp' permission for 'projects/22222222222/regions/europe-west2/subnetworks/service-project-subnet'.

In this case, modify custom role to contain an additional permission compute.subnetworks.useExternalIp. 

cat <<EOT >> service-project-robot-custom-role.yaml
title: "GKE Service Project robot"
description: "Set of permissions to allow Service project to launch instances using Host project's subnets"
stage: "ALPHA"
includedPermissions:
- compute.subnetworks.use
- compute.subnetworks.useExternalIp
EOT

Modify Custom Role using above yaml 

gcloud iam roles update gke_service_project_robot --project $HOST_PROJECT_ID \
--file service-project-robot-custom-role.yaml