diff --git a/README.md b/README.md index 6de857424c4..e9f1720b51b 100644 --- a/README.md +++ b/README.md @@ -14,17 +14,78 @@ The repositories listed below contain the definitions of the required components For convenience, an [installation script](https://raw.githubusercontent.com/antmicro/runner/vm-runners/scripts/install.sh) is available that installs dependencies, configures the system, clones the repository and builds the runner. + ## Installation and configuration -The manual below assumes that Debian Buster is used to deploy the runner. +### catx-ext-umich specific procedures + +Start from scratch by following the procedure below: + +``` bash +gcloud compute instances stop instance-2 + +gcloud compute instances delete instance-2 + +gcloud compute instances create instance-2 \ +--project=catx-ext-umich \ +--zone=us-central1-a \ +--machine-type=e2-standard-32 \ +--network-interface=network-tier=PREMIUM,subnet=default \ +--maintenance-policy=MIGRATE \ +--service-account=terraform-runner@catx-ext-umich.iam.gserviceaccount.com \ +--scopes=https://www.googleapis.com/auth/cloud-platform \ +--create-disk=auto-delete=yes,boot=yes,device-name=instance-2,\ +image=projects/debian-cloud/global/images/debian-10-buster-v20210916,\ +mode=rw,size=256,type=projects/catx-ext-umich/zones/us-central1-a/diskTypes/pd-balanced \ +--no-shielded-secure-boot \ +--shielded-vtpm \ +--shielded-integrity-monitoring \ +--reservation-affinity=any + +# Delete previous image archive in bucket +# TODO: How to delete uploaded image (Not just in bucket) +gsutil rm gs://catx-ext-umich-worker-bucket/scalenode-9e1d63d.tar.gz + +# Connect to refreshed instance +gcloud compute ssh instance-2 --ssh-flag="-ServerAliveInterval=30" +``` + +Before running `setup.sh` make sure that the following is done: +- Install `git` upon ssh to the created instance +- The archive in the bucket and corresponding image is deleted + - Screenshot to-be included +- Fill in the `TOKEN` in `coor.sh` + - The `TOKEN` can be found here: https://github.com/idea-fasoc/OpenFASOC/settings/actions/runners/new?arch=x64&os=linux under `configure`. The `TOKEN` expires in a hour. + +After everything is setup, run `setup.sh` to build everything from scratch. (This script assumes that all virtual infrastructure created by Terraform does not need to be re-configured.) ### Host prerequisites +The manual below assumes that Debian Buster is used to deploy the runner. + The following packages must be installed: * `build-essential` * [Terraform](https://www.terraform.io/docs/cli/install/apt.html) +```bash +sudo apt-get install software-properties-common + +curl -fsSL https://apt.releases.hashicorp.com/gpg | + sudo apt-key add - + +sudo apt-add-repository "deb [arch=$(dpkg --print-architecture)] https://apt.releases.hashicorp.com $(lsb_release -cs) main" +``` * [Google Cloud SDK](https://cloud.google.com/sdk/docs/install#deb) +```bash +echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | + sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list + +sudo apt-get install apt-transport-https ca-certificates gnupg +curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | + sudo apt-key --keyring /usr/share/keyrings/cloud.google.gpg add - + +sudo apt-get update && sudo apt-get install google-cloud-sdk +``` ### Installation steps @@ -46,22 +107,24 @@ export SERVICE_ACCOUNT_ID=runner-manager gcloud iam service-accounts create $SERVICE_ACCOUNT_ID gcloud projects add-iam-policy-binding $PROJECT \ - --member="serviceAccount:$SERVICE_ACCOUNT_ID@$PROJECT \ + --member="serviceAccount:$SERVICE_ACCOUNT_ID@$PROJECT.iam.gserviceaccount.com" \ --role="roles/compute.admin" gcloud projects add-iam-policy-binding $PROJECT \ - --member="serviceAccount:$SERVICE_ACCOUNT_ID@$PROJECT \ + --member="serviceAccount:$SERVICE_ACCOUNT_ID@$PROJECT.iam.gserviceaccount.com" \ --role="roles/iam.serviceAccountCreator" gcloud projects add-iam-policy-binding $PROJECT \ - --member="serviceAccount:$SERVICE_ACCOUNT_ID@$PROJECT \ + --member="serviceAccount:$SERVICE_ACCOUNT_ID@$PROJECT.iam.gserviceaccount.com" \ --role="roles/iam.serviceAccountUser" # Create and download SA key. # WARNING: the export below will be used by Terraform later. -export GOOGLE_APPLICATION_CREDENTIALS +# This command is for the OUTPUT_FILE option for "keys create" +export GOOGLE_APPLICATION_CREDENTIALS=~/key.out + gcloud iam service-accounts keys create $GOOGLE_APPLICATION_CREDENTIALS \ - --iam-account=$SERVICE_ACCOUNT_ID@$PROJECT + --iam-account=$SERVICE_ACCOUNT_ID@$PROJECT.iam.gserviceaccount.com # Create a GCP bucket for worker image. export BUCKET=$PROJECT-worker-bucket @@ -72,7 +135,7 @@ Build and upload the worker image: ```bash # Clone the repository -git clone https://github.com/antmicro/github-actions-runner-scalerunner.git +git clone --recursive https://github.com/antmicro/github-actions-runner-scalerunner.git cd github-actions-runner-scalerunner # Compile bzImage @@ -80,13 +143,47 @@ cd buildroot && make BR2_EXTERNAL=../overlay/ scalenode_gcp_defconfig && make # Prepare a disk for GCP ./make_gcp_image.sh +``` -# Upload the resulting tar archive -./upload_gcp_image.sh $PROJECT $BUCKET +### Adjust Service Account priviliges before uploading built disk to GCP + +Save the bucket's IAM policy to a temporary (arbitrary) JSON file +```bash +gsutil iam get gs://$BUCKET > /arbitrary/path/file.json +``` +Get the project name and default service account email address. Adjust filter accordingly if a different service account is used +```bash +export PROJECT=$(gcloud config get-value project) +export SA=$(gcloud iam service-accounts list --filter=default | + grep -E -o '[a-z0-9._%+-]+@[a-z0-9.-]+(\.[a-z0-9._%+-]+)?[a-z]{2,4}') +``` +Get the absolute path of the Bucket config file +```bash +export BUCKET_FILE=/arbitrary/path/file.json ``` +Using the `sed` utility to insert required permissions associated with the bucket +```bash +sed -i 's/"bindings": \[/"bindings": \[\ + {\ + "members": \[\ + "projectEditor:'"$PROJECT"'",\ + "projectOwner:'"$PROJECT"'",\ + "serviceAccount:'"$SA"'"\ + \],\ + "role": "roles\/storage.legacyBucketOwner"\ + \},/' $BUCKET_FILE +``` +Upload the modified bucket file back to GCloud +```bash +gsutil iam set $BUCKET_FILE gs://$BUCKET +``` +Upload the resulting tar archive -Setup virtual infrastructure using Terraform: +```bash +./upload_gcp_image.sh $PROJECT $BUCKET +``` +### Setup virtual infrastructure using Terraform: ```bash git clone https://github.com/antmicro/github-actions-runner-terraform.git terraform init && terraform apply @@ -95,7 +192,7 @@ terraform init && terraform apply Connect to the coordinator instance created in the previous step: ```bash -gcloud compute --zone ssh +gcloud compute ssh gha-runner-coordinator --zone=us-west1-a ``` Install and configure the runner on the coordinator instance: diff --git a/coor.sh b/coor.sh new file mode 100644 index 00000000000..98eb9729d0c --- /dev/null +++ b/coor.sh @@ -0,0 +1,12 @@ +sudo -i -u runner bash +cd ~/github-actions-runner + +export REPOSITORY_ORG=idea-fasoc \ + export REPOSITORY_NAME=OpenFASOC \ + export TOKEN= \ + export SLOTS=1 \ + export SCALE=1 + +./config.sh --url https://github.com/$REPOSITORY_ORG/$REPOSITORY_NAME \ + --token $TOKEN \ + --num $SLOTS diff --git a/setup.sh b/setup.sh new file mode 100644 index 00000000000..2ed6fd14ebb --- /dev/null +++ b/setup.sh @@ -0,0 +1,50 @@ +# Setup prerequisites +sudo apt install build-essential -y +sudo apt install git wget unzip rsync bc \ + libelf-dev autotools-dev automake \ + gcc-multilib texinfo dosfstools mtools -y + +# Install Go for compiling BzImage +cd ~ +wget https://golang.org/dl/go1.16.5.linux-amd64.tar.gz && \ + sudo tar -xzf go1.16.5.linux-amd64.tar.gz -C /usr/local/ && \ + export PATH=$PATH:/usr/local/go/bin + +# Install Terraform +sudo apt-get install software-properties-common -y +curl -fsSL https://apt.releases.hashicorp.com/gpg | + sudo apt-key add - +sudo apt-add-repository "deb [arch=$(dpkg --print-architecture)] https://apt.releases.hashicorp.com $(lsb_release -cs) main" +sudo apt update && sudo apt install terraform -y + +# Install Google Cloud SDK +echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | + sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list + +sudo apt-get install apt-transport-https ca-certificates gnupg -y +curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | + sudo apt-key --keyring /usr/share/keyrings/cloud.google.gpg add - + +sudo apt-get update && sudo apt-get install google-cloud-sdk -y + +# Rebuild image +git clone --recursive \ + https://github.com/antmicro/github-actions-runner-scalerunner.git && \ + cd github-actions-runner-scalerunner/buildroot && \ + make BR2_EXTERNAL=../overlay/ scalenode_gcp_defconfig && \ + make + +export PROJECT=catx-ext-umich && \ + export BUCKET=$PROJECT-worker-bucket + +# Make and upload image +cd ../ && \ + ./make_gcp_image.sh && \ + ./upload_gcp_image.sh $PROJECT $BUCKET + +# ssh into coordinator instance to setup runner and run conifg +cd ~/runner +export name=$(gcloud compute instances list | grep gha | awk '{print $1}') && \ + export zone=$(gcloud compute instances list | grep gha | awk '{print $2}') && \ + cat coor.sh | gcloud compute ssh $name --zone=$zone +