env/darwin/aws/README.md - build - Git at Google

 # Darwin builders on AWS

 Darwin builders on AWS run on [EC2 Mac
 Instances](https://aws.amazon.com/ec2/instance-types/mac/), which are dedicated
 Mac Mini hosts. These dedicated hosts must be allocated for at least 24 hours at
 a time. They can be reimaged at any time while allocated, but the reimaging
 process takes around an hour. Thus, for faster refresh time on hermetic
 builders, we run buildlets as MacOS guests inside of QEMU on the dedicated
 hosts.

 ## Creating a dedicated host

 Note that if you simply need more instances, an AMI with the final state is
 saved on the AWS account.

 To bring up a new host:

 1. In the EC2 console, go to "Dedicated Hosts" -> "Allocate Dedicated Host".
 2. Configure host type, zone. Instance family `mac1` is amd64; `mac2` is arm64.
 3. Enable "Instance auto-placement", which allows any instance to run on this
    host.
 4. Once the host is allocated and available, go to the "Instances" page and
    click "Launch an instance".
 5. Select a macOS AMI. If starting fresh, select the latest macOS version from
    "Quick Start". If simply adding more instances, a fully set-up AMI is saved
    in "My AMIs".
 6. Select a "Key pair" for SSH access. `ec2-go-builders` for official builders,
    a custom key for testing. You will need the private key to login.
 7. Configure a 200GB disk.
 8. If creating from a fully set-up AMI, uncheck "Allow SSH Traffic".
 9. Under "Advanced", select "Tenancy" -> "Dedicated host".
 10. Other settings can remain at default. Launch instance.

 If creating from a fully set-up AMI, you are done!

 SSH with the key pair using the "Public IPv4 DNS" address from the "Instances"
 page. This won't appear until the instance is booted.

 ```sh
 $ export KEY_PATH=~/.ssh/ec2-go-builders.pem
 $ ssh -i $KEY_PATH ec2-user@$INSTANCE
 ```

 [See the AWS
 docs](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-mac-instances.html#mac-instance-vnc)
 for setting up remote desktop access. Note that not all VNC client work with
 Apple's server. [Remmina](https://remmina.org/) works.

 The OS will only use 100GB of the disk by default. You must [increase the volume
 size](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-mac-instances.html#mac-instance-increase-volume)
 to utilize the full disk. This can be done while the disk is in use.

 Continue below to create a new guest image, or [skip ahead](#guest-creation) to
 use a pre-created image.

 ## Creating a new guest image

 Steps to create a new QEMU macOS guest image:

 1. Build (`make dist`) or
    [download](https://github.com/thenickdude/KVM-Opencore/releases) a copy of
    the OpenCore bootloader from
    https://github.com/thenickdude/KVM-Opencore.
      1. Grab the `.iso.gz` file, `gunzip` it, and rename to `opencore.img` (it
         is a raw disk image, not actually an `iso`).
 2. Create a macOS recovery disk image:
    1. Clone https://github.com/kholia/OSX-KVM.
    2. `cd scripts/monterey && make Monterey-recovery.dmg`
 3. Download the UTM QEMU fork and extract to `~/sysroot-macos-x86_64`.
    1. Available as `Sysroot-macos-x86_64` in
       https://github.com/utmapp/UTM/actions?query=event%3Arelease builds.
 4. Create a disk image to install macOS to.
    1. `DYLD_LIBRARY_PATH="$HOME/sysroot-macos-x86_64/lib"
       "$HOME/sysroot-macos-x86_64/bin/qemu-img" create -f qcow2
       macos-monterey.qcow2 128G`
 5. Determine the magic Apple OSK value.
    1. Either [read it directly from the machine](https://www.nicksherlock.com/2021/10/installing-macos-12-monterey-on-proxmox-7/#:~:text=Fetch%20the%20OSK%20authentication%20key), or [find it in some code](https://github.com/kholia/OSX-KVM/blob/master/OpenCore-Boot-macOS.sh#L45).
 6. Copy the shell scripts from this directory to `$HOME`.
 6. Use `$HOME/start-installer.sh macos-monterey.qcow2 opencore.img
    Monetery-recovery.dmg $OSK_VALUE` to launch the macOS installer in QEMU.

 NOTE: If networking isn't working on older versions of macOS, swap the
 networking flag in `qemu.sh`.

 This starts QEMU with the display on a VNC server at `localhost:5901`. Use SSH
 port forwarding to forward this to your local machine:

 ```
 $ ssh -i $KEY_PATH -L 5901:localhost:5901 -N ec2-user@$INSTANCE
 ```

 Then use a VNC client to connect to `localhost:5901`.

 1. Once connected, select "macOS Base Image" from the bootloader to launch the
    installer.
 2. In the installer, open the Disk Utility, find the ~128GB QEMU hard disk,
    click "Erase", name it "macOS", and leave other options at the default
    settings.
 3. When formatting is complete, close Disk Utililty, and select "Reinstall
    macOs Monterey".
 4. Click through the installer. The VM will reboot a few times. When it does,
    select "macOS Installer" from the bootloader to continue installation.
    Installation is complete when "macOS Installer" is replaced with "MacOS".
 5. Select "macOS" and go through the macOS setup as described in the [generic
    setup notes](../setup-notes.md).

 Once macOS is fully installed, we will install OpenCore on the primary disk and
 configure it to autoboot to macOS.

 1. In the guest, find the OpenCore and primary disks with `diskutil list`.
     * The OpenCore disk contains only one parition, of type "EFI". e.g., it may
       be /dev/disk0, EFI partition /dev/disk0s1.
     * The primary disk contains two paritions, one of type "EFI", one of type
       "Apple_APFS". e.g., it may be /dev/disk2, EFI partition /dev/disk2s1.
 2. Copy the OpenCore EFI partition over the primary disk EFI partition.
     * `sudo dd if=/dev/disk0s1 of=/dev/disk2s1`
 3. Mount the primary disk EFI partition to edit the configuration.
     * `sudo mkdir /Volumes/EFI`
     * `sudo mount -t msdos /dev/disk2s1 /Volumes/EFI`
 4. Open `/Volumes/EFI/EFI/OC/config.plist`.
     * Change the Misc -> Boot -> Timeout option from `0` to `1` to set the
       bootloader to automatically boot macOS with a 1s delay.
     * In the `7C436110-AB2A-4BBB-A880-FE41995C9F82`, section, add `-v` to the
       `boot-args` string value. This applies the `nvram` option mentioned in
       [the setup notes](../setup-notes.md).
 5. Shutdown the VM and boot it again with `start-mutable.sh`.

 ```sh
 $ $HOME/start-mutable.sh macos-monterey.qcow2 $OSK_VALUE
 ```

 Now complete the remainder of the [machine setup](../setup-notes.md). For SSH
 access, the guest should be reachable at 192.168.64.2 or 192.168.64.3 from the
 host machine.

 Copy complete images to `s3://go-builder-data/darwin/` for use on other
 builders.

 ## Set up automated guest creation {#guest-creation}

 1. Download the latest image from `s3://go-builder-data/darwin/` and save it to
    `$HOME/macos.qcow2`.
 2. Download the UTM QEMU fork and extract to `~/sysroot-macos-x86_64`.
    1. Available as `Sysroot-macos-x86_64` in
       https://github.com/utmapp/UTM/actions?query=event%3Arelease builds.
 3. Copy `bootptab` to `/etc/bootptab`.
 4. Restart the system DHCP server to pick up the new `bootptab`: `sudo /bin/launchctl unload -w /System/Library/LaunchDaemons/bootps.plist; sudo /bin/launchctl load -w /System/Library/LaunchDaemons/bootps.plist`.
 4. Build `golang.org/x/build/cmd/runqemubuildlet` and copy it to `$HOME`.
 5. Create `$HOME/loop1.sh`:

 ```sh
 #!/bin/bash

 while true; do
   echo "Running QEMU..."
   sudo $HOME/runqemubuildlet -guest-os=darwin -macos-version=${MACOS_VERSION?} -osk=${OSK_VALUE?} -guest-index=1 -buildlet-healthz-url="http://192.168.64.101:8080/healthz"
 done
 ```

 4. Create `$HOME/loop2.sh`:

 ```sh
 #!/bin/bash

 while true; do
   echo "Running QEMU..."
   sudo $HOME/runqemubuildlet -guest-os=darwin -macos-version=${MACOS_VERSION?} -osk=${OSK_VALUE?} -guest-index=2 -buildlet-healthz-url="http://192.168.64.102:8080/healthz"
 done
 ```

 Replace `${OSK_VALUE?} with the OSK value described in the previous section.

 5. Setup Automator:
   1. Open Automator
   2. File > New > Application
   3. Add "Run shell script"
   4. `open -a Terminal.app $HOME/loop1.sh`
   5. Save to desktop twice as `run-builder1`
   2. File > New > Application
   3. Add "Run shell script"
   4. `open -a Terminal.app $HOME/loop2.sh`
   5. Save to desktop twice as `run-builder2`

 Note that loop1.sh and loop2.sh guests will have a display at VNC port 5901 and
 5902, respectively.

 6. Setup login:
   1. Users & Groups > ec2-user > Login Items > add run-builder1
   2. Users & Groups > ec2-user > Login Items > add run-builder2
   3. Users & Groups > Login Options > auto-login ec2-user
   4. Desktop & Screensaver > uncheck show screensaver

 Once image is set up and working, stop the instance and create an AMI copy of
 the instance. On the EC2 "Instances" page, select the instance, and click
 "Actions" -> "Image and templates" -> "Create Image".

 Either create a new instance from this image with no SSH access, or edit the
 instance networking inbound rules to remove SSH access.


 ## References

 * [EC2 Mac Instances USer
   Guide](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-mac-instances.html):
   Contains most of the useful how-tos on working with Mac instances.
 * [Remmina](https://remmina.org/): VNC client that works with Apple's [VNC
   server](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-mac-instances.html#mac-instance-vnc)
   (not all clients work).
 * [QEMU MacOS install
   guide](https://www.nicksherlock.com/2021/10/installing-macos-12-monterey-on-proxmox-7/):
   Guide with similar instructions for MacOS in QEMU.
	# Darwin builders on AWS

	Darwin builders on AWS run on [EC2 Mac
	Instances](https://aws.amazon.com/ec2/instance-types/mac/), which are dedicated
	Mac Mini hosts. These dedicated hosts must be allocated for at least 24 hours at
	a time. They can be reimaged at any time while allocated, but the reimaging
	process takes around an hour. Thus, for faster refresh time on hermetic
	builders, we run buildlets as MacOS guests inside of QEMU on the dedicated
	hosts.

	## Creating a dedicated host

	Note that if you simply need more instances, an AMI with the final state is
	saved on the AWS account.

	To bring up a new host:

	1. In the EC2 console, go to "Dedicated Hosts" -> "Allocate Dedicated Host".
	2. Configure host type, zone. Instance family `mac1` is amd64; `mac2` is arm64.
	3. Enable "Instance auto-placement", which allows any instance to run on this
	host.
	4. Once the host is allocated and available, go to the "Instances" page and
	click "Launch an instance".
	5. Select a macOS AMI. If starting fresh, select the latest macOS version from
	"Quick Start". If simply adding more instances, a fully set-up AMI is saved
	in "My AMIs".
	6. Select a "Key pair" for SSH access. `ec2-go-builders` for official builders,
	a custom key for testing. You will need the private key to login.
	7. Configure a 200GB disk.
	8. If creating from a fully set-up AMI, uncheck "Allow SSH Traffic".
	9. Under "Advanced", select "Tenancy" -> "Dedicated host".
	10. Other settings can remain at default. Launch instance.

	If creating from a fully set-up AMI, you are done!

	SSH with the key pair using the "Public IPv4 DNS" address from the "Instances"
	page. This won't appear until the instance is booted.

	```sh
	$ export KEY_PATH=~/.ssh/ec2-go-builders.pem
	$ ssh -i $KEY_PATH ec2-user@$INSTANCE
	```

	[See the AWS
	docs](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-mac-instances.html#mac-instance-vnc)
	for setting up remote desktop access. Note that not all VNC client work with
	Apple's server. [Remmina](https://remmina.org/) works.

	The OS will only use 100GB of the disk by default. You must [increase the volume
	size](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-mac-instances.html#mac-instance-increase-volume)
	to utilize the full disk. This can be done while the disk is in use.

	Continue below to create a new guest image, or [skip ahead](#guest-creation) to
	use a pre-created image.

	## Creating a new guest image

	Steps to create a new QEMU macOS guest image:

	1. Build (`make dist`) or
	[download](https://github.com/thenickdude/KVM-Opencore/releases) a copy of
	the OpenCore bootloader from
	https://github.com/thenickdude/KVM-Opencore.
	1. Grab the `.iso.gz` file, `gunzip` it, and rename to `opencore.img` (it
	is a raw disk image, not actually an `iso`).
	2. Create a macOS recovery disk image:
	1. Clone https://github.com/kholia/OSX-KVM.
	2. `cd scripts/monterey && make Monterey-recovery.dmg`
	3. Download the UTM QEMU fork and extract to `~/sysroot-macos-x86_64`.
	1. Available as `Sysroot-macos-x86_64` in
	https://github.com/utmapp/UTM/actions?query=event%3Arelease builds.
	4. Create a disk image to install macOS to.
	1. `DYLD_LIBRARY_PATH="$HOME/sysroot-macos-x86_64/lib"
	"$HOME/sysroot-macos-x86_64/bin/qemu-img" create -f qcow2
	macos-monterey.qcow2 128G`
	5. Determine the magic Apple OSK value.
	1. Either [read it directly from the machine](https://www.nicksherlock.com/2021/10/installing-macos-12-monterey-on-proxmox-7/#:~:text=Fetch%20the%20OSK%20authentication%20key), or [find it in some code](https://github.com/kholia/OSX-KVM/blob/master/OpenCore-Boot-macOS.sh#L45).
	6. Copy the shell scripts from this directory to `$HOME`.
	6. Use `$HOME/start-installer.sh macos-monterey.qcow2 opencore.img
	Monetery-recovery.dmg $OSK_VALUE` to launch the macOS installer in QEMU.

	NOTE: If networking isn't working on older versions of macOS, swap the
	networking flag in `qemu.sh`.

	This starts QEMU with the display on a VNC server at `localhost:5901`. Use SSH
	port forwarding to forward this to your local machine:

	```
	$ ssh -i $KEY_PATH -L 5901:localhost:5901 -N ec2-user@$INSTANCE
	```

	Then use a VNC client to connect to `localhost:5901`.

	1. Once connected, select "macOS Base Image" from the bootloader to launch the
	installer.
	2. In the installer, open the Disk Utility, find the ~128GB QEMU hard disk,
	click "Erase", name it "macOS", and leave other options at the default
	settings.
	3. When formatting is complete, close Disk Utililty, and select "Reinstall
	macOs Monterey".
	4. Click through the installer. The VM will reboot a few times. When it does,
	select "macOS Installer" from the bootloader to continue installation.
	Installation is complete when "macOS Installer" is replaced with "MacOS".
	5. Select "macOS" and go through the macOS setup as described in the [generic
	setup notes](../setup-notes.md).

	Once macOS is fully installed, we will install OpenCore on the primary disk and
	configure it to autoboot to macOS.

	1. In the guest, find the OpenCore and primary disks with `diskutil list`.
	* The OpenCore disk contains only one parition, of type "EFI". e.g., it may
	be /dev/disk0, EFI partition /dev/disk0s1.
	* The primary disk contains two paritions, one of type "EFI", one of type
	"Apple_APFS". e.g., it may be /dev/disk2, EFI partition /dev/disk2s1.
	2. Copy the OpenCore EFI partition over the primary disk EFI partition.
	* `sudo dd if=/dev/disk0s1 of=/dev/disk2s1`
	3. Mount the primary disk EFI partition to edit the configuration.
	* `sudo mkdir /Volumes/EFI`
	* `sudo mount -t msdos /dev/disk2s1 /Volumes/EFI`
	4. Open `/Volumes/EFI/EFI/OC/config.plist`.
	* Change the Misc -> Boot -> Timeout option from `0` to `1` to set the
	bootloader to automatically boot macOS with a 1s delay.
	* In the `7C436110-AB2A-4BBB-A880-FE41995C9F82`, section, add `-v` to the
	`boot-args` string value. This applies the `nvram` option mentioned in
	[the setup notes](../setup-notes.md).
	5. Shutdown the VM and boot it again with `start-mutable.sh`.

	```sh
	$ $HOME/start-mutable.sh macos-monterey.qcow2 $OSK_VALUE
	```

	Now complete the remainder of the [machine setup](../setup-notes.md). For SSH
	access, the guest should be reachable at 192.168.64.2 or 192.168.64.3 from the
	host machine.

	Copy complete images to `s3://go-builder-data/darwin/` for use on other
	builders.

	## Set up automated guest creation {#guest-creation}

	1. Download the latest image from `s3://go-builder-data/darwin/` and save it to
	`$HOME/macos.qcow2`.
	2. Download the UTM QEMU fork and extract to `~/sysroot-macos-x86_64`.
	1. Available as `Sysroot-macos-x86_64` in
	https://github.com/utmapp/UTM/actions?query=event%3Arelease builds.
	3. Copy `bootptab` to `/etc/bootptab`.
	4. Restart the system DHCP server to pick up the new `bootptab`: `sudo /bin/launchctl unload -w /System/Library/LaunchDaemons/bootps.plist; sudo /bin/launchctl load -w /System/Library/LaunchDaemons/bootps.plist`.
	4. Build `golang.org/x/build/cmd/runqemubuildlet` and copy it to `$HOME`.
	5. Create `$HOME/loop1.sh`:

	```sh
	#!/bin/bash

	while true; do
	echo "Running QEMU..."
	sudo $HOME/runqemubuildlet -guest-os=darwin -macos-version=${MACOS_VERSION?} -osk=${OSK_VALUE?} -guest-index=1 -buildlet-healthz-url="http://192.168.64.101:8080/healthz"
	done
	```

	4. Create `$HOME/loop2.sh`:

	```sh
	#!/bin/bash

	while true; do
	echo "Running QEMU..."
	sudo $HOME/runqemubuildlet -guest-os=darwin -macos-version=${MACOS_VERSION?} -osk=${OSK_VALUE?} -guest-index=2 -buildlet-healthz-url="http://192.168.64.102:8080/healthz"
	done
	```

	Replace `${OSK_VALUE?} with the OSK value described in the previous section.

	5. Setup Automator:
	1. Open Automator
	2. File > New > Application
	3. Add "Run shell script"
	4. `open -a Terminal.app $HOME/loop1.sh`
	5. Save to desktop twice as `run-builder1`
	2. File > New > Application
	3. Add "Run shell script"
	4. `open -a Terminal.app $HOME/loop2.sh`
	5. Save to desktop twice as `run-builder2`

	Note that loop1.sh and loop2.sh guests will have a display at VNC port 5901 and
	5902, respectively.

	6. Setup login:
	1. Users & Groups > ec2-user > Login Items > add run-builder1
	2. Users & Groups > ec2-user > Login Items > add run-builder2
	3. Users & Groups > Login Options > auto-login ec2-user
	4. Desktop & Screensaver > uncheck show screensaver

	Once image is set up and working, stop the instance and create an AMI copy of
	the instance. On the EC2 "Instances" page, select the instance, and click
	"Actions" -> "Image and templates" -> "Create Image".

	Either create a new instance from this image with no SSH access, or edit the
	instance networking inbound rules to remove SSH access.


	## References

	* [EC2 Mac Instances USer
	Guide](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-mac-instances.html):
	Contains most of the useful how-tos on working with Mac instances.
	* [Remmina](https://remmina.org/): VNC client that works with Apple's [VNC
	server](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-mac-instances.html#mac-instance-vnc)
	(not all clients work).
	* [QEMU MacOS install
	guide](https://www.nicksherlock.com/2021/10/installing-macos-12-monterey-on-proxmox-7/):
	Guide with similar instructions for MacOS in QEMU.