Darwin builders on AWS

Darwin builders on AWS run on EC2 Mac Instances, which are dedicated Mac Mini hosts. These dedicated hosts must be allocated for at least 24 hours at a time. They can be reimaged at any time while allocated, but the reimaging process takes around an hour. Thus, for faster refresh time on hermetic builders, we run buildlets as MacOS guests inside of QEMU on the dedicated hosts.

Creating a dedicated host

Note that if you simply need more instances, an AMI with the final state is saved on the AWS account.

To bring up a new host:

1. In the EC2 console, go to “Dedicated Hosts” -> “Allocate Dedicated Host”.
2. Configure host type, zone. Instance family mac1 is amd64; mac2 is arm64.
3. Enable “Instance auto-placement”, which allows any instance to run on this host.
4. Once the host is allocated and available, go to the “Instances” page and click “Launch an instance”.
5. Select a macOS AMI. If starting fresh, select the latest macOS version from “Quick Start”. If simply adding more instances, a fully set-up AMI is saved in “My AMIs”.
6. Select a “Key pair” for SSH access. ec2-go-builders for official builders, a custom key for testing. You will need the private key to login.
7. Configure a 200GB disk.
8. If creating from a fully set-up AMI, uncheck “Allow SSH Traffic”.
9. Under “Advanced”, select “Tenancy” -> “Dedicated host”.
10. Other settings can remain at default. Launch instance.

If creating from a fully set-up AMI, you are done!

SSH with the key pair using the “Public IPv4 DNS” address from the “Instances” page. This won't appear until the instance is booted.

$ export KEY_PATH=~/.ssh/ec2-go-builders.pem
$ ssh -i $KEY_PATH ec2-user@$INSTANCE

See the AWS docs for setting up remote desktop access. Note that not all VNC client work with Apple's server. Remmina works.

The OS will only use 100GB of the disk by default. You must increase the volume size to utilize the full disk. This can be done while the disk is in use.

Continue below to create a new guest image, or skip ahead to use a pre-created image.

Creating a new guest image

Steps to create a new QEMU macOS guest image:

1. Build (make dist) or download a copy of the OpenCore bootloader from https://github.com/thenickdude/KVM-Opencore.
   1. Grab the .iso.gz file, gunzip it, and rename to opencore.img (it is a raw disk image, not actually an iso).
2. Create a macOS recovery disk image:
   1. Clone https://github.com/kholia/OSX-KVM.
   2. cd scripts/monterey && make Monterey-recovery.dmg
3. Download the UTM QEMU fork and extract to ~/sysroot-macos-x86_64.
   1. Available as Sysroot-macos-x86_64 in https://github.com/utmapp/UTM/actions?query=event%3Arelease builds.
4. Create a disk image to install macOS to.
   1. DYLD_LIBRARY_PATH="$HOME/sysroot-macos-x86_64/lib" "$HOME/sysroot-macos-x86_64/bin/qemu-img" create -f qcow2 macos-monterey.qcow2 128G
5. Determine the magic Apple OSK value.
   1. Either read it directly from the machine, or find it in some code.
6. Copy the shell scripts from this directory to $HOME.
7. Use $HOME/start-installer.sh macos-monterey.qcow2 opencore.img Monetery-recovery.dmg $OSK_VALUE to launch the macOS installer in QEMU.

NOTE: If networking isn't working on older versions of macOS, swap the networking flag in qemu.sh.

This starts QEMU with the display on a VNC server at localhost:5901. Use SSH port forwarding to forward this to your local machine:

$ ssh -i $KEY_PATH -L 5901:localhost:5901 -N ec2-user@$INSTANCE

Then use a VNC client to connect to localhost:5901.

1. Once connected, select “macOS Base Image” from the bootloader to launch the installer.
2. In the installer, open the Disk Utility, find the ~128GB QEMU hard disk, click “Erase”, name it “macOS”, and leave other options at the default settings.
3. When formatting is complete, close Disk Utililty, and select “Reinstall macOs Monterey”.
4. Click through the installer. The VM will reboot a few times. When it does, select “macOS Installer” from the bootloader to continue installation. Installation is complete when “macOS Installer” is replaced with “MacOS”.
5. Select “macOS” and go through the macOS setup as described in the generic setup notes.

Once macOS is fully installed, we will install OpenCore on the primary disk and configure it to autoboot to macOS.

1. In the guest, find the OpenCore and primary disks with diskutil list.
   • The OpenCore disk contains only one parition, of type “EFI”. e.g., it may be /dev/disk0, EFI partition /dev/disk0s1.
   • The primary disk contains two paritions, one of type “EFI”, one of type “Apple_APFS”. e.g., it may be /dev/disk2, EFI partition /dev/disk2s1.
2. Copy the OpenCore EFI partition over the primary disk EFI partition.
   • sudo dd if=/dev/disk0s1 of=/dev/disk2s1
3. Mount the primary disk EFI partition to edit the configuration.
   • sudo mkdir /Volumes/EFI
   • sudo mount -t msdos /dev/disk2s1 /Volumes/EFI
4. Open /Volumes/EFI/EFI/OC/config.plist.
   • Change the Misc -> Boot -> Timeout option from 0 to 1 to set the bootloader to automatically boot macOS with a 1s delay.
   • In the 7C436110-AB2A-4BBB-A880-FE41995C9F82, section, add -v to the boot-args string value. This applies the nvram option mentioned in the setup notes.
5. Shutdown the VM and boot it again with start-mutable.sh.

$ $HOME/start-mutable.sh macos-monterey.qcow2 $OSK_VALUE

Now complete the remainder of the machine setup. For SSH access, the guest should be reachable at 192.168.64.2 or 192.168.64.3 from the host machine.

Copy complete images to s3://go-builder-data/darwin/ for use on other builders.

Set up automated guest creation

1. Download the latest image from s3://go-builder-data/darwin/ and save it to $HOME/macos.qcow2.
2. Download the UTM QEMU fork and extract to ~/sysroot-macos-x86_64.
   1. Available as Sysroot-macos-x86_64 in https://github.com/utmapp/UTM/actions?query=event%3Arelease builds.
3. Copy bootptab to /etc/bootptab.
4. Restart the system DHCP server to pick up the new bootptab: sudo /bin/launchctl unload -w /System/Library/LaunchDaemons/bootps.plist; sudo /bin/launchctl load -w /System/Library/LaunchDaemons/bootps.plist.
5. Build golang.org/x/build/cmd/runqemubuildlet and copy it to $HOME.
6. Create $HOME/loop1.sh:

#!/bin/bash

while true; do
  echo "Running QEMU..."
  $HOME/runqemubuildlet -guest-os=darwin -macos-version=${MACOS_VERSION?} -osk=${OSK_VALUE?} -guest-index=1 -buildlet-healthz-url="http://192.168.64.101:8080/healthz"
done

4. Create $HOME/loop2.sh:

#!/bin/bash

while true; do
  echo "Running QEMU..."
  $HOME/runqemubuildlet -guest-os=darwin -macos-version=${MACOS_VERSION?} -osk=${OSK_VALUE?} -guest-index=2 -buildlet-healthz-url="http://192.168.64.102:8080/healthz"
done

Replace `${OSK_VALUE?} with the OSK value described in the previous section.

5. Setup Automator:
6. Open Automator
7. File > New > Application
8. Add “Run shell script”
9. open -a Terminal.app $HOME/loop1.sh
10. Save to desktop twice as run-builder1
11. File > New > Application
12. Add “Run shell script”
13. open -a Terminal.app $HOME/loop2.sh
14. Save to desktop twice as run-builder2

Note that loop1.sh and loop2.sh guests will have a display at VNC port 5901 and 5902, respectively.

6. Setup login:
7. Users & Groups > ec2-user > Login Items > add run-builder1
8. Users & Groups > ec2-user > Login Items > add run-builder2
9. Users & Groups > Login Options > auto-login ec2-user
10. Desktop & Screensaver > uncheck show screensaver

Once image is set up and working, stop the instance and create an AMI copy of the instance. On the EC2 “Instances” page, select the instance, and click “Actions” -> “Image and templates” -> “Create Image”.

Either create a new instance from this image with no SSH access, or edit the instance networking inbound rules to remove SSH access.

References

• EC2 Mac Instances USer Guide: Contains most of the useful how-tos on working with Mac instances.
• Remmina: VNC client that works with Apple's VNC server (not all clients work).
• QEMU MacOS install guide: Guide with similar instructions for MacOS in QEMU.