The mainline rootfs is for devices which use upstream Linux kernel with open source drivers on standard GNU ABI, rather than Android. So, for PinePhone/Librem 5/RPi.
The hybris rootfs is for devices which require Android kernel and its bionic ABI (hybris is for binding to the Android HAL from within Linux).
If you're porting a device, you almost certainly need the hybris version.