No one responded, so I figured I'd shine some light here. @walidham, Your port is built for armhf. You need to build for arm64 on the V20, we're not ARMv7, we're ARMv8.
So when building, you'd likely use ARCH=aarch64 instead of ARCH=armhf, if you made the device tree yourself, you may have other things that need done in the kernel's configuration.