Unfortunately, the ports.ubuntu.com repository is separate from the standard Ubuntu archive, and there are very few, if any, mirrors available that include armhf or arm64 packages. So, you can't really change the repository in this case.