When you can cross compile your programs you can copy them to your phone and run them.
How difficult this task will be depends largely what dependencies your programs have.
Before some time (Canonical was still working on Ubuntu Touch). I remounted the system rw and installed some programs via apt. Than they could be used in terminal like normal programs.
I do not know if apt can still be used, but with your own programs this should be a possible way. Maybe it is sufficient to have them only in your home directory.
By the way it should also be possible to build click apps with unconfined access. I never tried it my self, but I have used own build apps with extended privileges to access the sdcard in my phone.