Reverse Convergence? View & control your phone from computer (like VNC & RDP?)
-
@advocatux Yes. I noticed that. It is clearly an incomplete proof-of-concept, but it the closest thing I have found to "remote control" thus far.
-
After reading up a bit on libmirclient, it looks like if I could just setup a socket on the phone that forwards to a mir-server running on the computer, that might actually do something without being too traumatic a learning experience.
-
2 extra cents here: while trying Convergence over wireless again the other days with a Pro5 and a FP2, i noticed that both phones see eachothers in the Display section!
It is not working though it states Connected when trying fyi, not sure if it could be that opening a port or starting a missing process can make it work... -
@tera said in Reverse Convergence? View & control your phone from computer (like VNC & RDP?):
Convergence over wireless
Is there a place I can find out more about this wireless convergence? This is the first I've heard of it!
-
Oh... I see... there is no control channel. This is the wifi display link, the miracast protocol or a hardware dongle, etc.
-
So this is as far as I got... apparently there is a version mismatch between ubports and upstream mir, so ATM this does not work.
Once: install mir on your PC
sudo dnf install mir-demos qterminal socatOnce: install socat on the phone
adb shell sudo bash <enter unlock code/password> mount -o remount,rw / apt-get update apt-get install socat mount -o remount,ro /Each time
miral-app SOCKET=$(ls /run/user/*/miral_socket) socat UNIX-CONNECT:$SOCKET EXEC:'adb shell socat STDIO UNIX-LISTEN\:/tmp/display2,reuseaddr' & adb shell MIR_SOCKET=/tmp/display2 unity8-dashFAILURE
mirserver: Rejected and disconnected a client (Unsupported protocol version)
-
I don't know if this is exactly what you are looking for @Osndok...but I feel that your convergence vision it kind of something close to what this guy shows in this post (in particular where he explains about X-forwarding): Kris's blog
Have a look
-
Thanks @matteo ... I am aware of X forwarding, but not how it might be leveraged to actually use any of the touch apps.
-
@osndok I personally tested that and it works more or less in the way you can open an app which is sitting into your phone popping it up straight on your desktop computer.
-
This is a nice idea, but there are no pre-existing components you can plug together to make it work.
Here's a way forward...
Mir implements "platforms" as loadable modules, and what you need is a platform that, in addition to the input and display hardware on the phone (which are needed for normal use of the phone) will also allow a program running on a PC to emulate both display and input devices.
With this, the PC program connecting would seem to Unity8 just like connecting an physical display, keyboard and mouse and allow you to exploit the existing "convergence" support.
The "android platform" used by Mir the phone was abandoned by Canonical, so it is entirely under the control of UBports. Adding this proposed "remote display & input" capability is only a matter of programming.
Regarding X forwarding...
As noted above, you can run some X applications on the phone with X forwarding and have them appear on a PC. But these would typically be installed in a chroot or LXD container so this approach doesn't give you access to the phone applications.
It is also possible to use Mir's "Mir-on-X" platform with X forwarding (with a limited number of Mir applications) but even if that could be extended to support the range of applications that Unity8 needs on the phone it would conflict with using the phone as a phone.
-
@alan_g ... If I understand you correctly, this socket-sharing mechanism would then only work for the most trivial case (like an "hello world" example app); because anything that actually tries to interface with an expected phone-like gizmo via mir would not find it.
-
@osndok said in Reverse Convergence? View & control your phone from computer (like VNC & RDP?):
@alan_g ... If I understand you correctly, this socket-sharing mechanism would then only work for the most trivial case (like an "hello world" example app); because anything that actually tries to interface with an expected phone-like gizmo via mir would not find it.
You're asking about X forwarding?
It is feasible to
ssh -X myphone, switch to a chroot and then install and run an X11 application e.g.xeyes.The problem is that doesn't give access to phone apps which run on Unity8, e.g. Contacts.
-
No... I'm not really interested in x11 forwarding (for the same reasons you mention).
Although... originally, I was thinking there would have to be something special on the computer side for this to work (like a mir server), but I wonder how difficult it would be to make an adapter (similar to what you mentioned), that would expose a new mir display over an x11 display protocol.
Sorta like a hacked-down version of miral... or the "opposite" of xmir.
Then you could do something like:
ssh -X myphone phablet-control... and the 'phablet-control' binary would add a mir platform (as you mention), and present itself as an x11 frame (e.g. the window size is the reported display size, etc.); and then you wouldn't even need anything special on the PC side!
-
@osndok said in Reverse Convergence? View & control your phone from computer (like VNC & RDP?):
if I could "just plug in my phone and its screen pops up as a window" (like any other VM)... that would happily solve the majority of my "convergence problems".
Are you hoping for something that would work with Ubuntu Touch much like Vysor (or similar) works with Android? (Vysor demo video here). If so, I too would like to see something like that.
As for using Vysor with Android, it had some problems when I tried it about a year ago, and I haven’t tried it again recently.
I notice that Genymobile (makers of Genymotion Android emulator) is developing scrcpy (which is similar to Vysor) under what appears to be the Apache License.
The UBports team is busy with 1,000 other projects. But something similar to scrcpy, but for use with Ubuntu Touch, would be a great 1,001st project!
-
@osndok said in Reverse Convergence? View & control your phone from computer (like VNC & RDP?):
Although... originally, I was thinking there would have to be something special on the computer side for this to work (like a mir server), but I wonder how difficult it would be to make an adapter (similar to what you mentioned), that would expose a new mir display over an x11 display protocol.
... and the 'phablet-control' binary would add a mir platform (as you mention), and present itself as an x11 frame (e.g. the window size is the reported display size, etc.); and then you wouldn't even need anything special on the PC side!
I'm not implementing this, but on the face of it it would solve more problems that it creates. E.g. there's no need to define and secure a PC-phone protocol.
-
Yes, @gizmochicken ... something very much like Vysor... but (hopefully) without the huge web browser, and open source.
-
@osndok said in Reverse Convergence? View & control your phone from computer (like VNC & RDP?):
Yes, @gizmochicken ... something very much like Vysor... but (hopefully) without the huge web browser, and open source.
Have a look at the GitHub page for scrcpy. Additional info here and here.
-
@gizmochicken ... It's rather interesting, but can that really be expected to ever work with ubports? I don't think UBports even has java, by default, much less implementing these special attachment points:
import android.annotation.SuppressLint; import android.graphics.Point; import android.graphics.Rect; import android.media.MediaCodec; import android.media.MediaCodecInfo; import android.media.MediaFormat; import android.net.LocalServerSocket; import android.net.LocalSocket; import android.net.LocalSocketAddress; import android.os.Build; import android.os.IBinder; import android.os.IInterface; import android.os.RemoteException; import android.os.SystemClock; import android.util.Log; import android.view.InputDevice; import android.view.InputEvent; import android.view.IRotationWatcher; import android.view.KeyCharacterMap; import android.view.KeyEvent; import android.view.MotionEvent; import android.view.Surface; -
In the interest of linking both sides of this reference, I've opened a request-for-comments from the mir group, since they surely know way more about this than me:
-
@osndok said in Reverse Convergence? View & control your phone from computer (like VNC & RDP?):
It's rather interesting, but can [scrcpy] really be expected to ever work with ubports?
I didn’t mean to imply that scrcpy could ever work with Ubuntu Touch, or at least not without major modifications. Rather, I point to scrcpy as an example of (and possible framework for) what could be created for Ubuntu Touch.
For example, in place of the scrcpy server on the device, perhaps some sort of Mir platform/module (as mentioned by @alan_g above) could be employed. And if so, perhaps some of the scrcpy client code could be repurposed to work with such a Mir platform. But, yep, I would suppose that would be a lot of work.
