Reverse Convergence? View & control your phone from computer (like VNC & RDP?)

Osndok · 25 Jun 2018, 17:50

Ordinarily, we think of convergence as your phone being your workstation, when connected to the appropriate hardware; and this is desirable b/c a keyboard & mouse are far more effective than a touch interface... it's just quite bulky to carry around in your pocket. In this case the phone is in-charge, and doing everything.

There are a few problems with this, of course... for one, no matter how good phone processors get, they will never have the speed/memory/graphics of a PC. Another is that your 'workstation' is not useful without your phone... so you have no backup, and anything that occupies your phone (like a call) could interfere with your ability to use your workstation.

At the moment, the only true convergence I feel is on the command line. UBPorts stomps all over the Android shell, and I can open it remotely (ssh) or over usb (adb shell)... scp'ing files to-and-fro. It is truly a world-class app, and (ironically) 1000x more effective and easier to use from my computer than on the phone itself!

And that got me thinking... my phone's shell, in effect, is presented to me as any other shell... like any other SSH session... couldn't (roughly) the same thing be accomplished by just reversing the master/slave convergence relationship?

I already have a fine PC, and I'm already quite used to using virtual-machines (machines inside windows on my PC), so if I could "just plug in my phone and its screen pops up as a window" (like any other VM)... that would happily solve the majority of my "convergence problems".

There are so many little tasks that require "switching to my phone"... looking up a contact, getting a MFA code, changing an alarm time...

It would be so much more effecient if I could just do that from my computer, without having the mode-switch, focus-change, unlock-phone, hands-off-keyboard/mouse, swipe/tap, type-on-the-tiny-onscreen-keyboard, etc.

Then I wouldn't need a 3rd party app (or service!) to "send an SMS from my computer", or any complicated setup to sync my contacts & text messages... or... anything that my phone already can do well. Truly making workflows mobile-first (as anything I can do on my phone, I can then do on my PC).

...and if you think about it... even though the complexity is about the same, this offers you more functionality than classic convergence. It opens the door to copy/paste between your phone & PC... while keeping you used to (and in fact, using) the same apps that are on your phone. I'm so excited at this prospect, I can already imagine even the unity8 live-resize demo (where the interface changes from touch to windowed depending on screen size) only powered by a device I can unplug and take with me!

There are probably several ways to implement this, and I would love to hear of anyone's efforts at even basic remote control.

I've seen the mirscreencast binaries, so I imagine this is at least technically feasible.

I so pine for what I might express as:
ssh -X myphone unity8

Osndok · 25 Jun 2018, 20:05

Just found this:

https://github.com/ycheng/mir-vnc-server

advocatux · 25 Jun 2018, 21:21

@osndok that project doesn't see any action since 2014

Osndok · 25 Jun 2018, 21:33

@advocatux Yes. I noticed that. It is clearly an incomplete proof-of-concept, but it the closest thing I have found to "remote control" thus far.

Osndok · 25 Jun 2018, 22:12

After reading up a bit on libmirclient, it looks like if I could just setup a socket on the phone that forwards to a mir-server running on the computer, that might actually do something without being too traumatic a learning experience.

tera · 26 Jun 2018, 14:02

2 extra cents here: while trying Convergence over wireless again the other days with a Pro5 and a FP2, i noticed that both phones see eachothers in the Display section!
It is not working though it states Connected when trying fyi, not sure if it could be that opening a port or starting a missing process can make it work...

Osndok · 26 Jun 2018, 14:14

@tera said in Reverse Convergence? View & control your phone from computer (like VNC & RDP?):

Convergence over wireless

Is there a place I can find out more about this wireless convergence? This is the first I've heard of it!

Osndok · 26 Jun 2018, 14:18

Oh... I see... there is no control channel. This is the wifi display link, the miracast protocol or a hardware dongle, etc.

Osndok · 26 Jun 2018, 15:00

So this is as far as I got... apparently there is a version mismatch between ubports and upstream mir, so ATM this does not work.

Once: install mir on your PC

sudo dnf install mir-demos qterminal socat

Once: install socat on the phone

adb shell
sudo bash
<enter unlock code/password>
mount -o remount,rw /
apt-get update
apt-get install socat
mount -o remount,ro /

Each time

miral-app
SOCKET=$(ls /run/user/*/miral_socket)

socat UNIX-CONNECT:$SOCKET EXEC:'adb shell socat STDIO UNIX-LISTEN\:/tmp/display2,reuseaddr' &

adb shell MIR_SOCKET=/tmp/display2 unity8-dash

FAILURE

mirserver: Rejected and disconnected a client (Unsupported protocol version)

matteo · 26 Jun 2018, 15:05

I don't know if this is exactly what you are looking for @Osndok...but I feel that your convergence vision it kind of something close to what this guy shows in this post (in particular where he explains about X-forwarding): Kris's blog

Have a look

Osndok · 26 Jun 2018, 15:14

Thanks @matteo ... I am aware of X forwarding, but not how it might be leveraged to actually use any of the touch apps.

matteo · 26 Jun 2018, 15:17

@osndok I personally tested that and it works more or less in the way you can open an app which is sitting into your phone popping it up straight on your desktop computer.

alan_g · 26 Jun 2018, 15:58

This is a nice idea, but there are no pre-existing components you can plug together to make it work.

Here's a way forward...

Mir implements "platforms" as loadable modules, and what you need is a platform that, in addition to the input and display hardware on the phone (which are needed for normal use of the phone) will also allow a program running on a PC to emulate both display and input devices.

With this, the PC program connecting would seem to Unity8 just like connecting an physical display, keyboard and mouse and allow you to exploit the existing "convergence" support.

The "android platform" used by Mir the phone was abandoned by Canonical, so it is entirely under the control of UBports. Adding this proposed "remote display & input" capability is only a matter of programming.

Regarding X forwarding...

As noted above, you can run some X applications on the phone with X forwarding and have them appear on a PC. But these would typically be installed in a chroot or LXD container so this approach doesn't give you access to the phone applications.

It is also possible to use Mir's "Mir-on-X" platform with X forwarding (with a limited number of Mir applications) but even if that could be extended to support the range of applications that Unity8 needs on the phone it would conflict with using the phone as a phone.

Osndok · 26 Jun 2018, 16:32

@alan_g ... If I understand you correctly, this socket-sharing mechanism would then only work for the most trivial case (like an "hello world" example app); because anything that actually tries to interface with an expected phone-like gizmo via mir would not find it.

alan_g · 26 Jun 2018, 16:45

@osndok said in Reverse Convergence? View & control your phone from computer (like VNC & RDP?):

@alan_g ... If I understand you correctly, this socket-sharing mechanism would then only work for the most trivial case (like an "hello world" example app); because anything that actually tries to interface with an expected phone-like gizmo via mir would not find it.

You're asking about X forwarding?

It is feasible to ssh -X myphone, switch to a chroot and then install and run an X11 application e.g. xeyes.

The problem is that doesn't give access to phone apps which run on Unity8, e.g. Contacts.

Osndok · 26 Jun 2018, 17:11

No... I'm not really interested in x11 forwarding (for the same reasons you mention).

Although... originally, I was thinking there would have to be something special on the computer side for this to work (like a mir server), but I wonder how difficult it would be to make an adapter (similar to what you mentioned), that would expose a new mir display over an x11 display protocol.

Sorta like a hacked-down version of miral... or the "opposite" of xmir.

Then you could do something like:

ssh -X myphone phablet-control

... and the 'phablet-control' binary would add a mir platform (as you mention), and present itself as an x11 frame (e.g. the window size is the reported display size, etc.); and then you wouldn't even need anything special on the PC side!

GizmoChicken · 26 Jun 2018, 18:03

@osndok said in Reverse Convergence? View & control your phone from computer (like VNC & RDP?):

if I could "just plug in my phone and its screen pops up as a window" (like any other VM)... that would happily solve the majority of my "convergence problems".

Are you hoping for something that would work with Ubuntu Touch much like Vysor (or similar) works with Android? (Vysor demo video here). If so, I too would like to see something like that.

As for using Vysor with Android, it had some problems when I tried it about a year ago, and I haven’t tried it again recently.

I notice that Genymobile (makers of Genymotion Android emulator) is developing scrcpy (which is similar to Vysor) under what appears to be the Apache License.

The UBports team is busy with 1,000 other projects. But something similar to scrcpy, but for use with Ubuntu Touch, would be a great 1,001st project!

alan_g · 26 Jun 2018, 18:04

@osndok said in Reverse Convergence? View & control your phone from computer (like VNC & RDP?):

Although... originally, I was thinking there would have to be something special on the computer side for this to work (like a mir server), but I wonder how difficult it would be to make an adapter (similar to what you mentioned), that would expose a new mir display over an x11 display protocol.

... and the 'phablet-control' binary would add a mir platform (as you mention), and present itself as an x11 frame (e.g. the window size is the reported display size, etc.); and then you wouldn't even need anything special on the PC side!

I'm not implementing this, but on the face of it it would solve more problems that it creates. E.g. there's no need to define and secure a PC-phone protocol.

Osndok · 26 Jun 2018, 18:28

Yes, @gizmochicken ... something very much like Vysor... but (hopefully) without the huge web browser, and open source.

GizmoChicken · 26 Jun 2018, 18:37

@osndok said in Reverse Convergence? View & control your phone from computer (like VNC & RDP?):

Yes, @gizmochicken ... something very much like Vysor... but (hopefully) without the huge web browser, and open source.

Have a look at the GitHub page for scrcpy. Additional info here and here.