Reverse Convergence? View & control your phone from computer (like VNC & RDP?)

Osndok · 26 Jun 2018, 15:14

Thanks @matteo ... I am aware of X forwarding, but not how it might be leveraged to actually use any of the touch apps.

matteo · 26 Jun 2018, 15:17

@osndok I personally tested that and it works more or less in the way you can open an app which is sitting into your phone popping it up straight on your desktop computer.

alan_g · 26 Jun 2018, 15:58

This is a nice idea, but there are no pre-existing components you can plug together to make it work.

Here's a way forward...

Mir implements "platforms" as loadable modules, and what you need is a platform that, in addition to the input and display hardware on the phone (which are needed for normal use of the phone) will also allow a program running on a PC to emulate both display and input devices.

With this, the PC program connecting would seem to Unity8 just like connecting an physical display, keyboard and mouse and allow you to exploit the existing "convergence" support.

The "android platform" used by Mir the phone was abandoned by Canonical, so it is entirely under the control of UBports. Adding this proposed "remote display & input" capability is only a matter of programming.

Regarding X forwarding...

As noted above, you can run some X applications on the phone with X forwarding and have them appear on a PC. But these would typically be installed in a chroot or LXD container so this approach doesn't give you access to the phone applications.

It is also possible to use Mir's "Mir-on-X" platform with X forwarding (with a limited number of Mir applications) but even if that could be extended to support the range of applications that Unity8 needs on the phone it would conflict with using the phone as a phone.

Osndok · 26 Jun 2018, 16:32

@alan_g ... If I understand you correctly, this socket-sharing mechanism would then only work for the most trivial case (like an "hello world" example app); because anything that actually tries to interface with an expected phone-like gizmo via mir would not find it.

alan_g · 26 Jun 2018, 16:45

@osndok said in Reverse Convergence? View & control your phone from computer (like VNC & RDP?):

@alan_g ... If I understand you correctly, this socket-sharing mechanism would then only work for the most trivial case (like an "hello world" example app); because anything that actually tries to interface with an expected phone-like gizmo via mir would not find it.

You're asking about X forwarding?

It is feasible to ssh -X myphone, switch to a chroot and then install and run an X11 application e.g. xeyes.

The problem is that doesn't give access to phone apps which run on Unity8, e.g. Contacts.

Osndok · 26 Jun 2018, 17:11

No... I'm not really interested in x11 forwarding (for the same reasons you mention).

Although... originally, I was thinking there would have to be something special on the computer side for this to work (like a mir server), but I wonder how difficult it would be to make an adapter (similar to what you mentioned), that would expose a new mir display over an x11 display protocol.

Sorta like a hacked-down version of miral... or the "opposite" of xmir.

Then you could do something like:

ssh -X myphone phablet-control

... and the 'phablet-control' binary would add a mir platform (as you mention), and present itself as an x11 frame (e.g. the window size is the reported display size, etc.); and then you wouldn't even need anything special on the PC side!

GizmoChicken · 26 Jun 2018, 18:03

@osndok said in Reverse Convergence? View & control your phone from computer (like VNC & RDP?):

if I could "just plug in my phone and its screen pops up as a window" (like any other VM)... that would happily solve the majority of my "convergence problems".

Are you hoping for something that would work with Ubuntu Touch much like Vysor (or similar) works with Android? (Vysor demo video here). If so, I too would like to see something like that.

As for using Vysor with Android, it had some problems when I tried it about a year ago, and I haven’t tried it again recently.

I notice that Genymobile (makers of Genymotion Android emulator) is developing scrcpy (which is similar to Vysor) under what appears to be the Apache License.

The UBports team is busy with 1,000 other projects. But something similar to scrcpy, but for use with Ubuntu Touch, would be a great 1,001st project!

alan_g · 26 Jun 2018, 18:04

@osndok said in Reverse Convergence? View & control your phone from computer (like VNC & RDP?):

Although... originally, I was thinking there would have to be something special on the computer side for this to work (like a mir server), but I wonder how difficult it would be to make an adapter (similar to what you mentioned), that would expose a new mir display over an x11 display protocol.

... and the 'phablet-control' binary would add a mir platform (as you mention), and present itself as an x11 frame (e.g. the window size is the reported display size, etc.); and then you wouldn't even need anything special on the PC side!

I'm not implementing this, but on the face of it it would solve more problems that it creates. E.g. there's no need to define and secure a PC-phone protocol.

Osndok · 26 Jun 2018, 18:28

Yes, @gizmochicken ... something very much like Vysor... but (hopefully) without the huge web browser, and open source.

GizmoChicken · 26 Jun 2018, 18:37

@osndok said in Reverse Convergence? View & control your phone from computer (like VNC & RDP?):

Yes, @gizmochicken ... something very much like Vysor... but (hopefully) without the huge web browser, and open source.

Have a look at the GitHub page for scrcpy. Additional info here and here.

Osndok · 26 Jun 2018, 18:48

@gizmochicken ... It's rather interesting, but can that really be expected to ever work with ubports? I don't think UBports even has java, by default, much less implementing these special attachment points:

import android.annotation.SuppressLint;
import android.graphics.Point;
import android.graphics.Rect;
import android.media.MediaCodec;
import android.media.MediaCodecInfo;
import android.media.MediaFormat;
import android.net.LocalServerSocket;
import android.net.LocalSocket;
import android.net.LocalSocketAddress;
import android.os.Build;
import android.os.IBinder;
import android.os.IInterface;
import android.os.RemoteException;
import android.os.SystemClock;
import android.util.Log;
import android.view.InputDevice;
import android.view.InputEvent;
import android.view.IRotationWatcher;
import android.view.KeyCharacterMap;
import android.view.KeyEvent;
import android.view.MotionEvent;
import android.view.Surface;

Osndok · 26 Jun 2018, 21:01

In the interest of linking both sides of this reference, I've opened a request-for-comments from the mir group, since they surely know way more about this than me:

https://github.com/MirServer/mir/issues/430

GizmoChicken · 27 Jun 2018, 00:03

@osndok said in Reverse Convergence? View & control your phone from computer (like VNC & RDP?):

It's rather interesting, but can [scrcpy] really be expected to ever work with ubports?

I didn’t mean to imply that scrcpy could ever work with Ubuntu Touch, or at least not without major modifications. Rather, I point to scrcpy as an example of (and possible framework for) what could be created for Ubuntu Touch.

For example, in place of the scrcpy server on the device, perhaps some sort of Mir platform/module (as mentioned by @alan_g above) could be employed. And if so, perhaps some of the scrcpy client code could be repurposed to work with such a Mir platform. But, yep, I would suppose that would be a lot of work.

GizmoChicken · 27 Jun 2018, 05:54

@alan_g: In a post on the Ubuntu community forums, you mentioned that you aren't planning to add Wayland remote desktop support to Mir any time soon. Although maybe not soon, do you foresee adding such a feature to Mir at any time in the foreseeable future?

alan_g · 27 Jun 2018, 08:57

@gizmochicken said in Reverse Convergence? View & control your phone from computer (like VNC & RDP?):

@alan_g: In a post on the Ubuntu community forums, you mentioned that you aren't planning to add Wayland remote desktop support to Mir any time soon. Although maybe not soon, do you foresee adding such a feature to Mir at any time in the foreseeable future?

I can see the value of such a feature to the Mir project: Like any open source project, if someone wants to do the work and contribute it then we would all benefit.

However, the team at Canonical are obliged to prioritise their efforts: there's more value, for example, from implementing support for logind, for X11 applications, for the xdg-shell (stable) extension, or a number of other Wayland extensions (managing decorations, panels etc.).

FWIW Mir is hardly the only Wayland compositor to focus on having "the desktop" working well before planning support for "the remote desktop".

Osndok · 27 Jun 2018, 16:21

I'm happy to find that the mesa-x11 packages are already built for arm, and available to the phone... the actual amount of code that needs to be written for this might be incredibly small!

phablet@ubuntu-phablet:/$ apt-cache search mesa | grep -i x11
mir-platform-graphics-mesa-x4 - Display server for Ubuntu - platform library for X11 Mesa
mir-platform-graphics-mesa-x5 - Display server for Ubuntu - platform library for X11 Mesa
mir-platform-graphics-mesa-x6 - Display server for Ubuntu - platform library for X11 Mesa
mir-platform-graphics-mesa-x7 - Display server for Ubuntu - platform library for X11 Mesa
mir-platform-graphics-mesa-x8 - Display server for Ubuntu - platform library for X11 Mesa
mir-platform-graphics-mesa-x9 - Display server for Ubuntu - platform library for X11 Mesa
mir-platform-graphics-mesa-x10 - Display server for Ubuntu - platform library for X11 Mesa

Osndok · 27 Jun 2018, 16:38

Alan said this, quite relevant, in the github issue:

The phablet-control program does not need to inject a platform into the Unity8 process, it simply connects to the hypothetical, modified "mir-android-platform" using some as-yet-unspecified mechanism (probably a local socket).

Osndok · 27 Jun 2018, 20:25

So here is what I'm currently considering...

For discussion purposes, I'll give them labels:

Plan-A: minimal work, a bit hackish, but fully formed plan

Use the existing screencast api (DisplayName::virt) channel to get the visual contents of the screen over the mir socket
phablet-control project starts as a fork from the mirscreencast (or mirscreencast_to_stream) utility
waypoint: phablet-control causes the screen contents to appear in a 'remote' x11 window (so can't use DRI, xv, or the like)
find or create a protocol (protobufs) for sending key/click/mouse-position
if a magic socket path is found, phablet-control will send the relevant x11 events back to mir-android-platform (thus, phablet-control will work in a degraded view-only state on a phone w/o the mir-android-platform modifications)
mir-android-platform is modified to create-and-listen on a magic socket path, and interpret events therefrom
the window will not be resizable, and will always show what is on the screen
there will be a whole lot of network traffic even when idle, since we are basically pushing an uncompressed video stream through it
use of x11 implies no special app is needed on the PC side

Plan-B: more work, more unknowns, better result

create a fourth display in mir-android-platform: DisplayName::remote (?)
replace this step with the results of bribing Alan to disclose the rest of plan-b
resizing the x11 window gives you more/less space (like tablet/desktop mode)
comparing with the previous screen image might save a lot of bandwidth when idle
use of x11 implies no special app is needed on the PC side
since the phablet-control binary is on the phone, we don't have to worry too much about protocol breakage (its upgraded along with the mir-android-platform)

Plan-C: forget x11, USB is faster & more reliable

convince marius to bundle socat with the default image
mir-android-platform gets a new socket w/ undetermined protocol (maybe SPICE?)
phablet-control now lives on the PC side, and basically communicates with the socket by starting a child process with:
- adb shell socat UNIX-CONNECT:/run/phablet_control STDIO
more bandwidth means happier users
wired means less prone to interference (either way)
lack of x11's network transparency means less flexibility (like controlling your phone you left in the other room)
now we don't need the x11 libraries on the phone side

GizmoChicken · 27 Jun 2018, 21:54

@alan_g said in Reverse Convergence? View & control your phone from computer (like VNC & RDP?):

the team at Canonical are obliged to prioritise their efforts

Thanks much for the reply. Yep, I fully agree that the Mir team has higher priorities, and I'm certainly not suggesting that those priorities be changed in the near term. Rather, I'm just hoping that Wayland remote desktop support eventually appears on the wish list.

I suspect you're already fully aware of this, but I'll just note here that, according to a GNOME wiki page, remote desktop functionality isn't directly implemented in Mutter. Rather, GNOME Remote Desktop and Pipewire (and other applications that want to implement features related to screen casting and/or desktop sharing) can now rely on two APIs that were recently added to libmutter, namely org.gnome.Mutter.RemoteDesktop and org.gnome.Mutter.ScreenCast. More information here. And this seems to be how Fedora plans to bring Wayland remote desktop support to Fedora 29.

Forgive the naiveté of my question, for I know not complexity of what I ask, but I wonder: Would it be feasible to support the above mentioned APIs in Mir?

Osndok · 28 Jun 2018, 14:30

It occurs to me that this might also help those interested in automated app testing.