Reverse Convergence? View & control your phone from computer (like VNC & RDP?)

Osndok

So this is as far as I got... apparently there is a version mismatch between ubports and upstream mir, so ATM this does not work.

Once: install mir on your PC

sudo dnf install mir-demos qterminal socat

Once: install socat on the phone

adb shell
sudo bash
<enter unlock code/password>
mount -o remount,rw /
apt-get update
apt-get install socat
mount -o remount,ro /

Each time

miral-app
SOCKET=$(ls /run/user/*/miral_socket)

socat UNIX-CONNECT:$SOCKET EXEC:'adb shell socat STDIO UNIX-LISTEN\:/tmp/display2,reuseaddr' &

adb shell MIR_SOCKET=/tmp/display2 unity8-dash

FAILURE

mirserver: Rejected and disconnected a client (Unsupported protocol version)

matteo

I don't know if this is exactly what you are looking for @Osndok...but I feel that your convergence vision it kind of something close to what this guy shows in this post (in particular where he explains about X-forwarding): Kris's blog

Have a look

Osndok

Thanks @matteo ... I am aware of X forwarding, but not how it might be leveraged to actually use any of the touch apps.

matteo

@osndok I personally tested that and it works more or less in the way you can open an app which is sitting into your phone popping it up straight on your desktop computer.

alan_g

This is a nice idea, but there are no pre-existing components you can plug together to make it work.

Here's a way forward...

Mir implements "platforms" as loadable modules, and what you need is a platform that, in addition to the input and display hardware on the phone (which are needed for normal use of the phone) will also allow a program running on a PC to emulate both display and input devices.

With this, the PC program connecting would seem to Unity8 just like connecting an physical display, keyboard and mouse and allow you to exploit the existing "convergence" support.

The "android platform" used by Mir the phone was abandoned by Canonical, so it is entirely under the control of UBports. Adding this proposed "remote display & input" capability is only a matter of programming.

Regarding X forwarding...

As noted above, you can run some X applications on the phone with X forwarding and have them appear on a PC. But these would typically be installed in a chroot or LXD container so this approach doesn't give you access to the phone applications.

It is also possible to use Mir's "Mir-on-X" platform with X forwarding (with a limited number of Mir applications) but even if that could be extended to support the range of applications that Unity8 needs on the phone it would conflict with using the phone as a phone.

Osndok

@alan_g ... If I understand you correctly, this socket-sharing mechanism would then only work for the most trivial case (like an "hello world" example app); because anything that actually tries to interface with an expected phone-like gizmo via mir would not find it.

alan_g

@osndok said in Reverse Convergence? View & control your phone from computer (like VNC & RDP?):

@alan_g ... If I understand you correctly, this socket-sharing mechanism would then only work for the most trivial case (like an "hello world" example app); because anything that actually tries to interface with an expected phone-like gizmo via mir would not find it.

You're asking about X forwarding?

It is feasible to ssh -X myphone, switch to a chroot and then install and run an X11 application e.g. xeyes.

The problem is that doesn't give access to phone apps which run on Unity8, e.g. Contacts.

Osndok

No... I'm not really interested in x11 forwarding (for the same reasons you mention).

Although... originally, I was thinking there would have to be something special on the computer side for this to work (like a mir server), but I wonder how difficult it would be to make an adapter (similar to what you mentioned), that would expose a new mir display over an x11 display protocol.

Sorta like a hacked-down version of miral... or the "opposite" of xmir.

Then you could do something like:

ssh -X myphone phablet-control

... and the 'phablet-control' binary would add a mir platform (as you mention), and present itself as an x11 frame (e.g. the window size is the reported display size, etc.); and then you wouldn't even need anything special on the PC side!

GizmoChicken

@osndok said in Reverse Convergence? View & control your phone from computer (like VNC & RDP?):

if I could "just plug in my phone and its screen pops up as a window" (like any other VM)... that would happily solve the majority of my "convergence problems".

Are you hoping for something that would work with Ubuntu Touch much like Vysor (or similar) works with Android? (Vysor demo video here). If so, I too would like to see something like that.

As for using Vysor with Android, it had some problems when I tried it about a year ago, and I haven’t tried it again recently.

I notice that Genymobile (makers of Genymotion Android emulator) is developing scrcpy (which is similar to Vysor) under what appears to be the Apache License.

The UBports team is busy with 1,000 other projects. But something similar to scrcpy, but for use with Ubuntu Touch, would be a great 1,001st project!

alan_g

@osndok said in Reverse Convergence? View & control your phone from computer (like VNC & RDP?):

Although... originally, I was thinking there would have to be something special on the computer side for this to work (like a mir server), but I wonder how difficult it would be to make an adapter (similar to what you mentioned), that would expose a new mir display over an x11 display protocol.

... and the 'phablet-control' binary would add a mir platform (as you mention), and present itself as an x11 frame (e.g. the window size is the reported display size, etc.); and then you wouldn't even need anything special on the PC side!

I'm not implementing this, but on the face of it it would solve more problems that it creates. E.g. there's no need to define and secure a PC-phone protocol.

Osndok

Yes, @gizmochicken ... something very much like Vysor... but (hopefully) without the huge web browser, and open source.

GizmoChicken

@osndok said in Reverse Convergence? View & control your phone from computer (like VNC & RDP?):

Yes, @gizmochicken ... something very much like Vysor... but (hopefully) without the huge web browser, and open source.

Have a look at the GitHub page for scrcpy. Additional info here and here.

Osndok

@gizmochicken ... It's rather interesting, but can that really be expected to ever work with ubports? I don't think UBports even has java, by default, much less implementing these special attachment points:

import android.annotation.SuppressLint;
import android.graphics.Point;
import android.graphics.Rect;
import android.media.MediaCodec;
import android.media.MediaCodecInfo;
import android.media.MediaFormat;
import android.net.LocalServerSocket;
import android.net.LocalSocket;
import android.net.LocalSocketAddress;
import android.os.Build;
import android.os.IBinder;
import android.os.IInterface;
import android.os.RemoteException;
import android.os.SystemClock;
import android.util.Log;
import android.view.InputDevice;
import android.view.InputEvent;
import android.view.IRotationWatcher;
import android.view.KeyCharacterMap;
import android.view.KeyEvent;
import android.view.MotionEvent;
import android.view.Surface;

Osndok

In the interest of linking both sides of this reference, I've opened a request-for-comments from the mir group, since they surely know way more about this than me:

https://github.com/MirServer/mir/issues/430

GizmoChicken

@osndok said in Reverse Convergence? View & control your phone from computer (like VNC & RDP?):

It's rather interesting, but can [scrcpy] really be expected to ever work with ubports?

I didn’t mean to imply that scrcpy could ever work with Ubuntu Touch, or at least not without major modifications. Rather, I point to scrcpy as an example of (and possible framework for) what could be created for Ubuntu Touch.

For example, in place of the scrcpy server on the device, perhaps some sort of Mir platform/module (as mentioned by @alan_g above) could be employed. And if so, perhaps some of the scrcpy client code could be repurposed to work with such a Mir platform. But, yep, I would suppose that would be a lot of work.

GizmoChicken

@alan_g: In a post on the Ubuntu community forums, you mentioned that you aren't planning to add Wayland remote desktop support to Mir any time soon. Although maybe not soon, do you foresee adding such a feature to Mir at any time in the foreseeable future?

alan_g

@gizmochicken said in Reverse Convergence? View & control your phone from computer (like VNC & RDP?):

@alan_g: In a post on the Ubuntu community forums, you mentioned that you aren't planning to add Wayland remote desktop support to Mir any time soon. Although maybe not soon, do you foresee adding such a feature to Mir at any time in the foreseeable future?

I can see the value of such a feature to the Mir project: Like any open source project, if someone wants to do the work and contribute it then we would all benefit.

However, the team at Canonical are obliged to prioritise their efforts: there's more value, for example, from implementing support for logind, for X11 applications, for the xdg-shell (stable) extension, or a number of other Wayland extensions (managing decorations, panels etc.).

FWIW Mir is hardly the only Wayland compositor to focus on having "the desktop" working well before planning support for "the remote desktop".

Osndok

I'm happy to find that the mesa-x11 packages are already built for arm, and available to the phone... the actual amount of code that needs to be written for this might be incredibly small!

phablet@ubuntu-phablet:/$ apt-cache search mesa | grep -i x11
mir-platform-graphics-mesa-x4 - Display server for Ubuntu - platform library for X11 Mesa
mir-platform-graphics-mesa-x5 - Display server for Ubuntu - platform library for X11 Mesa
mir-platform-graphics-mesa-x6 - Display server for Ubuntu - platform library for X11 Mesa
mir-platform-graphics-mesa-x7 - Display server for Ubuntu - platform library for X11 Mesa
mir-platform-graphics-mesa-x8 - Display server for Ubuntu - platform library for X11 Mesa
mir-platform-graphics-mesa-x9 - Display server for Ubuntu - platform library for X11 Mesa
mir-platform-graphics-mesa-x10 - Display server for Ubuntu - platform library for X11 Mesa

Osndok

Alan said this, quite relevant, in the github issue:

The phablet-control program does not need to inject a platform into the Unity8 process, it simply connects to the hypothetical, modified "mir-android-platform" using some as-yet-unspecified mechanism (probably a local socket).

Osndok

So here is what I'm currently considering...

For discussion purposes, I'll give them labels:

Plan-A: minimal work, a bit hackish, but fully formed plan

Use the existing screencast api (DisplayName::virt) channel to get the visual contents of the screen over the mir socket
phablet-control project starts as a fork from the mirscreencast (or mirscreencast_to_stream) utility
waypoint: phablet-control causes the screen contents to appear in a 'remote' x11 window (so can't use DRI, xv, or the like)
find or create a protocol (protobufs) for sending key/click/mouse-position
if a magic socket path is found, phablet-control will send the relevant x11 events back to mir-android-platform (thus, phablet-control will work in a degraded view-only state on a phone w/o the mir-android-platform modifications)
mir-android-platform is modified to create-and-listen on a magic socket path, and interpret events therefrom
the window will not be resizable, and will always show what is on the screen
there will be a whole lot of network traffic even when idle, since we are basically pushing an uncompressed video stream through it
use of x11 implies no special app is needed on the PC side

Plan-B: more work, more unknowns, better result

create a fourth display in mir-android-platform: DisplayName::remote (?)
replace this step with the results of bribing Alan to disclose the rest of plan-b
resizing the x11 window gives you more/less space (like tablet/desktop mode)
comparing with the previous screen image might save a lot of bandwidth when idle
use of x11 implies no special app is needed on the PC side
since the phablet-control binary is on the phone, we don't have to worry too much about protocol breakage (its upgraded along with the mir-android-platform)

Plan-C: forget x11, USB is faster & more reliable

convince marius to bundle socat with the default image
mir-android-platform gets a new socket w/ undetermined protocol (maybe SPICE?)
phablet-control now lives on the PC side, and basically communicates with the socket by starting a child process with:
- adb shell socat UNIX-CONNECT:/run/phablet_control STDIO
more bandwidth means happier users
wired means less prone to interference (either way)
lack of x11's network transparency means less flexibility (like controlling your phone you left in the other room)
now we don't need the x11 libraries on the phone side