Reverse Convergence? View & control your phone from computer (like VNC & RDP?)

Osndok

@gizmochicken ... It's rather interesting, but can that really be expected to ever work with ubports? I don't think UBports even has java, by default, much less implementing these special attachment points:

import android.annotation.SuppressLint;
import android.graphics.Point;
import android.graphics.Rect;
import android.media.MediaCodec;
import android.media.MediaCodecInfo;
import android.media.MediaFormat;
import android.net.LocalServerSocket;
import android.net.LocalSocket;
import android.net.LocalSocketAddress;
import android.os.Build;
import android.os.IBinder;
import android.os.IInterface;
import android.os.RemoteException;
import android.os.SystemClock;
import android.util.Log;
import android.view.InputDevice;
import android.view.InputEvent;
import android.view.IRotationWatcher;
import android.view.KeyCharacterMap;
import android.view.KeyEvent;
import android.view.MotionEvent;
import android.view.Surface;

Osndok

In the interest of linking both sides of this reference, I've opened a request-for-comments from the mir group, since they surely know way more about this than me:

https://github.com/MirServer/mir/issues/430

GizmoChicken

@osndok said in Reverse Convergence? View & control your phone from computer (like VNC & RDP?):

It's rather interesting, but can [scrcpy] really be expected to ever work with ubports?

I didn’t mean to imply that scrcpy could ever work with Ubuntu Touch, or at least not without major modifications. Rather, I point to scrcpy as an example of (and possible framework for) what could be created for Ubuntu Touch.

For example, in place of the scrcpy server on the device, perhaps some sort of Mir platform/module (as mentioned by @alan_g above) could be employed. And if so, perhaps some of the scrcpy client code could be repurposed to work with such a Mir platform. But, yep, I would suppose that would be a lot of work.

GizmoChicken

@alan_g: In a post on the Ubuntu community forums, you mentioned that you aren't planning to add Wayland remote desktop support to Mir any time soon. Although maybe not soon, do you foresee adding such a feature to Mir at any time in the foreseeable future?

alan_g

@gizmochicken said in Reverse Convergence? View & control your phone from computer (like VNC & RDP?):

@alan_g: In a post on the Ubuntu community forums, you mentioned that you aren't planning to add Wayland remote desktop support to Mir any time soon. Although maybe not soon, do you foresee adding such a feature to Mir at any time in the foreseeable future?

I can see the value of such a feature to the Mir project: Like any open source project, if someone wants to do the work and contribute it then we would all benefit.

However, the team at Canonical are obliged to prioritise their efforts: there's more value, for example, from implementing support for logind, for X11 applications, for the xdg-shell (stable) extension, or a number of other Wayland extensions (managing decorations, panels etc.).

FWIW Mir is hardly the only Wayland compositor to focus on having "the desktop" working well before planning support for "the remote desktop".

Osndok

I'm happy to find that the mesa-x11 packages are already built for arm, and available to the phone... the actual amount of code that needs to be written for this might be incredibly small!

phablet@ubuntu-phablet:/$ apt-cache search mesa | grep -i x11
mir-platform-graphics-mesa-x4 - Display server for Ubuntu - platform library for X11 Mesa
mir-platform-graphics-mesa-x5 - Display server for Ubuntu - platform library for X11 Mesa
mir-platform-graphics-mesa-x6 - Display server for Ubuntu - platform library for X11 Mesa
mir-platform-graphics-mesa-x7 - Display server for Ubuntu - platform library for X11 Mesa
mir-platform-graphics-mesa-x8 - Display server for Ubuntu - platform library for X11 Mesa
mir-platform-graphics-mesa-x9 - Display server for Ubuntu - platform library for X11 Mesa
mir-platform-graphics-mesa-x10 - Display server for Ubuntu - platform library for X11 Mesa

Osndok

Alan said this, quite relevant, in the github issue:

The phablet-control program does not need to inject a platform into the Unity8 process, it simply connects to the hypothetical, modified "mir-android-platform" using some as-yet-unspecified mechanism (probably a local socket).

Osndok

So here is what I'm currently considering...

For discussion purposes, I'll give them labels:

Plan-A: minimal work, a bit hackish, but fully formed plan

Use the existing screencast api (DisplayName::virt) channel to get the visual contents of the screen over the mir socket
phablet-control project starts as a fork from the mirscreencast (or mirscreencast_to_stream) utility
waypoint: phablet-control causes the screen contents to appear in a 'remote' x11 window (so can't use DRI, xv, or the like)
find or create a protocol (protobufs) for sending key/click/mouse-position
if a magic socket path is found, phablet-control will send the relevant x11 events back to mir-android-platform (thus, phablet-control will work in a degraded view-only state on a phone w/o the mir-android-platform modifications)
mir-android-platform is modified to create-and-listen on a magic socket path, and interpret events therefrom
the window will not be resizable, and will always show what is on the screen
there will be a whole lot of network traffic even when idle, since we are basically pushing an uncompressed video stream through it
use of x11 implies no special app is needed on the PC side

Plan-B: more work, more unknowns, better result

create a fourth display in mir-android-platform: DisplayName::remote (?)
replace this step with the results of bribing Alan to disclose the rest of plan-b
resizing the x11 window gives you more/less space (like tablet/desktop mode)
comparing with the previous screen image might save a lot of bandwidth when idle
use of x11 implies no special app is needed on the PC side
since the phablet-control binary is on the phone, we don't have to worry too much about protocol breakage (its upgraded along with the mir-android-platform)

Plan-C: forget x11, USB is faster & more reliable

convince marius to bundle socat with the default image
mir-android-platform gets a new socket w/ undetermined protocol (maybe SPICE?)
phablet-control now lives on the PC side, and basically communicates with the socket by starting a child process with:
- adb shell socat UNIX-CONNECT:/run/phablet_control STDIO
more bandwidth means happier users
wired means less prone to interference (either way)
lack of x11's network transparency means less flexibility (like controlling your phone you left in the other room)
now we don't need the x11 libraries on the phone side

GizmoChicken

@alan_g said in Reverse Convergence? View & control your phone from computer (like VNC & RDP?):

the team at Canonical are obliged to prioritise their efforts

Thanks much for the reply. Yep, I fully agree that the Mir team has higher priorities, and I'm certainly not suggesting that those priorities be changed in the near term. Rather, I'm just hoping that Wayland remote desktop support eventually appears on the wish list.

I suspect you're already fully aware of this, but I'll just note here that, according to a GNOME wiki page, remote desktop functionality isn't directly implemented in Mutter. Rather, GNOME Remote Desktop and Pipewire (and other applications that want to implement features related to screen casting and/or desktop sharing) can now rely on two APIs that were recently added to libmutter, namely org.gnome.Mutter.RemoteDesktop and org.gnome.Mutter.ScreenCast. More information here. And this seems to be how Fedora plans to bring Wayland remote desktop support to Fedora 29.

Forgive the naiveté of my question, for I know not complexity of what I ask, but I wonder: Would it be feasible to support the above mentioned APIs in Mir?

Osndok

It occurs to me that this might also help those interested in automated app testing.