the team at Canonical are obliged to prioritise their efforts

Thanks much for the reply. Yep, I fully agree that the Mir team has higher priorities, and I'm certainly not suggesting that those priorities be changed in the near term. Rather, I'm just hoping that Wayland remote desktop support eventually appears on the wish list.

I suspect you're already fully aware of this, but I'll just note here that, according to a GNOME wiki page, remote desktop functionality isn't directly implemented in Mutter. Rather, GNOME Remote Desktop and Pipewire (and other applications that want to implement features related to screen casting and/or desktop sharing) can now rely on two APIs that were recently added to libmutter, namely org.gnome.Mutter.RemoteDesktop and org.gnome.Mutter.ScreenCast. More information here. And this seems to be how Fedora plans to bring Wayland remote desktop support to Fedora 29.

Forgive the naiveté of my question, for I know not complexity of what I ask, but I wonder: Would it be feasible to support the above mentioned APIs in Mir?

For discussion purposes, I'll give them labels:

Plan-A: minimal work, a bit hackish, but fully formed plan

• Use the existing screencast api (DisplayName::virt) channel to get the visual contents of the screen over the mir socket
• phablet-control project starts as a fork from the mirscreencast (or mirscreencast_to_stream) utility
• waypoint: phablet-control causes the screen contents to appear in a 'remote' x11 window (so can't use DRI, xv, or the like)
• find or create a protocol (protobufs) for sending key/click/mouse-position
• if a magic socket path is found, phablet-control will send the relevant x11 events back to mir-android-platform (thus, phablet-control will work in a degraded view-only state on a phone w/o the mir-android-platform modifications)
• mir-android-platform is modified to create-and-listen on a magic socket path, and interpret events therefrom
• the window will not be resizable, and will always show what is on the screen
• there will be a whole lot of network traffic even when idle, since we are basically pushing an uncompressed video stream through it
• use of x11 implies no special app is needed on the PC side

Plan-B: more work, more unknowns, better result

• create a fourth display in mir-android-platform: DisplayName::remote (?)
• replace this step with the results of bribing Alan to disclose the rest of plan-b
• resizing the x11 window gives you more/less space (like tablet/desktop mode)
• comparing with the previous screen image might save a lot of bandwidth when idle
• use of x11 implies no special app is needed on the PC side
• since the phablet-control binary is on the phone, we don't have to worry too much about protocol breakage (its upgraded along with the mir-android-platform)

Plan-C: forget x11, USB is faster & more reliable

• convince marius to bundle socat with the default image
• mir-android-platform gets a new socket w/ undetermined protocol (maybe SPICE?)
• phablet-control now lives on the PC side, and basically communicates with the socket by starting a child process with:
  • adb shell socat UNIX-CONNECT:/run/phablet_control STDIO
• more bandwidth means happier users
• wired means less prone to interference (either way)
• lack of x11's network transparency means less flexibility (like controlling your phone you left in the other room)
• now we don't need the x11 libraries on the phone side

The phablet-control program does not need to inject a platform into the Unity8 process, it simply connects to the hypothetical, modified "mir-android-platform" using some as-yet-unspecified mechanism (probably a local socket).

phablet@ubuntu-phablet:/$ apt-cache search mesa | grep -i x11
mir-platform-graphics-mesa-x4 - Display server for Ubuntu - platform library for X11 Mesa
mir-platform-graphics-mesa-x5 - Display server for Ubuntu - platform library for X11 Mesa
mir-platform-graphics-mesa-x6 - Display server for Ubuntu - platform library for X11 Mesa
mir-platform-graphics-mesa-x7 - Display server for Ubuntu - platform library for X11 Mesa
mir-platform-graphics-mesa-x8 - Display server for Ubuntu - platform library for X11 Mesa
mir-platform-graphics-mesa-x9 - Display server for Ubuntu - platform library for X11 Mesa
mir-platform-graphics-mesa-x10 - Display server for Ubuntu - platform library for X11 Mesa

@alan_g: In a post on the Ubuntu community forums, you mentioned that you aren't planning to add Wayland remote desktop support to Mir any time soon. Although maybe not soon, do you foresee adding such a feature to Mir at any time in the foreseeable future?

I can see the value of such a feature to the Mir project: Like any open source project, if someone wants to do the work and contribute it then we would all benefit.

However, the team at Canonical are obliged to prioritise their efforts: there's more value, for example, from implementing support for logind, for X11 applications, for the xdg-shell (stable) extension, or a number of other Wayland extensions (managing decorations, panels etc.).

FWIW Mir is hardly the only Wayland compositor to focus on having "the desktop" working well before planning support for "the remote desktop".

It's rather interesting, but can [scrcpy] really be expected to ever work with ubports?

I didn’t mean to imply that scrcpy could ever work with Ubuntu Touch, or at least not without major modifications. Rather, I point to scrcpy as an example of (and possible framework for) what could be created for Ubuntu Touch.

For example, in place of the scrcpy server on the device, perhaps some sort of Mir platform/module (as mentioned by @alan_g above) could be employed. And if so, perhaps some of the scrcpy client code could be repurposed to work with such a Mir platform. But, yep, I would suppose that would be a lot of work.

• https://github.com/MirServer/mir/issues/430

import android.annotation.SuppressLint;
import android.graphics.Point;
import android.graphics.Rect;
import android.media.MediaCodec;
import android.media.MediaCodecInfo;
import android.media.MediaFormat;
import android.net.LocalServerSocket;
import android.net.LocalSocket;
import android.net.LocalSocketAddress;
import android.os.Build;
import android.os.IBinder;
import android.os.IInterface;
import android.os.RemoteException;
import android.os.SystemClock;
import android.util.Log;
import android.view.InputDevice;
import android.view.InputEvent;
import android.view.IRotationWatcher;
import android.view.KeyCharacterMap;
import android.view.KeyEvent;
import android.view.MotionEvent;
import android.view.Surface;

Yes, @gizmochicken ... something very much like Vysor... but (hopefully) without the huge web browser, and open source.

Have a look at the GitHub page for scrcpy. Additional info here and here.

Although... originally, I was thinking there would have to be something special on the computer side for this to work (like a mir server), but I wonder how difficult it would be to make an adapter (similar to what you mentioned), that would expose a new mir display over an x11 display protocol.

... and the 'phablet-control' binary would add a mir platform (as you mention), and present itself as an x11 frame (e.g. the window size is the reported display size, etc.); and then you wouldn't even need anything special on the PC side!

I'm not implementing this, but on the face of it it would solve more problems that it creates. E.g. there's no need to define and secure a PC-phone protocol.

if I could "just plug in my phone and its screen pops up as a window" (like any other VM)... that would happily solve the majority of my "convergence problems".

Are you hoping for something that would work with Ubuntu Touch much like Vysor (or similar) works with Android? (Vysor demo video here). If so, I too would like to see something like that.

As for using Vysor with Android, it had some problems when I tried it about a year ago, and I haven’t tried it again recently.

I notice that Genymobile (makers of Genymotion Android emulator) is developing scrcpy (which is similar to Vysor) under what appears to be the Apache License.

The UBports team is busy with 1,000 other projects. But something similar to scrcpy, but for use with Ubuntu Touch, would be a great 1,001st project!

Although... originally, I was thinking there would have to be something special on the computer side for this to work (like a mir server), but I wonder how difficult it would be to make an adapter (similar to what you mentioned), that would expose a new mir display over an x11 display protocol.

Sorta like a hacked-down version of miral... or the "opposite" of xmir.

Then you could do something like:

ssh -X myphone phablet-control

... and the 'phablet-control' binary would add a mir platform (as you mention), and present itself as an x11 frame (e.g. the window size is the reported display size, etc.); and then you wouldn't even need anything special on the PC side!

@alan_g ... If I understand you correctly, this socket-sharing mechanism would then only work for the most trivial case (like an "hello world" example app); because anything that actually tries to interface with an expected phone-like gizmo via mir would not find it.

You're asking about X forwarding?

It is feasible to ssh -X myphone, switch to a chroot and then install and run an X11 application e.g. xeyes.

The problem is that doesn't give access to phone apps which run on Unity8, e.g. Contacts.

Here's a way forward...

Mir implements "platforms" as loadable modules, and what you need is a platform that, in addition to the input and display hardware on the phone (which are needed for normal use of the phone) will also allow a program running on a PC to emulate both display and input devices.

With this, the PC program connecting would seem to Unity8 just like connecting an physical display, keyboard and mouse and allow you to exploit the existing "convergence" support.

The "android platform" used by Mir the phone was abandoned by Canonical, so it is entirely under the control of UBports. Adding this proposed "remote display & input" capability is only a matter of programming.

Regarding X forwarding...

As noted above, you can run some X applications on the phone with X forwarding and have them appear on a PC. But these would typically be installed in a chroot or LXD container so this approach doesn't give you access to the phone applications.

It is also possible to use Mir's "Mir-on-X" platform with X forwarding (with a limited number of Mir applications) but even if that could be extended to support the range of applications that Unity8 needs on the phone it would conflict with using the phone as a phone.

Have a look

Once: install mir on your PC

sudo dnf install mir-demos qterminal socat

Once: install socat on the phone

adb shell
sudo bash
<enter unlock code/password>
mount -o remount,rw /
apt-get update
apt-get install socat
mount -o remount,ro /

Each time

miral-app
SOCKET=$(ls /run/user/*/miral_socket)

socat UNIX-CONNECT:$SOCKET EXEC:'adb shell socat STDIO UNIX-LISTEN\:/tmp/display2,reuseaddr' &

adb shell MIR_SOCKET=/tmp/display2 unity8-dash

FAILURE

mirserver: Rejected and disconnected a client (Unsupported protocol version)

Convergence over wireless

Is there a place I can find out more about this wireless convergence? This is the first I've heard of it!