UBports Robot Logo UBports Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    • Register
    • Login

    Reverse Convergence? View & control your phone from computer (like VNC & RDP?)

    Scheduled Pinned Locked Moved Lomiri (was Unity8)
    30 Posts 6 Posters 9.6k Views 3 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
      Reply
      • Reply as topic
      Log in to reply
      This topic has been deleted. Only users with topic management privileges can see it.
      • O Offline
        Osndok
        last edited by Osndok

        After reading up a bit on libmirclient, it looks like if I could just setup a socket on the phone that forwards to a mir-server running on the computer, that might actually do something without being too traumatic a learning experience.

        1 Reply Last reply Reply Quote 0
        • T Offline
          tera
          last edited by tera

          2 extra cents here: while trying Convergence over wireless again the other days with a Pro5 and a FP2, i noticed that both phones see eachothers in the Display section!
          It is not working though it states Connected when trying fyi, not sure if it could be that opening a port or starting a missing process can make it work...

          O 2 Replies Last reply Reply Quote 0
          • O Offline
            Osndok @tera
            last edited by Osndok

            @tera said in Reverse Convergence? View & control your phone from computer (like VNC & RDP?):

            Convergence over wireless

            Is there a place I can find out more about this wireless convergence? This is the first I've heard of it!

            1 Reply Last reply Reply Quote 0
            • O Offline
              Osndok @tera
              last edited by

              Oh... I see... there is no control channel. This is the wifi display link, the miracast protocol or a hardware dongle, etc.

              1 Reply Last reply Reply Quote 0
              • O Offline
                Osndok
                last edited by Osndok

                So this is as far as I got... apparently there is a version mismatch between ubports and upstream mir, so ATM this does not work. ๐Ÿ˜ž

                Once: install mir on your PC

                sudo dnf install mir-demos qterminal socat
                

                Once: install socat on the phone

                adb shell
                sudo bash
                <enter unlock code/password>
                mount -o remount,rw /
                apt-get update
                apt-get install socat
                mount -o remount,ro /
                

                Each time

                miral-app
                SOCKET=$(ls /run/user/*/miral_socket)
                
                socat UNIX-CONNECT:$SOCKET EXEC:'adb shell socat STDIO UNIX-LISTEN\:/tmp/display2,reuseaddr' &
                
                adb shell MIR_SOCKET=/tmp/display2 unity8-dash
                

                FAILURE

                mirserver: Rejected and disconnected a client (Unsupported protocol version)

                1 Reply Last reply Reply Quote 0
                • M Offline
                  matteo
                  last edited by

                  I don't know if this is exactly what you are looking for @Osndok...but I feel that your convergence vision it kind of something close to what this guy shows in this post (in particular where he explains about X-forwarding): Kris's blog

                  Have a look ๐Ÿ˜‰

                  O 1 Reply Last reply Reply Quote 0
                  • O Offline
                    Osndok @matteo
                    last edited by

                    Thanks @matteo ... I am aware of X forwarding, but not how it might be leveraged to actually use any of the touch apps.

                    M 1 Reply Last reply Reply Quote 0
                    • M Offline
                      matteo @Osndok
                      last edited by

                      @osndok I personally tested that and it works more or less in the way you can open an app which is sitting into your phone popping it up straight on your desktop computer.

                      1 Reply Last reply Reply Quote 0
                      • alan_gA Away
                        alan_g @Osndok
                        last edited by

                        This is a nice idea, but there are no pre-existing components you can plug together to make it work.

                        Here's a way forward...

                        Mir implements "platforms" as loadable modules, and what you need is a platform that, in addition to the input and display hardware on the phone (which are needed for normal use of the phone) will also allow a program running on a PC to emulate both display and input devices.

                        With this, the PC program connecting would seem to Unity8 just like connecting an physical display, keyboard and mouse and allow you to exploit the existing "convergence" support.

                        The "android platform" used by Mir the phone was abandoned by Canonical, so it is entirely under the control of UBports. Adding this proposed "remote display & input" capability is only a matter of programming.


                        Regarding X forwarding...

                        As noted above, you can run some X applications on the phone with X forwarding and have them appear on a PC. But these would typically be installed in a chroot or LXD container so this approach doesn't give you access to the phone applications.

                        It is also possible to use Mir's "Mir-on-X" platform with X forwarding (with a limited number of Mir applications) but even if that could be extended to support the range of applications that Unity8 needs on the phone it would conflict with using the phone as a phone.

                        1 Reply Last reply Reply Quote 3
                        • O Offline
                          Osndok
                          last edited by

                          @alan_g ... If I understand you correctly, this socket-sharing mechanism would then only work for the most trivial case (like an "hello world" example app); because anything that actually tries to interface with an expected phone-like gizmo via mir would not find it.

                          alan_gA 1 Reply Last reply Reply Quote 0
                          • alan_gA Away
                            alan_g @Osndok
                            last edited by

                            @osndok said in Reverse Convergence? View & control your phone from computer (like VNC & RDP?):

                            @alan_g ... If I understand you correctly, this socket-sharing mechanism would then only work for the most trivial case (like an "hello world" example app); because anything that actually tries to interface with an expected phone-like gizmo via mir would not find it.

                            You're asking about X forwarding?

                            It is feasible to ssh -X myphone, switch to a chroot and then install and run an X11 application e.g. xeyes.

                            The problem is that doesn't give access to phone apps which run on Unity8, e.g. Contacts.

                            1 Reply Last reply Reply Quote 0
                            • O Offline
                              Osndok
                              last edited by Osndok

                              No... I'm not really interested in x11 forwarding (for the same reasons you mention).

                              Although... originally, I was thinking there would have to be something special on the computer side for this to work (like a mir server), but I wonder how difficult it would be to make an adapter (similar to what you mentioned), that would expose a new mir display over an x11 display protocol.

                              Sorta like a hacked-down version of miral... or the "opposite" of xmir.

                              Then you could do something like:

                              ssh -X myphone phablet-control
                              

                              ... and the 'phablet-control' binary would add a mir platform (as you mention), and present itself as an x11 frame (e.g. the window size is the reported display size, etc.); and then you wouldn't even need anything special on the PC side!

                              alan_gA 1 Reply Last reply Reply Quote 0
                              • G Offline
                                GizmoChicken @Osndok
                                last edited by GizmoChicken

                                @osndok said in Reverse Convergence? View & control your phone from computer (like VNC & RDP?):

                                if I could "just plug in my phone and its screen pops up as a window" (like any other VM)... that would happily solve the majority of my "convergence problems".

                                Are you hoping for something that would work with Ubuntu Touch much like Vysor (or similar) works with Android? (Vysor demo video here). If so, I too would like to see something like that.

                                As for using Vysor with Android, it had some problems when I tried it about a year ago, and I havenโ€™t tried it again recently.

                                I notice that Genymobile (makers of Genymotion Android emulator) is developing scrcpy (which is similar to Vysor) under what appears to be the Apache License.

                                The UBports team is busy with 1,000 other projects. But something similar to scrcpy, but for use with Ubuntu Touch, would be a great 1,001st project!

                                O 1 Reply Last reply Reply Quote 1
                                • alan_gA Away
                                  alan_g @Osndok
                                  last edited by

                                  @osndok said in Reverse Convergence? View & control your phone from computer (like VNC & RDP?):

                                  Although... originally, I was thinking there would have to be something special on the computer side for this to work (like a mir server), but I wonder how difficult it would be to make an adapter (similar to what you mentioned), that would expose a new mir display over an x11 display protocol.

                                  ... and the 'phablet-control' binary would add a mir platform (as you mention), and present itself as an x11 frame (e.g. the window size is the reported display size, etc.); and then you wouldn't even need anything special on the PC side!

                                  I'm not implementing this, but on the face of it it would solve more problems that it creates. E.g. there's no need to define and secure a PC-phone protocol.

                                  1 Reply Last reply Reply Quote 1
                                  • O Offline
                                    Osndok @GizmoChicken
                                    last edited by

                                    Yes, @gizmochicken ... something very much like Vysor... but (hopefully) without the huge web browser, and open source.

                                    1 Reply Last reply Reply Quote 1
                                    • G Offline
                                      GizmoChicken
                                      last edited by GizmoChicken

                                      @osndok said in Reverse Convergence? View & control your phone from computer (like VNC & RDP?):

                                      Yes, @gizmochicken ... something very much like Vysor... but (hopefully) without the huge web browser, and open source.

                                      Have a look at the GitHub page for scrcpy. Additional info here and here.

                                      O 1 Reply Last reply Reply Quote 0
                                      • O Offline
                                        Osndok @GizmoChicken
                                        last edited by

                                        @gizmochicken ... It's rather interesting, but can that really be expected to ever work with ubports? I don't think UBports even has java, by default, much less implementing these special attachment points:

                                        import android.annotation.SuppressLint;
                                        import android.graphics.Point;
                                        import android.graphics.Rect;
                                        import android.media.MediaCodec;
                                        import android.media.MediaCodecInfo;
                                        import android.media.MediaFormat;
                                        import android.net.LocalServerSocket;
                                        import android.net.LocalSocket;
                                        import android.net.LocalSocketAddress;
                                        import android.os.Build;
                                        import android.os.IBinder;
                                        import android.os.IInterface;
                                        import android.os.RemoteException;
                                        import android.os.SystemClock;
                                        import android.util.Log;
                                        import android.view.InputDevice;
                                        import android.view.InputEvent;
                                        import android.view.IRotationWatcher;
                                        import android.view.KeyCharacterMap;
                                        import android.view.KeyEvent;
                                        import android.view.MotionEvent;
                                        import android.view.Surface;
                                        
                                        G 1 Reply Last reply Reply Quote 0
                                        • O Offline
                                          Osndok
                                          last edited by Osndok

                                          In the interest of linking both sides of this reference, I've opened a request-for-comments from the mir group, since they surely know way more about this than me:

                                          • https://github.com/MirServer/mir/issues/430
                                          1 Reply Last reply Reply Quote 0
                                          • G Offline
                                            GizmoChicken @Osndok
                                            last edited by

                                            @osndok said in Reverse Convergence? View & control your phone from computer (like VNC & RDP?):

                                            It's rather interesting, but can [scrcpy] really be expected to ever work with ubports?

                                            I didnโ€™t mean to imply that scrcpy could ever work with Ubuntu Touch, or at least not without major modifications. Rather, I point to scrcpy as an example of (and possible framework for) what could be created for Ubuntu Touch.

                                            For example, in place of the scrcpy server on the device, perhaps some sort of Mir platform/module (as mentioned by @alan_g above) could be employed. And if so, perhaps some of the scrcpy client code could be repurposed to work with such a Mir platform. But, yep, I would suppose that would be a lot of work.

                                            1 Reply Last reply Reply Quote 0
                                            • G Offline
                                              GizmoChicken
                                              last edited by

                                              @alan_g: In a post on the Ubuntu community forums, you mentioned that you aren't planning to add Wayland remote desktop support to Mir any time soon. Although maybe not soon, do you foresee adding such a feature to Mir at any time in the foreseeable future?

                                              alan_gA 1 Reply Last reply Reply Quote 0
                                              • First post
                                                Last post