UBports Robot Logo UBports Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    • Register
    • Login

    Reverse Convergence? View & control your phone from computer (like VNC & RDP?)

    Scheduled Pinned Locked Moved Lomiri (was Unity8)
    30 Posts 6 Posters 9.5k Views 3 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
      Reply
      • Reply as topic
      Log in to reply
      This topic has been deleted. Only users with topic management privileges can see it.
      • O Offline
        Osndok
        last edited by Osndok

        Ordinarily, we think of convergence as your phone being your workstation, when connected to the appropriate hardware; and this is desirable b/c a keyboard & mouse are far more effective than a touch interface... it's just quite bulky to carry around in your pocket. In this case the phone is in-charge, and doing everything.

        There are a few problems with this, of course... for one, no matter how good phone processors get, they will never have the speed/memory/graphics of a PC. Another is that your 'workstation' is not useful without your phone... so you have no backup, and anything that occupies your phone (like a call) could interfere with your ability to use your workstation.

        At the moment, the only true convergence I feel is on the command line. UBPorts stomps all over the Android shell, and I can open it remotely (ssh) or over usb (adb shell)... scp'ing files to-and-fro. It is truly a world-class app, and (ironically) 1000x more effective and easier to use from my computer than on the phone itself!

        And that got me thinking... my phone's shell, in effect, is presented to me as any other shell... like any other SSH session... couldn't (roughly) the same thing be accomplished by just reversing the master/slave convergence relationship?

        I already have a fine PC, and I'm already quite used to using virtual-machines (machines inside windows on my PC), so if I could "just plug in my phone and its screen pops up as a window" (like any other VM)... that would happily solve the majority of my "convergence problems".

        There are so many little tasks that require "switching to my phone"... looking up a contact, getting a MFA code, changing an alarm time...

        It would be so much more effecient if I could just do that from my computer, without having the mode-switch, focus-change, unlock-phone, hands-off-keyboard/mouse, swipe/tap, type-on-the-tiny-onscreen-keyboard, etc.

        Then I wouldn't need a 3rd party app (or service!) to "send an SMS from my computer", or any complicated setup to sync my contacts & text messages... or... anything that my phone already can do well. Truly making workflows mobile-first (as anything I can do on my phone, I can then do on my PC).

        ...and if you think about it... even though the complexity is about the same, this offers you more functionality than classic convergence. It opens the door to copy/paste between your phone & PC... while keeping you used to (and in fact, using) the same apps that are on your phone. I'm so excited at this prospect, I can already imagine even the unity8 live-resize demo (where the interface changes from touch to windowed depending on screen size) only powered by a device I can unplug and take with me!

        There are probably several ways to implement this, and I would love to hear of anyone's efforts at even basic remote control.

        I've seen the mirscreencast binaries, so I imagine this is at least technically feasible.

        I so pine for what I might express as:
        ssh -X myphone unity8

        alan_gA G 2 Replies Last reply Reply Quote 3
        • O Offline
          Osndok
          last edited by

          Just found this:

          • https://github.com/ycheng/mir-vnc-server
          advocatuxA 1 Reply Last reply Reply Quote 1
          • advocatuxA Offline
            advocatux @Osndok
            last edited by

            @osndok that project doesn't see any action since 2014

            O 1 Reply Last reply Reply Quote 0
            • O Offline
              Osndok @advocatux
              last edited by

              @advocatux Yes. I noticed that. It is clearly an incomplete proof-of-concept, but it the closest thing I have found to "remote control" thus far.

              1 Reply Last reply Reply Quote 1
              • O Offline
                Osndok
                last edited by Osndok

                After reading up a bit on libmirclient, it looks like if I could just setup a socket on the phone that forwards to a mir-server running on the computer, that might actually do something without being too traumatic a learning experience.

                1 Reply Last reply Reply Quote 0
                • T Offline
                  tera
                  last edited by tera

                  2 extra cents here: while trying Convergence over wireless again the other days with a Pro5 and a FP2, i noticed that both phones see eachothers in the Display section!
                  It is not working though it states Connected when trying fyi, not sure if it could be that opening a port or starting a missing process can make it work...

                  O 2 Replies Last reply Reply Quote 0
                  • O Offline
                    Osndok @tera
                    last edited by Osndok

                    @tera said in Reverse Convergence? View & control your phone from computer (like VNC & RDP?):

                    Convergence over wireless

                    Is there a place I can find out more about this wireless convergence? This is the first I've heard of it!

                    1 Reply Last reply Reply Quote 0
                    • O Offline
                      Osndok @tera
                      last edited by

                      Oh... I see... there is no control channel. This is the wifi display link, the miracast protocol or a hardware dongle, etc.

                      1 Reply Last reply Reply Quote 0
                      • O Offline
                        Osndok
                        last edited by Osndok

                        So this is as far as I got... apparently there is a version mismatch between ubports and upstream mir, so ATM this does not work. ๐Ÿ˜ž

                        Once: install mir on your PC

                        sudo dnf install mir-demos qterminal socat
                        

                        Once: install socat on the phone

                        adb shell
                        sudo bash
                        <enter unlock code/password>
                        mount -o remount,rw /
                        apt-get update
                        apt-get install socat
                        mount -o remount,ro /
                        

                        Each time

                        miral-app
                        SOCKET=$(ls /run/user/*/miral_socket)
                        
                        socat UNIX-CONNECT:$SOCKET EXEC:'adb shell socat STDIO UNIX-LISTEN\:/tmp/display2,reuseaddr' &
                        
                        adb shell MIR_SOCKET=/tmp/display2 unity8-dash
                        

                        FAILURE

                        mirserver: Rejected and disconnected a client (Unsupported protocol version)

                        1 Reply Last reply Reply Quote 0
                        • M Offline
                          matteo
                          last edited by

                          I don't know if this is exactly what you are looking for @Osndok...but I feel that your convergence vision it kind of something close to what this guy shows in this post (in particular where he explains about X-forwarding): Kris's blog

                          Have a look ๐Ÿ˜‰

                          O 1 Reply Last reply Reply Quote 0
                          • O Offline
                            Osndok @matteo
                            last edited by

                            Thanks @matteo ... I am aware of X forwarding, but not how it might be leveraged to actually use any of the touch apps.

                            M 1 Reply Last reply Reply Quote 0
                            • M Offline
                              matteo @Osndok
                              last edited by

                              @osndok I personally tested that and it works more or less in the way you can open an app which is sitting into your phone popping it up straight on your desktop computer.

                              1 Reply Last reply Reply Quote 0
                              • alan_gA Offline
                                alan_g @Osndok
                                last edited by

                                This is a nice idea, but there are no pre-existing components you can plug together to make it work.

                                Here's a way forward...

                                Mir implements "platforms" as loadable modules, and what you need is a platform that, in addition to the input and display hardware on the phone (which are needed for normal use of the phone) will also allow a program running on a PC to emulate both display and input devices.

                                With this, the PC program connecting would seem to Unity8 just like connecting an physical display, keyboard and mouse and allow you to exploit the existing "convergence" support.

                                The "android platform" used by Mir the phone was abandoned by Canonical, so it is entirely under the control of UBports. Adding this proposed "remote display & input" capability is only a matter of programming.


                                Regarding X forwarding...

                                As noted above, you can run some X applications on the phone with X forwarding and have them appear on a PC. But these would typically be installed in a chroot or LXD container so this approach doesn't give you access to the phone applications.

                                It is also possible to use Mir's "Mir-on-X" platform with X forwarding (with a limited number of Mir applications) but even if that could be extended to support the range of applications that Unity8 needs on the phone it would conflict with using the phone as a phone.

                                1 Reply Last reply Reply Quote 3
                                • O Offline
                                  Osndok
                                  last edited by

                                  @alan_g ... If I understand you correctly, this socket-sharing mechanism would then only work for the most trivial case (like an "hello world" example app); because anything that actually tries to interface with an expected phone-like gizmo via mir would not find it.

                                  alan_gA 1 Reply Last reply Reply Quote 0
                                  • alan_gA Offline
                                    alan_g @Osndok
                                    last edited by

                                    @osndok said in Reverse Convergence? View & control your phone from computer (like VNC & RDP?):

                                    @alan_g ... If I understand you correctly, this socket-sharing mechanism would then only work for the most trivial case (like an "hello world" example app); because anything that actually tries to interface with an expected phone-like gizmo via mir would not find it.

                                    You're asking about X forwarding?

                                    It is feasible to ssh -X myphone, switch to a chroot and then install and run an X11 application e.g. xeyes.

                                    The problem is that doesn't give access to phone apps which run on Unity8, e.g. Contacts.

                                    1 Reply Last reply Reply Quote 0
                                    • O Offline
                                      Osndok
                                      last edited by Osndok

                                      No... I'm not really interested in x11 forwarding (for the same reasons you mention).

                                      Although... originally, I was thinking there would have to be something special on the computer side for this to work (like a mir server), but I wonder how difficult it would be to make an adapter (similar to what you mentioned), that would expose a new mir display over an x11 display protocol.

                                      Sorta like a hacked-down version of miral... or the "opposite" of xmir.

                                      Then you could do something like:

                                      ssh -X myphone phablet-control
                                      

                                      ... and the 'phablet-control' binary would add a mir platform (as you mention), and present itself as an x11 frame (e.g. the window size is the reported display size, etc.); and then you wouldn't even need anything special on the PC side!

                                      alan_gA 1 Reply Last reply Reply Quote 0
                                      • G Offline
                                        GizmoChicken @Osndok
                                        last edited by GizmoChicken

                                        @osndok said in Reverse Convergence? View & control your phone from computer (like VNC & RDP?):

                                        if I could "just plug in my phone and its screen pops up as a window" (like any other VM)... that would happily solve the majority of my "convergence problems".

                                        Are you hoping for something that would work with Ubuntu Touch much like Vysor (or similar) works with Android? (Vysor demo video here). If so, I too would like to see something like that.

                                        As for using Vysor with Android, it had some problems when I tried it about a year ago, and I havenโ€™t tried it again recently.

                                        I notice that Genymobile (makers of Genymotion Android emulator) is developing scrcpy (which is similar to Vysor) under what appears to be the Apache License.

                                        The UBports team is busy with 1,000 other projects. But something similar to scrcpy, but for use with Ubuntu Touch, would be a great 1,001st project!

                                        O 1 Reply Last reply Reply Quote 1
                                        • alan_gA Offline
                                          alan_g @Osndok
                                          last edited by

                                          @osndok said in Reverse Convergence? View & control your phone from computer (like VNC & RDP?):

                                          Although... originally, I was thinking there would have to be something special on the computer side for this to work (like a mir server), but I wonder how difficult it would be to make an adapter (similar to what you mentioned), that would expose a new mir display over an x11 display protocol.

                                          ... and the 'phablet-control' binary would add a mir platform (as you mention), and present itself as an x11 frame (e.g. the window size is the reported display size, etc.); and then you wouldn't even need anything special on the PC side!

                                          I'm not implementing this, but on the face of it it would solve more problems that it creates. E.g. there's no need to define and secure a PC-phone protocol.

                                          1 Reply Last reply Reply Quote 1
                                          • O Offline
                                            Osndok @GizmoChicken
                                            last edited by

                                            Yes, @gizmochicken ... something very much like Vysor... but (hopefully) without the huge web browser, and open source.

                                            1 Reply Last reply Reply Quote 1
                                            • G Offline
                                              GizmoChicken
                                              last edited by GizmoChicken

                                              @osndok said in Reverse Convergence? View & control your phone from computer (like VNC & RDP?):

                                              Yes, @gizmochicken ... something very much like Vysor... but (hopefully) without the huge web browser, and open source.

                                              Have a look at the GitHub page for scrcpy. Additional info here and here.

                                              O 1 Reply Last reply Reply Quote 0
                                              • First post
                                                Last post