Simulating a "Ubports" 16.04 on x86 hardware (or VM) - work in progress



  • A.S: This is something like a working document for me and others to join efforts to get the convergence going from the x86 side. I will not have time (and already spent too much of it) working on this for the next 3 weeks since i have some pretty important exams and if i don't stop now it will probalby be very bad. However, maybe some of you might want to try this out a little further and once i come back from university i will find that all of this works ;-P Good Luck to all who try this out and please post any success/faliure (the last one is often even more important ;-)) on this thread!

    Simulating a "Ubports" 16.04 on x86 hardware (or VM)

    What does this mean? Well there are some tablets out there with x86 hardware which desperately need to get hold of the UBports experience, but how to do this?

    Idea

    • Goal: Have a Ubuntu 16.04 with Yunit (atop mir) running snaps & (phone) clicks
    • Path
      • Install Ubuntu 16.04
      • Set up Yunit
      • Set up sth. like 15.04 (armhf) container to run phone apps

    Why is this usefull? This is not really Ubports, however it is somehow what i personally think we can explore and try out to show where the project can head in the mid-term. (In the long term it is essential to migrate the UITK to QQC2-suru or Kirigami or whatever and update the ubuntu-sdk-framework from 15.04.6 which is not available in 16.04 aka migrate the 15.04 click apps to whatever we will be using afterwards). If this works, the pressure behind updating the underlying Ubuntu version is lowered (at least i believe so)

    Roadmap

    1. Set up 16.04 (x86) with Yunit and test basics - ~100% done
      1. Setup 16.04.3 in VM download 16.04 here - works
      2. Install Yunit as described on their blog - works
      3. Test apt install method! - works
      4. Test snaps! - works (some do not and for dash integration of new/removed packages, reboot required)
      5. Test clicks! - fails (cause missing 15.04.6 framework and all clicks fro armhf only AFAIK)
      6. Installing things from software center - works (also gtk+ apps)
    2. Set up 15.04 and test clicks - ~0% done
      1. Setup 15.04 in VM download 15.04 here - todo (which version would be best?! PROBLEM: armhf images not available for download so testing this is hard to compare to LXC (armhf) container)
      2. Try to get clicks running (openstore etc.) - todo (maybe stable-phone-overlay ppa needed?!)
    3. Set up 15.04 (armhf) LXC container on 16.04.3 (x64) and connect both - ~10% done
      1. Use 16.04 with yunit as described in (1) and set up LXD how-to here (update: this way of installing won't work directly since there are no vivid containers @ linuxcontainers.org however there are the sdk images @ https://sdk-images.canonical.com/releases/ ) see also additional ressources at the end of this document - work-in-progress... PROBLEM: cross-architecture container don't work that well, some possible work-arounds i see:
        1. Try qemu ("real" virtualization) in seamless mode to emulate an armhf environment (downside: ressource intensive)
        2. Compile the clicks from the openstore for x86 architectures (32-bit binaries work on 64-bit but not vice versa so 32-bit would suffice) and perhaps use a 15.04-amd64/i386 container instead of armhf
      2. Launch an Ubuntu 15.04 (armhf) container - todo (maybe impossible, might work with sth. like qemu-user-static but has downsides)
      3. Install dependencies and configure the container (internally at first) to enable (phone) clicks as done in (2) - todo
      4. Test if installing clicks (headless) works - todo
      5. Make host hardware and files accessible for Container (some usefull links: 1, 2, 3, 4 a quick and dirty thing would be to use something like mirscreencast just as a proof-of-concept) - todo
      6. Test graphical clicks! - todo
      7. Test audio (in/out) clicks! - todo
      8. Test networking - todo
      9. Play around with phone apps! - todo
    4. Polishing - ~1% done
      1. Write the docu! (best if that gets done while testing) - wip
      2. Gather what is still missing - done, almost everything ;-P
      3. auto-start container on host bootup -todo
      4. write script which does all the installing/configuring on vanilla 16.04 - todo
      5. See how the appstore can be mde to automatically use the container to install clicks (if necessary...) - todo
      6. Make cool videos of this! - todo

    Docu

    This is

    (A) for people to reproduce and work further on the project and

    (B) so we can write a bash script or something to do all necessary steps on a vanilla 16.04 system

    The ways to install Virtualbox/Qemu or anything like this are not explained for now since some may want to try this on an actual device. On an actual device, i would also recommend using isorespin.sh from Ian Morrison (http://linuxiumcomau.blogspot.de/) to respin the iso with the most recent kernel. This often helps with hardware problems from my experience!

    Setting up 16.04.3 (64-bit)

    For doing this maybe it's usefull to move inside a directory, however i'm leaving this out for now since i am not your janitor.

    wget http://releases.ubuntu.com/16.04/ubuntu-16.04.3-desktop-amd64.iso
    # in the script we should check sha256sum or something afterwards, i always do this manually (sorry, i'm a noob ;-P)
    wget http://releases.ubuntu.com/16.04/SHA256SUMS
    sha256sum ubuntu-16.04.3-desktop-amd64.iso
    # compare with what is listed in the SHA256SUMS file
    less SHA256SUMS
    

    Now you can either Respin this iso or directly move to your VM/tablet and install Ubuntu as usual...

    Installing yunit

    As described on the yunit blog:

    wget -qO - https://archive.yunit.io/yunit.gpg.key | sudo apt-key add
    echo 'deb [arch=amd64] http://archive.yunit.io/ubuntu/ xenial main' | sudo tee /etc/apt/sources.list.d/yunit.list
    echo 'deb-src http://archive.yunit.io/ubuntu/ xenial main' | sudo tee --append /etc/apt/sources.list.d/yunit.list
    sudo apt update
    sudo apt upgrade
    sudo apt install yunit-desktop
    

    Setting up LXD and 15.04 (armhf) container - failed attempt

    ATTENTION: This does not work as described in the whole section and is just an "extended log" for all who are interested

    sudo apt install lxd
    newgrp lxd
    lxc remote add images images.linuxcontainers.org
    # there are no vivid images @ linuxcontainers.org, need to do this manually
    lxc image list images:
    lxc launch images:ubuntu/vivid/armhf ubuntu-15.04-armhf
    

    TODO If this works, the container has to be auto-started on host bootup! TODO

    # Testing on the ubuntu 16.04.3
    sudo apt install lxd
    # says already newest version so ommiting creating a new group
    lxc remote add ubuntu-sdk-images sdk-images.canonical.com
    lxc image list ubuntu-sdk-images
    ## Okay this does not work, i wanted to avoid installing more of the sdk than needed on
    ## the host but let's try it as proposed in the second link under additional ressources
    sudo add-apt-repository ppa:ubuntu-sdk-team/tools-development
    sudo apt-get update
    sudo apt install ubuntu-sdk-tools
    # Now listing the available remotes:
    lxc remote list
    # damn, the sdk images have been deleted, so we have to download them manually and install
    # as discussed by stephane graber in the 4th link under additional ressources...
    

    We have 2 tarballs for every container inside (https://sdk-images.canonical.com/releases/vivid/) with the following combinations:

    • amd64 - amd64
    • amd64 - armhf
    • armhf - armhf
    • i386 - armhf
    • i386 - i386

    And i am assuming the first means the host, the latter the container architecture. Since trying this on an amd64 host, i choose amd64 - armhf to download and according to stephane, we can install the metadata tarball (the tiny one) and rootfs tarball (the bigger one) with the following command

    lxc image import <metadata tarball> <rootfs tarball>

    wget https://sdk-images.canonical.com/releases/vivid/ubuntu-sdk-15.04-amd64-armhf-lxd.tar.xz
    wget https://sdk-images.canonical.com/releases/vivid/ubuntu-sdk-15.04-amd64-armhf-root.tar.xz
    lxc image import ubuntu-sdk-15.04-amd64-armhf-lxd.tar.gz ubuntu-sdk-15.04-amd64-armhf-rootfs.tar.gz
    # This should state "Transferring image: xx%" and then "image imported with fingerprint: xyz"
    

    Now we can list our image and then try launching it

    lxc image list

    we see that currently, no alias is set and since i am lazy and don't want to type the fingerprint all the time, we edit the image

    lxc image edit <fingerprint>

    and add the line

      aliases: 15.04
    

    but this does not seem to work right now... (it doesn't update the aliases) strangely, it states the architecture of the container to be x86_64 so this may be the wrong container...

    To prevent this problem, i deleted the container again and imported again attaching the --alias=15.04 flag

    lxc image delete <fingerprint>
    lxc image import ubuntu-sdk-15.04-amd64-armhf-lxd.tar.gz ubuntu-sdk-15.04-amd64-armhf-rootfs.tar.gz --alias=15.04
    

    This works and now we can just use this alias for referring. Let's test it with simply opening a bash shell in the next step...

    It seems, we have to launch the lxc container from the image we just imported:

    lxc launch 15.04 ubuntu-15.04-armhf
    

    However, trying the "armhf-armhf" image will give the error:

    error: The image architecture is incompatible with the target server
    

    Damn, this seems to indicate that the guest container has to have the same architecture as the host :-(

    (alhough i was able to run a i386 image on 64-bits but perhaps) Testing this thesis on Ubuntu 16.04.3 (x64, unity7, x11, qemu-user-static installed) with a xenial-armhf (fails, but with a more complex error code), testing with artful-amd64 (works) testing with artful-armhf (fails)

    Trying again

    Assuming we already installed LXC, now only getting the correct (hopefully?!) images from the old sdk and setting up the static qemu-binaries to emulate armhf...

    sudo apt install qemu-user-static
    wget https://sdk-images.canonical.com/releases/vivid/ubuntu-sdk-15.04-armhf-armhf-lxd.tar.xz
    wget https://sdk-images.canonical.com/releases/vivid/ubuntu-sdk-15.04-armhf-armhf-root.tar.xz
    lxc image import ubuntu-sdk-15.04-armhf-armhf-lxd.tar.gz ubuntu-sdk-15.04-armhf-armhf-root.tar.gz --alias=15.04-armhf
    

    However, qemu-user-static was already installed in the VM system and again we get the error...

    error: The image architecture is incompatible with the target server
    

    Theoretically moving to container to apply changes needed for click

    lxc exec 15.04 /bin/bash
    

    This does however not work currently :-(

    TODO find out what is needed to get this to work. TODO

    Setting up LXD and 15.04 (amd64) container with recompilation of clicks

    LXD

    sudo apt install lxd	
    # instead of apt you can also use snap on snap enabled systems
    newgrp lxd
    

    The remote for the sdk-images seems to have been removed from the LXD config so we get the image manually...

    wget https://sdk-images.canonical.com/releases/vivid/ubuntu-sdk-15.04-amd64-armhf-lxd.tar.xz
    wget https://sdk-images.canonical.com/releases/vivid/ubuntu-sdk-15.04-amd64-armhf-root.tar.xz
    lxc image import ubuntu-sdk-15.04-amd64-armhf-lxd.tar.gz ubuntu-sdk-15.04-amd64-armhf-rootfs.tar.gz --alias=15.04-amd64
    # This should state "Transferring image: xx%" and then "image imported with fingerprint: xyz"
    

    Now we have an image based on the 15.04-amd64-rootfs, we can start a container by using

    lxc launch <image-alias> <container name>
    # in our case
    lxc launch 15.04-amd64 ubuntu-15.04-amd64
    

    Compiling the clicks

    Attention: The openstore in this example compiled and installed successfully on 16.04. However it has some problem with actually running which has probably to do with mir. So not everything may work exactly as described below

    The theoretical workflow once all dependencies needed is:

    • get the source code (git clone <repo.git>)
    • qmake <appname.pro>
    • make install
    • and launch

    So far we have not moved inside the container, you can for example start a bash shell in the container by using

    lxc exec ubuntu-15.04-amd64 /bin/bash
    
    Dependencies

    Beside packages that should be included in the system anyway (build-essential, gcc) a few things may be needed additionally (i will show the error code i got when trying to compile the openstore on a ubuntu 16.04.3 amd64 without it in italics)

    sudo apt install build-essentials git bzr libclick-0.4-dev qt5-qmake qtdeclarative5-dev
    

    Unfortunately, we also need ubuntu-sdk-qmake-extras which however is only provided via PPA for 14.04/16.04 here. TODO: find alternative source for this

    Getting the code

    Most apps (AFAIK) in the openstore have a link to their sources. Move to the sources and clone them to your local folder

    Launchpad

    you navigate to the "code" tab and the necessary command to clone is displayed, something like:

    bzr branch lp:<yourapp>
    # example: uNav navigation app by Marcos Costales
    bzr branch lp:unav
    

    Github

    you click on the green "clone or download" button and copy the link (something like "https://github.com/[...].git") into the following command

    git clone <your link>
    # example: uMatrix messenger app by Larrea Mikel
    git clone https://github.com/LarreaMikel/uMatriks.git
    
    Compiling & Installing

    So since these apps are Qt(QML) based we start with qmake. This is a symlink to qmake-qt4 and/or qmake-qt5 respectively (you can have both simultaneously, however choosing the right one is necessary for the process to work) which are actually located in (/usr/lib/x86_64-linux-gnu/qt<4 or 5>/bin/qmake). You can specifiy which qt version to use by appending a flag to the qmake command:

    qmake -qt=qt<4 or 5> <yourapp.pro>
    # example: openstore by Brian Douglass
    qmake -qt=qt5 openstore.pro
    
    • If "manifest.json is not found" sometime the manifest in the sources has the ending .in, to make this work just use mv manifest.json.in manifest.json inside the source folder

    Once this has finished without being aborted, you can run

    make install
    
    Testing

    You should now be able to test what happens when you execute the binary at (/lib/x86_64-linux-gnu/bin/<yourapp>)

    cd /lib/x86_64-linux-gnu/bin/ 
    ./<yourapp>
    # example: openstore
    ./openstore
    

    Additional Ressources

    Usefull when using a VM

    since most time we just need a shell, it's usefull to easily ssh into the VM. for this purpose, you can go to the settings of the VM (in Virtualbox that is), navigate to network -> NAT (adapter 1), open the additional feature and go to "port forwarding" and add a new rule:

    Name Protocoll HOST-IP HOST-port GUEST-IP GUEST-port
    ssh TCP 3022 22

    In the VM install openssh-server

    sudo apt install openssh-server
    

    and in the host you can simply do

    ssh -p 3022 <user>@127.0.0.1 
    


  • Two more things:

    I just can't stop "wasting" my study breaks....



  • Are you still looking into this? (or hows your study going :) )



  • Last time I heard, Yunit was only running on 64-bit machines. I presume Touch runs on 32-bit processors. As a non-techie with old hardware, this prompts some questions:

    17.10 is very chuggy on my 2GB RAM 32-bit machine. I am sadly going to have to look at alternatives at some point - could Touch be ported to a laptop? [Oddly, as unpolished as it was, I used to quite like using Unity 8.]

    Increasingly phones are going 64-bit... Will Touch be portable onto those devices? Could one Core and Snap Ubuntu onto a 64-bit phone now?



  • @doniks thx for asking, studying is taking up most of my time but once in a while i have a few minutes to test a few things.

    Regarding this topic: I have made little progress that is notable, however i could successfully compile and run a few UT apps on an 16.04 x86-64 system (whitout packaging them as clicks/snaps/debs) but have run into some trouble with mir when doing this with yunit installed (with Unity 7 + X11 it works very well). I have to work up the docs a little in here and maybe next weekend or so i can give you some more usefull info :-)

    @3arn0wl the problem is not that Yunit is not running on 32-bit hardware but that it is not packaged by the Yunit team (or any other team i know) so theoretically for you there is a possibility to just compile all packages inside the official Yunit deb repo for 16.04 but for 32-bit instead of 64-bit. (No coding work necessary, you just have to use the exact same sources and compile it on a PC near you)

    About your tablet: i don't know whether Yunit will be a big improvement for speed since it has not seen many performance enhancements (AFAIK). Can you give more specific info about your device (Manufacturer, Name, ID)?

    64-bit should be no problem for UBports (they/we even got a 64 and 32-bit servers so that both architectures are being build for by the native architecture) and in the end of the canonicel days they even said they will only focus on 64-bit (arm64 for ARM btw) which is why only BQ M10 and Meizu Pro 5 were supposed to receive Ubuntu 16.04



  • @demokrit said in Simulating a "Ubports" 16.04 on x86 hardware (or VM) - work in progress:

    @doniks thx for asking, studying is taking up most of my time but once in a while i have a few minutes to test a few things.

    good for you!

    Regarding this topic: I have made little progress that is notable, however i could successfully compile and run a few UT apps

    oh nice! good for us :)

    maybe I should give it a shot. I was a bit intimidated by the length and complexity of the instructions. do you think it would work on 17.10 on x86 hw? I guess the sdk and mir would be significant challenges.

    on an 16.04 x86-64 system (whitout packaging them as clicks/snaps/debs) but have run into some trouble with mir when doing this with yunit installed (with Unity 7 + X11 it works very well).

    mhhmm ... when you say Unity 7 + X11 you mean .... without mir .... so which UT apps run without mir .... maybe I'm not understanding you right.

    I have to work up the docs a little in here and maybe next weekend or so i can give you some more usefull info :-)

    looking forward ... whenever your study permits



  • Hi @demokrit

    I have a 10 year old MacBook!

    "... to just compile..." how easy is that? What steps would I need to follow?



  • @demokrit Lubuntu will release a 32-bit LTS version in April - could Yunit be compiled on that?



  • @demokrit
    awesome work !



  • @doniks sorry i didn't really have the time to work any further but i think i can at least respind :-)

    maybe I should give it a shot. I was a bit intimidated by the length and complexity of the instructions. do you think it would work on 17.10 on x86 hw? I guess the sdk and mir would be significant challenges.

    I wouldn't really recommend following these instructions yet since it seems to be a path with some serious problems: 1. Using virtualisation for running clicks like the 15.04 armhf container will result in drastical performance loss from what i've read in the meantime and 2. the armhf virtualisation seems to break on a regular basis (qemu-static) and the length is because also failed attempts are included. This is only true for the container stuff above because a PPA with yunit/mir is in the works here so 17.10 should not be a problem. (it's however 64-bit AFAIK so no 32-bit yet)

    mhhmm ... when you say Unity 7 + X11 you mean .... without mir .... so which UT apps run without mir .... maybe I'm not understanding you right.

    I meant that i build some apps that are currently only packaged as clicks for the phone from source in an 16.04.3 x86_64 on unity7 atop X11 which you can read about in detail here and also a short overview which apps seem to work or at least start is being set up here (btw, sorry for the confusing docu style in that repos, at some day i will fuse all those documents which talk about the same stuff into one xD). As of yet i could successfully build and start OpenStore, Clock-App and InstantFX, but i have only tested starting, resizing windows and clicking one or two buttons which is my definition of "working" ;-P. I guess they work since they rely on Qt which also works on X11 not only on mir but i don't really know that stuff.

    Ah one more thing; i tried building locally with Yunit installed and interestingly there seems to be a Problem with mir so i cannot start the OpenStore (the irony is somewhat bugging me). Haven't tried with another App, i will report as soon as there is more news.

    @3arn0wl

    "... to just compile..." how easy is that? What steps would I need to follow?

    There is a guide what to do in the offical yunit repos, however the first line says something about "only working on 17.04" but that is no imminant problem since 17.04 is supported until January and available for 32-bit PCs here.

    Lubuntu will release a 32-bit LTS version in April - could Yunit be compiled on that?

    I am almost 100% sure that compiling on Lubuntu 18.04 will work but i don't know how hard it will be.

    @mike thanks :-)



  • :) Thanks for the pointer, @demokrit



  • @3arn0wl if you try this, please tell us how it went (the good, the bad and the ugly) so we can file a bug report if necessary. However i have asked John Salatas from Yunit whether this should work on 32-bits or other Ubuntu derivatives than 17.04 and he thinks it should work so go ahead and try it. If i find my old laptop + some spare time i will try as well but don't wait for me, i'm having a todo list the size of a mountain ;-D



  • :) Of course, @demokrit. It might be a while before I give it a go too.



  • @demokrit said in Simulating a "Ubports" 16.04 on x86 hardware (or VM) - work in progress:

    mhhmm ... when you say Unity 7 + X11 you mean .... without mir .... so which UT apps run without mir .... maybe I'm not understanding you right.

    I meant that i build some apps that are currently only packaged as clicks for the phone from source in an 16.04.3 x86_64 on unity7 atop X11

    As of yet i could successfully build and start OpenStore, Clock-App and InstantFX

    I'm really surprised! I was quite convinced that a UT "phone" app, ie, an app developed and built against the ubuntu sdk would require mir. But you the way you describe it you have these apps running on 16.04, unity 7, ie, X! This is really blowing my mind.

    I guess they work since they rely on Qt which also works on X11 not only on mir but i don't really know that stuff.

    Seems like. I'm really, really surprised.

    Ah one more thing; i tried building locally with Yunit installed and interestingly there seems to be a Problem with mir so i cannot start the OpenStore (the irony is somewhat bugging me). Haven't tried with another App, i will report as soon as there is more news.

    Strange indeed. Do you have the error still at hand?


Log in to reply
 

Looks like your connection to UBports Forum was lost, please wait while we try to reconnect.