Introducing Miroil

alan_g · 2 Dec 2020, 15:26

Why Miroil

Ubports relies on Mir for a large part of the graphics stack and Mir has been changing in ways that make that problematic. In particular:

Mir 2.x has made some server APIs private and these are used by the UBPorts QtMir and unity-system-compositor (unless that's been renamed to lomiri-system-compositor); and,
The other problem is that some features exposed by the mirclient API are not supported by Mir's Wayland support.

Andrew Koenig's "Fundamental Theorem of Software Engineering" says that "we can solve any problem by introducing an extra level of indirection." Miroil is an extra level of indirection that solves the problem of Lomiri depending on features that Mir no longer offers.

A bit of history

While Canonical was working on Ubuntu Touch there was a plan to update QtMir to remove the direct dependencies on mirserver and migrate some code into the (then separate) MirAL project that would provide the functionality QtMir needed from mirserver. As a first step much of the code that depended on mirserver was moved to a src/platforms/mirserver/miral directory in QtMir. This has been updated and extended over the years, but the intended move of the code out of QtMir into MirAL never happened.

A similar exercise for code in unity-system-compositor was started by @WebDrake and that too stalled before code was moved.

Introducing Miroil

Because of events this year I have a few more vacation days to use by the end of the year than I normally have. So I've decided to start a project to complete the work described above. (Although Miroil lives in the Mir repository it isn't going to be a significant part of my "day job" as Mir development lead.)

The point of Miroil is to provide the support from Mir needed for Ubuntu Touch to move to Mir 2.x and Wayland. This will require further input from downstream before it is fully functional and I'm appealing for any interested developers to help.

Current status of Miroil

I've created a feature branch from Mir "master" that included this new library and APIs and added the work already completed in QtMir.

I've taken a look at the unity-system-compositor work, but that is still incomplete and I don't currently have the time to bring that to a state ready to move into Miroil.

The future of Miroil

For any interested developers, it is possible to work on Miroil on "desktop" and test with QtMir, Lomiri and unity-system-compositor.

While the current work in progress is based on Mir 2.x it is sufficiently isolated from the rest of Mir to make a backport to Mir 1.x a simple "cherry-pick" once it reaches a sufficient level of maturity to be of interest to Lomiri development. (And doing that would simplify Lomiri migration to Mir 2.x.)

My plans

With my "hobby project" hat on, I've still got some (but not much) vacation time left to spend on this, and plan to look into providing a Wayland extension that maps easily to the "trusted prompt session" APIs from mirclient.

With my "Mir team lead" hat on, I'm happy to accept something like Miroil into the Mir source tree to support Lomiri development. And ensuring that it continues to build, pass tests and deploy. But further development will be a very low priority for a small team with other responsibilities.

Miroil needs you!

I won't have time to do everything that needs doing. But I will try to help anyone interested in joining this effort and am trying to provide a template to follow for the different aspects of work needed, in particular, server APIs and Wayland extensions.

[edit]
If anyone has the time to document how to build QtMir on the desktop and how to run the test program the project contains that would reduce the barrier for others interested in helping. (Also the same for unity-system-compositor and Lomiri.)
[end-edit]

References

General advice on developing Mir	https://mir-server.io/doc/getting_involved_in_mir.html
The Miroil PR	https://github.com/MirServer/mir/pull/1820
QtMir	https://github.com/ubports/qtmir
The unity-system-compositor PR	https://github.com/ubports/unity-system-compositor/pull/4

alan_g · 2 Dec 2020, 15:26

Status 2020-12-02

Firstly, thanks for all the interest. I hope to make it easier for others to get involved but haven't achieved that yet. It won't get done in just the bits of time I am able to spare.

Today's the last opportunity I'll have to spend time on miroil until "the holidays" (and I don't yet know how much I'll able to do then), so I'll spend it writing a status report.

Wayland extensions

There hasn't been much change in the miroil code since last week's post, the only thing is that I started drafting a Wayland extension protocol to support the "trusted prompt sessions" used by Lomiri. I've not yet resolved all the issues with the protocol definition and not started the code to implement or use it. However, what I have done might provide a skeleton for anyone wanting to implement additional protocols for Lomiri.

QtMir

QtMir is the key thing to get going with miroil as that will "magically" make Lomiri work with Mir 2.x.

However, it is not ready to use with miroil at present, especially the Mir-2.x based branch I've been working on. There are some dependencies on both mirclient and mirserver that need the "extra level of indirection" to be written before it can be reimplemented in miroil.

I've been thinking of rebasing miroil on Mir-1.8 to make this work a bit easier, but not found the time to do this. (Rebasing should be trivial though.)

AFAIK nobody has yet stepped up and documented a development process for using QtMir and Mir/miroil on the desktop. Although I did get one volunteer. (You know who you are!)

unity-system-compositor

No change here.

alan_g · 10 Dec 2020, 09:17

In the latest Q&A @mariogrip said that he's been working on documenting and improving the process of building the Lomiri stack. Once that is published (following editorial checks by @UniSuperBox) I expect it to include details for building QtMir and, hopefully, also running the test example in that project.

alan_g · 23 Dec 2020, 17:22

Some work-in-progress notes

I've had a little time to document some things about building Mir and QtMir to work on this. I've not yet got all the way to where it is simple, but I'll document where I've got to in the hope that it is useful.

I've set up a new MirOil-for-Lomiri github project for this work. This contains forks of (so far) Mir and QtMir. The ultimate intention is, of course, to merge the changes upstream but this keeps the ongoing work in one place.

The Mir project has the default branch set to a version of Miroil based on Mir 1.8 and that will be the initial focus of work.

The QtMir project has the default branch set to a copy of the "xenial" branch and that will be the initial focus of work.

Where to start from?

I decided that an Ubuntu 16.04 base was the best place to start as everything worked there at one time. But I soon found that there were a lot of QtMir dependencies that are more recent that the 16.04 archive. So I then decided to add the UBports repo.

So, steps to follow along:

Install 16.04 either on a VM or (as I did) as a partition on a development laptop. (What follows is probably not a good idea on your main computer.)
Add the UBports archive

sudo apt-add-repository http://repo.ubports.com/
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 4BD4B4D6DBB583F1
sudo apt update
sudo apt dist-upgrade

This allows you to use apt to install any dependencies.

Building Mir

You’ll need a few development tools installed:

sudo apt install devscripts equivs git

With these installed you can checkout Mir, get the remaining dependencies and build:

git clone --recursive https://github.com/MirOil-for-Lomiri/mir.git
cd mir
sed -i /.*wlcs.*/d debian/control
mk-build-deps -i -s sudo
mkdir build
cd  build
cmake -DMIR_ENABLE_WLCS_TESTS=off ..
make

Then install in /usr/local:

sudo make install
sudo ldconfig

Building QtMir

You’ll need a few development tools installed:

sudo apt install devscripts equivs git

With these installed you can checkout QtMir, get the remaining dependencies and build:

git clone https://github.com/MirOil-for-Lomiri/qtmir.git
cd qtmir
mk-build-deps -i -s sudo
mkdir -p cmake-build-debug
cd cmake-build-debug
cmake -DCMAKE_INSTALL_PREFIX=/usr/local ..
make -j 6 all test

Then install in /usr/local:

sudo make install
sudo ldconfig

Now it should be possible to run a demo program using something like the following:

cd ../demos
QT_QPA_PLATFORM=mirserver qmlscene qml-demo-shell/qml-demo-shell.qml

But this didn't work for me (it failed to find the mirserver plugin) and I've not figured out the next step. Yet...

alan_g · 24 Dec 2020, 12:48

Finally, the QtMir demo runs!

What follows may not be the best way to hack Qt, so suggestions are welcome. Starting from the previous post:

sudo ln -s /usr/local/lib/qt5/plugins/platforms/libqpa-mirserver.so /usr/lib/x86_64-linux-gnu/qt5/plugins/platforms/

Allows Qt to find the platform.

Then, Mir fails to start because "zero sized software cursor image is invalid", so remove the cursor and...

$ MIR_SERVER_CURSOR=null QT_QPA_PLATFORM=mirserver qmlscene qml-demo-shell/qml-demo-shell.qml
...
qtmir.screens: Added Screen with id 1 and geometry QRect(0,0 1280x930)
qtmir.screens: Screen::setMirDisplayBuffer Screen(0x7f48c4541260) 0x7f48c4532118 0x7f48c4532108
qtmir.screens: =======================================
qtmir.screens: Screen(0x7f48c4541260) - id: 1 geometry: QRect(0,0 1280x930) window: 0x0 type: "Unknown" scale: 1
qtmir.screens: =======================================
file:///home/alan/CLionProjects/qtmir/demos/qml-demo-shell/qml-demo-shell.qml:2 module "Unity.Screens" is not installed

[2020-12-24 12:12:40.033503] <information> mirserver: Stopping
qtmir.screens: QtCompositor::stop

Now unityscreensplugin is something QtMir provides and it is built...

find /usr/local/ -name libunityscreensplugin.so
/usr/local/lib/qt5/qml/Unity/Screens/libunityscreensplugin.so

So Let's make that easier to find too:

sudo ln -s /usr/local/lib/qt5/qml/Unity/ /usr/lib/x86_64-linux-gnu/qt5/qml/

And that gets another error:

(process:5405): GLib-GIO-ERROR **: Settings schema 'com.canonical.qtmir' is not installed

Which can be solved:

sudo ln -s /usr/local/share/glib-2.0/schemas/com.canonical.qtmir.gschema.xml /usr/share/glib-2.0/schemas/
sudo glib-compile-schemas /usr/share/glib-2.0/schemas/
MIR_SERVER_CURSOR=null QT_QPA_PLATFORM=mirserver qmlscene qml-demo-shell/qml-demo-shell.qml

mardy · 25 Dec 2020, 21:04

@alan_g said in Introducing Miroil:

What follows may not be the best way to hack Qt, so suggestions are welcome. Starting from the previous post:
sudo ln -s /usr/local/lib/qt5/plugins/platforms/libqpa-mirserver.so /usr/lib/x86_64-linux-gnu/qt5/plugins/platforms/
Allows Qt to find the platform.

You can also try:

export QT_PLUGIN_PATH=/usr/local/lib/qt5/plugins

instead of symlinking the Qt plugins. I believe there's no need appending the other system plugins directory as it should be somehow already embedded in the Qt libs.

Though, if you are using a (virtual) marchine specifically setup for this goal, it might be better to just set the install prefix to the system Qt:

cmake \
    -DCMAKE_INSTALL_PREFIX=/usr/ \
    -DCMAKE_INSTALL_LIBDIR=lib/x86_64-linux-gnu \
    ..

alan_g · 30 Dec 2020, 15:50

@mardy thanks, the environment variable looks like a good option. But even though this is a setup specifically for this development I like to comfort of being able to nuke /usr/local, so I'll continue to install there.

alan_g · 30 Dec 2020, 16:15

Running some clients

I've been doing some experimentation with the setup above.

Firstly, I had to revise the QtMir commandline to make it possible to connect mirclient based clients:

MIR_SERVER_ENABLE_MIRCLIENT= MIR_SERVER_CURSOR=null QT_QPA_PLATFORM=mirserver qmlscene qml-demo-shell/qml-demo-shell.qml

After this I could connect, for example the "egltriangle" example:

~$ mir_demo_client_egltriangle --desktop_file_hint=/usr/share/applications/thunderbird.desktop
Window is on output 1: 96 DPI, scale 1.0x, monitor form factor, 60.00Hz
Window exposed

I also found that I could (somewhat surprisingly) connect s/w rendered Wayland clients:

$ mir_demo_client_wayland --desktop_file_hint=/usr/share/applications/thunderbird.desktop 
Got geometry: (338mm × 245mm)@(0, 0)
Got mode: 1280×930@60000 (flags: 3)
Output scale: 1
Output events done
^CSignal 2 received. Good night.

(The --desktop_file_hint... argument is something that QtMir consumes - it just has to point at a .desktop file that exists on the system.)

Sadly, EGL based Wayland clients crash the server:

$ mir_demo_client_wayland_egl_spinner --desktop_file_hint=/usr/share/applications/thunderbird.desktop
~~~
qtmir.surfaces: MirSurface[0x214cc80,"thunderbird"]::registerView(35079680) after=1
qtmir.surfaces: MirSurface[0x214cc80,"thunderbird"]::setReady()
qtmir.applications: Application["thunderbird"]::setInternalState(state=Running)
qtmir.surfaces: MirSurface[0x214cc80,"thunderbird"]::updateExposure(true)
terminate called after throwing an instance of 'std::logic_error'
  what():  Buffer does not support GL rendering
Aborted (core dumped)

I'm going to try a couple of other QtMir branches to see if I can get a more stable starting point.

NB: So far, I've not made any changes to QtMir, I've just shown how to build Mir and QtMir locally and run one against the other.

branch	what works	what doesn't
`xenial`	mirclient and s/w wayland	egl wayland
`xenial_-_edge_-_wayland`	mirclient and egl wayland	s/w wayland
`xenial_-_edge_-_wayland_-_mir18`	mirclient and s/w wayland and egl wayland	(none)

That's reason enough to base this work on xenial_-_edge_-_wayland_-_mir18, I'll update the miroil branch accordingly.

alan_g · 30 Dec 2020, 18:32

@alan_g said in Introducing Miroil:

NB: So far, I've not made any changes to QtMir, I've just shown how to build Mir and QtMir locally and run one against the other.

I've now made one change. Using miroil::GLBuffer in QtMir in place of the original version from QtMir. But that's all I'll achieve today. (PRs welcome!)

jonius · 6 Jan 2021, 17:25

At building QtMir I run into

/home/miroil/qtmir/src/modules/Unity/Application/mirbuffersgtexture.h:20:30: fatal error: miroil/mirbuffer.h: No such file or directory

the file is installed at /usr/local/include/miroil/miroil/mirbuffer.h. It looks like there is one miroil too much in there. Or is GCC supposed to find it in a sub folder?

alan_g · 6 Jan 2021, 17:34

@jonius I don't have access to my test system just now, but:

pkg-config --cflags miroil

Should return -I/usr/include/miroil (amongst others), and that should be picked up by cmake when run in the QtMir directory.

jonius · 6 Jan 2021, 18:55

@alan_g pkg-config lists the include dir correctly (/usr/local/include/miroil). I also printed the include dirs in CMake and the directory was there. Running make with VERBOSE=1 shows that it does not get passed to the compiler for some reason:

cd /home/miroil/qtmir/cmake-build-debug/tests/modules/SurfaceManager && /usr/bin/c++   -DGTEST_VERSION_MAJOR=1 -DGTEST_VERSION_MINOR=7 -DGTEST_VERSION_PATCH=0 -DQT_CORE_LIB -DQT_DBUS_LIB -DQT_DISABLE_DEPRECATED_BEFORE=0x050900 -DQT_GUI_LIB -DQT_NETWORK_LIB -DQT_NO_DEBUG -DQT_NO_KEYWORDS -DQT_QML_LIB -DQT_QUICK_LIB -DQT_SENSORS_LIB -DQT_TESTCASE_BUILDDIR=\"/home/miroil/qtmir/cmake-build-debug\" -DQT_TESTLIB_LIB -DQT_USE_QSTRINGBUILDER -DQT_USING_OPENGL -I/home/miroil/qtmir/cmake-build-debug/tests/modules/SurfaceManager -I/home/miroil/qtmir/tests/modules/SurfaceManager -isystem /usr/include/x86_64-linux-gnu/qt5/QtCore -isystem /usr/include/x86_64-linux-gnu/qt5 -isystem /usr/include/libdrm -I/home/miroil/qtmir/tests/include -I/home/miroil/qtmir/src/common -I/home/miroil/qtmir/src/platforms/mirserver -I/home/miroil/qtmir/src/modules -I/home/miroil/qtmir/tests/framework -isystem /usr/local/include/miral -isystem /usr/local/include/mirclient -isystem /usr/local/include/mircookie -isystem /usr/local/include/mircore -isystem /usr/local/include/mirtest -isystem /usr/local/include/mirserver -isystem /usr/local/include/mirplatform -isystem /usr/local/include/mircommon -isystem /usr/local/include/mirrenderer -isystem /usr/include/uuid -isystem /usr/lib/x86_64-linux-gnu/qt5/mkspecs/linux-g++ -isystem /usr/include/x86_64-linux-gnu/qt5/QtDBus -isystem /usr/include/x86_64-linux-gnu/qt5/QtQml -isystem /usr/include/x86_64-linux-gnu/qt5/QtNetwork -isystem /usr/include/x86_64-linux-gnu/qt5/QtQuick -isystem /usr/include/x86_64-linux-gnu/qt5/QtGui -isystem /usr/include/x86_64-linux-gnu/qt5/QtFontDatabaseSupport/5.12.9 -isystem /usr/include/x86_64-linux-gnu/qt5/QtThemeSupport/5.12.9 -isystem /usr/include/x86_64-linux-gnu/qt5/QtEventDispatcherSupport/5.12.9 -isystem /usr/include/x86_64-linux-gnu/qt5/QtEglSupport/5.12.9 -isystem /usr/include/x86_64-linux-gnu/qt5/QtServiceSupport/5.12.9 -isystem /usr/include/x86_64-linux-gnu/qt5/QtSensors -isystem /usr/include/x86_64-linux-gnu/qt5/QtTest  -fPIC -Wall -fno-strict-aliasing -Werror -Wextra -O2 -g -DNDEBUG   -fPIC -DQT_GUI_LIB -DQT_CORE_LIB -std=gnu++14 -o CMakeFiles/surfacemanager_test.dir/surface_manager_test.cpp.o -c /home/miroil/qtmir/tests/modules/SurfaceManager/surface_manager_test.cpp

jonius · 1 Jun 2021, 19:34

I can workaround the issue by adding the include directories to the target directly (see Commit). Weird issue...

jonius · 1 Jun 2021, 20:36

@jonius next issue is a failing test (ApplicationManager), which fails here.

The demo runs into a segfault as well. (MIR_SERVER_CURSOR=null QT_QPA_PLATFORM=mirserver qmlscene qml-demo-shell/qml-demo-shell.qml).

btw, I am on Ubuntu 16.04 in a VM.

alan_g · 7 Jan 2021, 12:22

@jonius Not sure why you're seeing different things. Maybe something with the graphics setup in your VM?

Can you run the mir examples? For example:

$ mir-smoke-test-runner
...
I: Smoke testing complete with returncode 0

If that also shows problems, this solution might help (depending on your choice of VM):

https://discourse.ubuntu.com/t/problems-running-electron-app-in-mir-kiosk-on-kvm/16196/6

jonius · 7 Jan 2021, 19:17

@alan_g said in Introducing Miroil:

mir-smoke-test-runner

I: The following clients failed to execute successfully:
I:     mir_demo_client_prerendered_frames
I:     mir_demo_client_animated_cursor
I:     mir_demo_client_render_surface
I:     mir_demo_client_basic
I:     mir_demo_client_flicker
I:     mir_demo_client_fingerpaint
I:     mir_demo_client_release_at_exit
I:     mir_demo_client_eglstateswitcher
I:     mir_demo_client_wayland
I:     mir_demo_client_target
I:     mir_demo_client_tooltip
I:     mir_demo_client_eglsquare
I:     mir_demo_client_cursors
I:     mir_demo_client_egldiamond_render_surface
I:     mir_demo_client_egltriangle
I:     mir_demo_client_chain_jumping_buffers
I:     mir_demo_client_touch_validator
I:     mir_demo_client_wayland_egl_spinner
I:     mir_demo_client_progressbar
I:     mir_demo_client_camera
I:     mir_demo_client_prompt_session
I:     mir_demo_client_multiwin
I:     mir_demo_client_scroll
I:     mir_demo_client_display_config
I:     mir_demo_client_screencast
I:     mir_demo_client_eglcounter
I:     mir_demo_client_multistream
I:     mir_demo_client_input_shapes
I:     mir_demo_client_eglplasma
I:     mir_demo_client_eglflash
I:     mir_demo_client_pointer_confinement
I: Smoke testing complete with returncode -1

I am using VirtualBox. I am not sure how to apply the suggestions in the link. Which VM solution would you suggest? Otherwise I'll install the system again on a separate partition without VM.

alan_g · 7 Jan 2021, 20:26

@jonius I don't have VirtualBox to hand to give detailed instructions. But it should work if you enable "the right" graphics options. Not sure what it's called. Something like 'passthrough' or '3D' maybe?

jonius · 1 Aug 2021, 19:00

@alan_g
I was hit by this bug. Now with 3D acceleration enabled most of the smoke tests succeed. Only mir_demo_client_eglplasma still failes. And the unit test I mentioned above continues to fail as well. Will test on separate installation without VM this weekend.

MIR_SERVER_CURSOR=null QT_QPA_PLATFORM=mirserver qmlscene qml-demo-shell/qml-demo-shell.qml works partially. The circle rotates, but the mouse cursor is not visible and clicking the scale buttons results in file:///home/miroil/qtmir/demos/qml-demo-shell/qml-demo-shell.qml:28: Error: Unknown method parameter type: FormFactor.

alan_g · 8 Jan 2021, 23:53

@jonius said in Introducing Miroil:

the mouse cursor is not visible

That's sort of expected. I disabled the cursor in Mir because QtMir tries to set a zero sized image instead of hiding it. And Mir treats that as an error.

I've not looked at how it works for Lomiri. I suspect it renders its own cursor.

jonius · 9 Jan 2021, 23:48

@alan_g installed Ubuntu 16.04 on my notebook directly via Ubuntu Touch (ISODrive is a great tool). Unfortunately the situation here is worse than in my VM on an Ubuntu 20.04 host. I guess the outdated amdgpu for my Radeon Vega 8 Graphics in Ubuntu 16.04 is to blame. The demo runs like in the VM, the same unit test fails, but many smoke tests fail:

I:     mir_demo_client_camera
I:     mir_demo_client_tooltip
I:     mir_demo_client_eglsquare
I:     mir_demo_client_eglplasma
I:     mir_demo_client_eglstateswitcher
I:     mir_demo_client_target
I:     mir_demo_client_scroll
I:     mir_demo_client_cursors
I:     mir_demo_client_egltriangle
I:     mir_demo_client_egldiamond_render_surface
I:     mir_demo_client_display_config
I:     mir_demo_client_eglcounter
I:     mir_demo_client_touch_validator
I:     mir_demo_client_animated_cursor
I:     mir_demo_client_pointer_confinement
I:     mir_demo_client_input_shapes
I:     mir_demo_client_eglflash

So I guess best is to switch back to my VM? Is the failing tests something I could ignore for now?