Qml tests using TestCase

Krille

I'm currently searching for a way to write automated tests for my Qml apps. I've heard about Autopilot but couldn't find a documentation how to use it. However there is the TestCase Component (https://doc.qt.io/qt-5/qml-qttest-testcase.html) for Qml and I've made it working by just switch the qmlscene to qmltestrunner. I've written a dirty script to run tests (run_tests.sh):

#!/bin/bash

# Switch to test mode
sed '3iExec=qmltestrunner' fluffychat.desktop.in -i

# Test
clickable clean build-libs desktop

# Switch back to production mode
sed '3d' fluffychat.desktop.in -i

So now I can write some tests like this one (tst_main.qml):

import QtQuick 2.0
import QtTest 1.2

TestCase {
    name: "MathTests"

    function test_math() {
        compare(2 + 2, 4, "2 + 2 = 4")
    }

    function test_fail() {
        compare(2 + 2, 5, "2 + 2 = 5")
    }
}

which is working fine with this script. But I have two problems:

The script needs to be executed in a graphical way. So I assume that will make problems for GitLab CI...
The current solution is very dirty. It would be awesome to have something like a "clickable test" command. I have tried:

clickable run "qmltestrunner"

But this gives:

Using docker container "clickable/ubuntu-sdk:16.04-armhf-ec0904fa-133f-4754-bb80-2510473d4668"
qmltestrunner: could not find a Qt installation of ''

Maybe we can fix this by installing some packages in the container?

dobey

@Krille said in Qml tests using TestCase:

The script needs to be executed in a graphical way. So I assume that will make problems for GitLab CI...

xvfb-run -a qmltestrunner

@Krille said in Qml tests using TestCase:

Maybe we can fix this by installing some packages in the container?

QT_SELECT=qt5 xvfb-run -a qmltestrunner

Or install the qt5-default package in the container.

Krille

Thanks for your answer. So in summary I have tried the following command:

clickable --arch="amd64" run "xvfb-run -a qmltestrunner"

Now there is no more a graphical problem and no other error. But nothing happens after executing this command except:

Using docker container "clickable/ubuntu-sdk:16.04-amd64-5a95c706-276c-493d-9fff-1507a27ab289"

And then it hangs forever.

Brian said: "ok, running xvfb-run -a qmltestrunner directly in the docker container gives me: "

QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'
Could not initialize GLX
Aborted (core dumped)

UniSuperBox

Hmm... It's possible that you also need to set QT_QPA_PLATFORM=xcb either using export or by putting it before your command.

dobey

@UniSuperBox said in Qml tests using TestCase:

Hmm... It's possible that you also need to set QT_QPA_PLATFORM=xcb either using export or by putting it before your command.

No, that is not the problem. Qt will select that automatically.

dobey

@Krille Ah, sorry. I forgot the extra option one needs to ensure color depth of the virtual display.

Try xvfb-run -a -s '-screen 0 540x960x24' qmltestrunner instead.

Not sure why clickable didn't show you the error and exit, though. Maybe a bug in clickable?

Krille

Okay, so now I've tested:

clickable --arch="amd64" run "xfvb-run -a -s '-screen 0 540x960x24' qmltestrunner"

Leads me to:

bash: xfvb-run: command not found

Maybe we need to install xfvb in the container? Is there a way how I can install this in the container by myself?

UniSuperBox

Sorry... xvfb, not xfvb. It stands for "X Virtual FrameBuffer."

dobey

@Krille Sorry, was a typo. I've edited the post now. It's xvfb-run

Krille

Haha okay. So now:

clickable --arch="amd64" run "xvfb-run -a -s '-screen 0 540x960x24' qmltestrunner"

And again the previous result: Nothing happens anymore I will try to run the command in the container to get an error report...

domubpkm

@Krille Hi. I don't know anything about what you are doing but my eyes see again xfvb above whereas it is spoken about xvfb. Just to be sure it's normal, i prefered to prevent you.

Krille

@domubpkm Oh, sure sorry. But nothing changes...

Krille

Btw I have found that this is propably the same reason why the UI tests (at least) in the messages app are still disabled: https://github.com/ubports/messaging-app/blob/xenial/tests/qml/CMakeLists.txt#L18

dobey

@Krille said in Qml tests using TestCase:

Btw I have found that this is propably the same reason why the UI tests (at least) in the messages app are still disabled: https://github.com/ubports/messaging-app/blob/xenial/tests/qml/CMakeLists.txt#L18

The tests aren't disabled. If xvfb is installed and tests are run, then these tests which depend on xvfb should be run too. If however, you are using crossbuilder then the tests will not be run, because tests do not get run during the build then, as you cannot run arm binaries native on x86.