Issues with clickable gdb / gdbserver

Talkless · 22 Oct 2022, 17:00

Hi,

I've setup clickable inside Ubuntu 22.04 (Virtualbox on Debian Sid) to test PureMaps (3.1.1 tag) on Volla Phone X.

I successfully built (and ran) PureMaps using instructions in https://github.com/rinigus/pure-maps/tree/master/packaging/click#ubuntu-touch-packaging

I've tried to run clickable gdbserver and clickable gdb (https://clickable-ut.dev/en/latest/debugging.html#on-device), but it seems that remote application quits after invoking c command in gdb:

$ clickable gdb --arch arm64 
Starting GDB for "/home/vincas/code/puremaps/pure-maps.git/build/aarch64-linux-gnu/app/install/lib/aarch64-linux-gnu/bin/pure-maps".
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
The target architecture is assumed to be aarch64
Reading symbols from /home/vincas/code/puremaps/pure-maps.git/build/aarch64-linux-gnu/app/install/lib/aarch64-linux-gnu/bin/pure-maps...(no debugging symbols found)...done.
Remote debugging using localhost:3333
warning: Unable to find dynamic linker breakpoint function.
GDB will be unable to debug shared library initializers
and track explicitly loaded dynamic code.
0x0000007fb7fd2cc0 in ?? ()
(gdb) c
Continuing.
warning: Could not load shared library symbols for 40 libraries, e.g. /usr/lib/aarch64-linux-gnu/libtls-padding.so.
Use the "info sharedlibrary" command to see the complete listing.
Do you need "set solib-search-path" or "set sysroot"?
[Inferior 1 (process 11745) exited normally]

Meanwhile in separate clickable gdbserver terminal:

$ clickable gdbserver --arch arm64 
/tmp/clickable/gdbserver: 1 file pushed. 0.5 MB/s (318792 bytes in 0.561s)
SSH is recommended for the "gdbserver" command. If you experience any issues, try again with "--ssh"
vincas@ubuntu-ubports:~/code/puremaps/pure-maps.git$

Tried --system-gdb, --ssh 127.0.0.1:2222 variants - result is the same, I can't debug remotely.

Any hints?

Thanks!

Talkless · 22 Oct 2022, 17:04

It seems it is attached to correct process?

(gdb) info proc
process 13597
warning: target file /proc/13597/cmdline contained unexpected null characters
cmdline = '/opt/click.ubuntu.com/.click/users/phablet/pure-maps.jonnius/lib/aarch64-linux-gnu/bin/pure-maps'
cwd = '/opt/click.ubuntu.com/pure-maps.jonnius/3.1.1'
exe = '/opt/click.ubuntu.com/pure-maps.jonnius/3.1.1/lib/aarch64-linux-gnu/bin/pure-maps'

Talkless · 22 Oct 2022, 17:07

I gent a little bit more info if I run gdbserver via ssh:

$ clickable gdbserver --ssh 127.0.0.1:2222 --arch arm64 --system-gdb
Pseudo-terminal will not be allocated because stdin is not a terminal.
Pseudo-terminal will not be allocated because stdin is not a terminal.
Pseudo-terminal will not be allocated because stdin is not a terminal.
bind [127.0.0.1]:3333: Address already in use
Process /opt/click.ubuntu.com/.click/users/phablet/pure-maps.jonnius/lib/aarch64-linux-gnu/bin/pure-maps created; pid = 15366
Listening on port 3333
Remote debugging from host 127.0.0.1
library "libpq_cust.so" not found
Creating a QMirClientScreen now

Child exited with status 0

jonius · 23 Oct 2022, 18:30

Stupid question: Did you build with --debug? And did you build clean (debug and release build cache are not separated)? And did you install the package with the debug build?

Talkless · 24 Oct 2022, 17:46

@jonius said in Issues with clickable gdb / gdbserver:

Stupid question: Did you build with --debug? And did you build clean (debug and release build cache are not separated)? And did you install the package with the debug build?

I did forgot to use --debug, but it did not help:

clickable build --debug --clean --arch arm64 --skip-review

And I don't see --debug flag for install, so not sure how to install debug version if it's separated :confused_face: .

Talkless · 24 Oct 2022, 17:56

@jonius Maybe I should not use Ubuntu 22.04 VM for development, but older 20.04?

jonius · 30 Oct 2022, 17:41

@talkless the host OS doesn't play a role. You can use any Linux distro that supports docker and python and even Mac and WSL have some limited support.

As build and release builds are not separate, clickable install will just install whatever has been built before, may it be a debug build or not.

I just stumbled upon this line:

bind [127.0.0.1]:3333: Address already in use

Are you using that port for something already? Could you try to choose another one (on both, gdb and gdbserver)?

Talkless · 31 Oct 2022, 13:21

@jonius said in Issues with clickable gdb / gdbserver:

@talkless the host OS doesn't play a role

What I had in mind that maybe there's issues with newer gdb in newer Ubuntu. I did see some asserts of gdb itself while dealing with this debugging issues.

Chaning port did not help.

Maybe I have to rebuild all dependencies with --debug? But it makes no sense, you can attach debugger to any application. You might not find symbols you want because of lack of symbols, optimization, etc, but it should at least run....

Talkless · 31 Oct 2022, 14:06

@jonius said in Issues with clickable gdb / gdbserver:

@talkless the host OS doesn't play a role.

Oh, because clickable works from within docker image, right..?

Talkless · 8 Nov 2022, 18:25

I've built dependencies and app to armhf, to try debugging on my old BQ Aquaris E5. clickable gdb gives me illegal instruction:

$ clickable gdb --arch armhf
Starting GDB for "/home/vincas/code/puremaps/pure-maps.git/build/arm-linux-gnueabihf/app/install/lib/arm-linux-gnueabihf/bin/pure-maps".
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
The target architecture is assumed to be arm
Reading symbols from /home/vincas/code/puremaps/pure-maps.git/build/arm-linux-gnueabihf/app/install/lib/arm-linux-gnueabihf/bin/pure-maps...done.
Remote debugging using localhost:3333
warning: Unable to find dynamic linker breakpoint function.
GDB will be unable to debug shared library initializers
and track explicitly loaded dynamic code.
0xb6fd79c0 in ?? ()
(gdb) c
Continuing.

Program received signal SIGILL, Illegal instruction.
0xb5a3ef48 in ?? ()

The app launches fine if started via Ubuntu Touch app drawer.

jonius · 6 Nov 2022, 21:04

All I can say so far is that I can reproduce the issue with Pure Maps, but not with the CMake app template from Clickable.

jonius · 7 Nov 2022, 19:18

You can set a break point at main and step through it. This way you will find that it exits because the parser fails to parse command line arguments. it returns at line 112:

109	  // command line
110	  CmdLineParser *parser = CmdLineParser::instance();
111	  if (!parser->parse(app->arguments()))
112	    return 0;

jonius · 8 Nov 2022, 18:25

@talkless Could you please try to add libc6-dbg to the dependencies_target before running the gdb command?

jonius · 8 Nov 2022, 20:15

Commenting out the CmdLineParser lines allows you to debug Pure Maps btw.

Talkless · 12 Nov 2022, 16:54

@jonius Have you actually seen these source lines within gdb? Because there's no symbols for me:

Reading symbols from /home/vincas/code/puremaps/pure-maps.git/build/aarch64-linux-gnu/app/install/lib/aarch64-linux-gnu/bin/pure-maps...(no debugging symbols found)...done.

Also, commenting out command line parsing did nothing, and even adding qFatal() did not make app to die in normal run. As if clickable install --arch arm64 just installs some old release build...

Also, clickable install takes 3 minutes to upload app into my Volla Phone X...

Also, clickable launch logs does not show any output, no "logs"...

So far, much pain...

Talkless · 12 Nov 2022, 18:11

It seems I forgot to add --arch arm64 together with --debug, so I was building.. probably desktop version all the time.

Yes, removing CmdLineParser lines allows to keep debugging further.

Does that mean that clickable gdb launches application incorrectly, without some expected arguments that where provided via .desktop file (or whatever) for example?

jonius · 13 Nov 2022, 16:50

@talkless You can run the gdbserver command with --verbose to see the command line parameters. I suspected the --desktop_file_hint which is necessary for the display server. Adding that one as an allowed parameter to the parser doesn't help though. So no idea, to be honest.

Talkless · 27 Nov 2022, 13:05

Oh boy, I've updated clickable (it suggested that there's latest version):

pip3 install --user --upgrade clickable-ut

Updated images:

clickable update-images

Rebuilt everything:

clickable script prepare-deps
clickable build --libs --arch arm64
clickable build --arch arm64 --debug

But it fails to install picotts on final app build step:

Installing libraries
  /home/vincas/code/puremaps/pure-maps.git/build/aarch64-linux-gnu/s2geometry/install/lib/libs2.so
Installing binaries
  /home/vincas/code/puremaps/pure-maps.git/build/aarch64-linux-gnu/mimic/install/bin/mimic
find: '/home/vincas/code/puremaps/pure-maps.git/build/aarch64-linux-gnu/picotts/install/usr/bin/': No such file or directory
Command exited with non-zero exit status 1, see above for details. This is most likely not a problem with Clickable.

Not sure why it install pico if it's full build:

$ file clickable.yaml 
clickable.yaml: symbolic link to packaging/click/full-build.yaml

jonius · 28 Nov 2022, 19:31

Are you sure you have picotts built? It makes sense for it to be in the full build, because it offers more languages than mimic.

I just found another bug in the latest version of Clickable related to installing QML modules. Until it is released, you find the fix here: https://gitlab.com/clickable/clickable/-/merge_requests/553

Talkless · 11 Dec 2022, 14:29

@jonius said in Issues with clickable gdb / gdbserver:

Are you sure you have picotts built?

Picotts seems to be cloned in prepare-deps step, and in build --libs step I see this:

[92mBuilding picotts[0m
[93mCached container is outdated[0m
Sending build context to Docker daemon  3.072kB


Step 1/3 : FROM clickable/amd64-16.04-arm64:16.04.5
 ---> a783c3e92703
Step 2/3 : RUN echo set debconf/frontend Noninteractive | debconf-communicate && echo set debconf/priority critical | debconf-communicate
 ---> Using cache
 ---> e1b83f8da45f
Step 3/3 : RUN apt-get update && apt-get install -y --force-yes --no-install-recommends libtool:arm64 libpopt-dev:arm64 && apt-get clean
 ---> Running in b280d7a9aa37
Get:1 http://ports.ubuntu.com/ubuntu-ports xenial InRelease [247 kB]
Get:2 http://ppa.launchpad.net/bhdouglass/clickable/ubuntu xenial InRelease [15.4 kB]
Get:3 http://archive.ubuntu.com/ubuntu xenial InRelease [247 kB]
Get:4 https://deb.nodesource.com/node_16.x xenial InRelease [4584 B]
Get:5 http://repo.ubports.com xenial InRelease [21.2 kB]
Get:6 http://ppa.launchpad.net/mardy/qbs-on-lts/ubuntu xenial InRelease [15.4 kB]
Get:7 https://deb.nodesource.com/node_16.x xenial/main amd64 Packages [774 B]
Get:8 http://ports.ubuntu.com/ubuntu-ports xenial-updates InRelease [99.8 kB]
Get:9 https://deb.nodesource.com/node_16.x xenial/main arm64 Packages [772 B]
Get:10 http://ppa.launchpad.net/bhdouglass/clickable/ubuntu xenial/main amd64 Packages [784 B]
Get:11 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [99.8 kB]
Get:12 http://ports.ubuntu.com/ubuntu-ports xenial-security InRelease [99.8 kB]
Get:13 http://archive.ubuntu.com/ubuntu xenial-security InRelease [99.8 kB]
Get:14 http://ppa.launchpad.net/bhdouglass/clickable/ubuntu xenial/main arm64 Packages [784 B]
Get:15 http://ppa.launchpad.net/mardy/qbs-on-lts/ubuntu xenial/main amd64 Packages [1125 B]
Get:16 http://repo.ubports.com xenial/main amd64 Packages [2111 kB]
Get:17 http://ppa.launchpad.net/mardy/qbs-on-lts/ubuntu xenial/main arm64 Packages [1129 B]
Get:18 http://ports.ubuntu.com/ubuntu-ports xenial/main arm64 Packages [1466 kB]
Get:19 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages [1558 kB]
Get:20 http://ports.ubuntu.com/ubuntu-ports xenial/multiverse arm64 Packages [146 kB]
Get:21 http://ports.ubuntu.com/ubuntu-ports xenial/universe arm64 Packages [9493 kB]
Get:22 http://archive.ubuntu.com/ubuntu xenial/restricted amd64 Packages [14.1 kB]
Get:23 http://archive.ubuntu.com/ubuntu xenial/multiverse amd64 Packages [176 kB]
Get:24 http://archive.ubuntu.com/ubuntu xenial/universe amd64 Packages [9827 kB]
Get:25 http://repo.ubports.com xenial/main arm64 Packages [2106 kB]
Get:26 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main arm64 Packages [1556 kB]
Get:27 http://ports.ubuntu.com/ubuntu-ports xenial-updates/multiverse arm64 Packages [17.2 kB]
Get:28 http://ports.ubuntu.com/ubuntu-ports xenial-updates/universe arm64 Packages [1466 kB]
Get:29 http://ports.ubuntu.com/ubuntu-ports xenial-security/main arm64 Packages [1118 kB]
Get:30 http://ports.ubuntu.com/ubuntu-ports xenial-security/multiverse arm64 Packages [3332 B]
Get:31 http://ports.ubuntu.com/ubuntu-ports xenial-security/universe arm64 Packages [943 kB]
Get:32 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [2560 kB]
Get:33 http://archive.ubuntu.com/ubuntu xenial-updates/restricted amd64 Packages [16.4 kB]
Get:34 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse amd64 Packages [26.2 kB]
Get:35 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [1544 kB]
Get:36 http://archive.ubuntu.com/ubuntu xenial-security/main amd64 Packages [2051 kB]
Get:37 http://archive.ubuntu.com/ubuntu xenial-security/restricted amd64 Packages [15.9 kB]
Get:38 http://archive.ubuntu.com/ubuntu xenial-security/multiverse amd64 Packages [8820 B]
Get:39 http://archive.ubuntu.com/ubuntu xenial-security/universe amd64 Packages [984 kB]
Fetched 40.2 MB in 4s (8160 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
Suggested packages:
  libtool-doc gfortran | fortran95-compiler gcj-jdk
Recommended packages:
  libltdl-dev
The following NEW packages will be installed:
  libpopt-dev:arm64 libtool
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 232 kB of archives.
After this operation, 1079 kB of additional disk space will be used.
Get:1 http://ports.ubuntu.com/ubuntu-ports xenial/main arm64 libpopt-dev arm64 1.16-10 [38.5 kB]
Get:2 http://archive.ubuntu.com/ubuntu xenial/main amd64 libtool all 2.4.6-0.1 [193 kB]
[91mdebconf: delaying package configuration, since apt-utils is not installed
[0mFetched 232 kB in 0s (815 kB/s)
Selecting previously unselected package libpopt-dev:arm64.
(Reading database ... 
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 54550 files and directories currently installed.)
Preparing to unpack .../libpopt-dev_1.16-10_arm64.deb ...
Unpacking libpopt-dev:arm64 (1.16-10) ...
Selecting previously unselected package libtool.
Preparing to unpack .../libtool_2.4.6-0.1_all.deb ...
Unpacking libtool (2.4.6-0.1) ...
Setting up libpopt-dev:arm64 (1.16-10) ...
Setting up libtool (2.4.6-0.1) ...
[91mW: --force-yes is deprecated, use one of the options starting with --allow instead.
[0mRemoving intermediate container b280d7a9aa37
 ---> 339b027b7540
Successfully built 339b027b7540
Successfully tagged clickable/amd64-16.04-arm64:16.04.5-8a63b05d-ed0b-4ef1-89f6-7d72d35c4d74
make: Nothing to be done for 'all'.
make: 'install' is up to date.
[92mBuilding mimic[0m

Not sure if these final lines are actually OK:

Successfully built 339b027b7540
Successfully tagged clickable/amd64-16.04-arm64:16.04.5-8a63b05d-ed0b-4ef1-89f6-7d72d35c4d74
make: Nothing to be done for 'all'.
make: 'install' is up to date