GPS positioning stops working after a random duration

gpatel-fr

I never used GPS a lot , but now that you are speaking of it, I'm beginning to vaguely remember that there could be one or 2 times when GPS stopped mysteriously indeed.

I'd say that with a process calling back into another, the first process locking everything just because one time it did not get a reply from the client layer would be a fragile design, maybe Google has done this but this does not look like something people designing a system for billions of users would do.
If there is some retry mechanism, restarting the UT service could be of help.

slowcyclist

@gpatel-fr Hey, you were right! By merely restarting the service, positioning becomes available again (no need to reboot!), and adding Restart=on-failure in the service file does the job automatically.
Alas, it's not really a fix for the problem : all the applications that were using the gps at the time of the service failure/restart quit as soon as you try to get back to them. Restarting PureMaps is much faster than rebooting, but you still need to acquire a new gps fix...

I've seen other errors than segfaults in the same piece of code :

Mar 03 15:01:50 ubuntu-phablet lomiri-location-serviced-wrapper[6592]: malloc(): unaligned fastbin chunk detected
Mar 03 15:01:50 ubuntu-phablet systemd[1]: lomiri-location-service.service: Main process exited, code=killed, status=6/ABRT
Mar 03 15:01:50 ubuntu-phablet systemd[1]: lomiri-location-service.service: Failed with result 'signal'.

Mar 03 15:08:19 ubuntu-phablet lomiri-location-serviced-wrapper[7546]: malloc_consolidate(): unaligned fastbin chunk detected
Mar 03 15:08:19 ubuntu-phablet systemd[1]: lomiri-location-service.service: Main process exited, code=killed, status=6/ABRT
Mar 03 15:08:19 ubuntu-phablet systemd[1]: lomiri-location-service.service: Failed with result 'signal'.

It seems this daemon's code did not evolve much lately. It was working well on my Xperia X which was armhf while now I have random alignment errors or segfaults on the FP5 which is arm64. But if the architecture is the reason, it should likely affect all arm64 phones running UBports, unless compilations options/checks are set differently on the FP5...

I guess I'll try to rebuild this piece of code with debug symbols, but then guidance for running it with gdb would be welcome.

gpatel-fr

@slowcyclist

I don't recall ever having things like that in my logs. I am not running the same version as you, I stick to stable. Not sure if it could make a difference since AFAIK your daily version was the same as the 24.04-1.2 stable 12 days ago, I take it that the problem is older than that for you ? I never tried the daily, do you really upgrade your phone every day ?

WillemHexspoor

@slowcyclist You're not the only one... I'm having the exact same issue, also on FP5. I'm running 24.04-1.2 stable

gpatel-fr

@WillemHexspoor said:

I'm having the exact same issue, also on FP5. I'm running 24.04-1.2 stable

hum, do you have only the same symptoms, or do you also have segfaults and unaligned errors in your logs ?

I have the same phone with the exact same version, I may have had the same symptoms (not sure), but definitely not the unaligned errors, at least for the 5 weeks worth of journal data that I have on my phone. @slowcyclist don't use the same version of UT I think, it's daily instead of stable.

WillemHexspoor

@gpatel-fr

My apologies, I replied to the initial post of @slowcyclist, and while re-reading my own post it is indeed not very clear.
I'm having the both mallooc and segfaults. Especially the segfaults result into freezing navigation apps.

Mar 07 09:42:25 ubuntu-phablet lomiri-location-serviced-wrapper[1708]: malloc(): unaligned fastbin chunk detected
Mar 07 09:42:25 ubuntu-phablet systemd[1]: lomiri-location-service.service: Main process exited, code=killed, status=6/ABRT
Mar 07 09:42:25 ubuntu-phablet systemd[1]: lomiri-location-service.service: Failed with result 'signal'.
Mar 07 09:42:25 ubuntu-phablet systemd[1]: lomiri-location-service.service: Consumed 6.437s CPU time.

Mar 08 20:48:56 ubuntu-phablet systemd[1]: lomiri-location-service.service: Main process exited, code=killed, status=11/SEGV
Mar 08 20:48:56 ubuntu-phablet systemd[1]: lomiri-location-service.service: Failed with result 'signal'.
Mar 08 20:48:56 ubuntu-phablet systemd[1]: lomiri-location-service.service: Consumed 3.987s CPU time.

gpatel-fr

@WillemHexspoor

thanks for the precisions, now my guess is that I have yet not seen this problem because I have not used GPS a lot in the recent past.

slowcyclist

~~I finally nailed it! After a simple change and recompiling a package, I could have gps working continuously for over 3 hours, which had never happened previously on my FP5.~~

The problem was indeed in ubuntu_application_gps_hidl_for_hybris.cpp part of the package libubuntu-platform-hardware-api3 whose repo is at https://gitlab.com/ubports/development/core/platform-api, and lomiri-location-services calls this library to access the android drivers (through hybris). The problem was that the code continued to use a chunk of memory that had been supposedly freed (use-after-free) and the memory corruption only became visible when the allocator reused that freed memory chunk for a new allocation. At that point two pieces of code competed for the same memory, leading to crashing. The issue is more likely to happen quickly when many applications are running, explaining the somewhat random duration before the crash.

~~On my Xperia X, a different piece of code was used (ubuntu_application_gps_for_hybris.cpp, not ubuntu_application_gps_hidl_for_hybris.cpp) which explains why it never had this problem.~~

That gps_hidl piece of code has not been touched since 2020 and I believe UBports uses it in all phones running running Android 10+ vendor blobs, meaning that the problem should not be specific to the FP5. Yet, I find it really strange that no one reported the issue previously and that only @WillemHexspoor confirmed it here. Does that mean that UBports users either never use their gps for extended periods (e.g. routing with PureMaps or uNav) or that they only do so under waydroid, or that other phones access the gps through a different scheme (another library)?

~~I have filed a merge request to that package.~~

EDIT: I take back all what I said yesterday, it eventually crashed all the same after (except after a very long time (~10h), and then this morning again after only 3h... So the root cause is not yet found... I'm digging further.

gpatel-fr

@slowcyclist

Interesting find, thanks for the effort.

@slowcyclist said:

never use their gps for extended periods

Yes that's the case for me, although these last days I just tried that, and I got effectively 2 occurences of GPS not updating any more (launched Puremaps and Gps app together to make it more interesting). For me the reason is that the FP5 is gulping energy at such a rate that I try to stop services when not actually in use.

The only difference is that the GPS service (lomiri-location-service) never stopped for me (according to systemctl status) so no crash with unaligned errors, although I got the second time (did not think to check for that first time) segfaults and several restarts of sensorfwd service, maybe caused by this problem.

slowcyclist

Sorry for the previous noise... I continued working on that issue and I believe I now have a truly working fix : Gps positions continuously obtained for over 24h, twice. For that, I had to patch two repos, touching both the android side (hybris) and the linux side. While examining the code I realized I also could upgrade it so that the fp5 can now use more than than 32 (gps-only) satellites and provides more data (notably altitude uncertainty which was blank in sensor status so far). When the weather is clear my phone sees ~80 sats outdoors, but the android-side code clips at 64, and I'm not going to fight that... With the 64 sats clip the fp5 now still gets fixes indoors at my place while before it would stop when getting inside.
I'll post the corresponding merge requests in the two repos in the next days.

gpatel-fr

@slowcyclist

wow.
In the course of your investigations did you happen to take a look at the xtra service ? I have looked on and off at it and have not yet even understood why sometimes it downloads stuff from the qualcomm site(s) and then decides to not do anything for days.