Contacts dissapereance on RC v146 (2022-09-29)

jezek

Hello, today I updated to new RC (v146) and after restart all the contacts were gone. Except a few (5-6). When reverted back to stable, all the contacts were back again.

Has anyone experienced this?

Thanks.

flohack

Just for clarification what happens on the next OTA re contacts:

Software has been exchanged to replace the old contacts backend with a newer one
The old user data remains in your device
A new Db is created, and a migrator runs one time to move all contacts into the new Db (thats the current blocker, the migrator wont start on all devices due to a very difficult datetime issue in the recovery mode)
A flag file is created to not migrate more than once
Google contacts sync is disabled for now

Thats all. No contacts can be lost. In case anything goes really wrong the old database is there, and with another small emergency OTA we can help all users to re-migrate it for example. So we got most of scenarios covered.

I disagree to keep this pending until everything with sync is fixed. This means we cannot release another OTA, or any update to users. This means 16.04 is dead in the water (which it is already kinda) and we come closer to a cutoff date where everything must first work in 20.04 - imagine all of what we have queued now, approx 35 items, if one of them breaks in 20.04 which is very likely, we can again not release.

Also there is no point of saying that this is ok, simply put all hands on 20.04: It still does not have a release date, and most of all a lot of devices might not get upgraded if we cannot motivate the porters to do the necessary changes.

We need to be able to push 1, 2 or even 3 more OTAs before 20.04

Lakotaubp

@jezek There is a change being made to how the Contacts app works which are being worked through. They have reached RC and above but not stable. New RC releases are on hold iirc for now as the issue is worked out. I think this was mentioned in the last Q&A but need to check.
The best answer for now if you want to use RC is export the contacts in Vcard format while on stable and import them back in when on RC. Local (on device contacts should not be affected).

https://ubports.com/blog/ubports-news-1/post/ubuntu-touch-q-a-120-3863

flohack

Yes its ak nown issue and being worked on. The new migrator does not run on startup. Your contacts are there but in the wrong db ^^

lduboeuf

@jezek meanwhile, if you want to stay on RC and having your contacts you can run manually: /usr/share/session-migration/migrate_from_eds

jezek

@lduboeuf Hello. I've marked this topic as solved before I tried the script, you suggested. I expected it to be working, but today I tried and it didn't work.

What have I done:

Upgraded to RC v146
Checked if contacts are gone in the contacts-app. Yes, as before there are only 5 contacts.
Logged to phone via ssh.
Figured out that the script is in /usr/share/session-migration/scripts/migrate_from_eds and runned it. The script ended right after it started with no output or error.
Checked the contacts-app. No new contacts were there. Only the 5.

After this I viewed the script, figured out, that it checks if it has run before (by writing a file to ~/.local/share/evolution/addressbook/system/migrated_to_qtcontacts at the end of script), then checks for existing a contacts db ~/.local/share/evolution/addressbook/system/contacts.db, imports it via /usr/bin/eds_importer and writes the result to the migrated_to_qtcontacts file.

I didn't think long enough and harshly deleted the migrated_to_qtcontacts file, because I thought, the import didn't run properly because I run it from ssh, which was wrong thinking I realized a while after.

Then I run the /usr/share/session-migration/scripts/migrate_from_eds again, but now from phone terminal. This time there was an output:

The 'mergePresenceChanges' option has not been configured - presence changes will only be reported via ContactManagerEngine::contactsPresenceChanged()

But I think, that the eds_importer run without error because the newly created migrated_to_qtcontacts contained a 0:

phablet@ubuntu-phablet:~$ ll .local/share/evolution/addressbook/system/migrated_to_qtcontacts 
-rw-rw-r-- 1 phablet phablet 2 okt  1 17:59 .local/share/evolution/addressbook/system/migrated_to_qtcontacts
phablet@ubuntu-phablet:~$ cat .local/share/evolution/addressbook/system/migrated_to_qtcontacts 
0

After that I checked the contacts-app again and figured out, that the 5 contacts were doubled.

After reverting the phone to stable channel, the contacts are alright again.

Also then I realized, that I made a listing of the migrated_to_qtcontacts file before I harshly deleted it and it was like this:

phablet@ubuntu-phablet:~$ ll .local/share/evolution/addressbook/system/migrated_to_qtcontacts 
-rw-rw-r-- 1 phablet phablet 2 sep 15 17:42 .local/share/evolution/addressbook/system/migrated_to_qtcontacts

@Flohack See the date "sep. 15" . This was the day I tried the RC channel first time.

~~So my conclusion is, that the migration script kicked in after first rc install, but it has some faults and doesn't import all contacts. In my case only 5 from cca 400.~~ Edit: Wrong conclusion, see next posts.

lduboeuf

@jezek
If you want to be in a "clean" state before switching to RC. You must remove the ~/.local/share/evolution/addressbook/system/migrated_to_qtcontacts file and delete the content of ~/.local/share/system/privileged/Contacts

then after RC install try to run the script /usr/share/session-migration/scripts/migrate_from_eds if you see that you don't have any contacts.

jezek

@lduboeuf I've followed your advice and deleted the file and directory you suggested in previous post and installed the rc v146.

After installation I checked contacts and no contacts were found (the automatic migration didn't kick in). Then I said to myself, this is good, blank slate, just tap the synchronize button to sync with google account and everything will be fine. But the sync button was nowhere to find. I checked the Accounts in Settings. The google account is there, but only Calendar is ticked to use the account there (in stable there was Contacts and Calendar ticked).

Then I checked the ~/.local/share/evolution/addressbook/system/contacts.db with sqlite3 command and really. There are only 5 contacts there in the database file. After running the migrate_from_eds script, the 5 contacts were migrated successfully.

My previous assumption, that the migration script failed was wrong. The script worked. But the Contacts app somehow lost the ability to fetch contacts from System Accounts (for me it's google account configured there). Anyone else experienced this?

messayisto

@jezek Somewhere someone mentioned that contacts-sync is disabled in current RC due to various issues. I think it was @Flohack who said this, i have forgotten where.?

lduboeuf

@messayisto yes you're right, Contact sync will be disabled temporarily in next OTA.

jezek

@lduboeuf said in Contacts dissapereance on RC v146 (2022-09-29):

@messayisto yes you're right, Contact sync will be disabled temporarily in next OTA.

In the next stable OTA? If yes, than I think it's a big drawback/regression and the OTA should be postponed until it is figured out.

lduboeuf

@jezek said in Contacts dissapereance on RC v146 (2022-09-29):

@lduboeuf said in Contacts dissapereance on RC v146 (2022-09-29):

@messayisto yes you're right, Contact sync will be disabled temporarily in next OTA.

In the next stable OTA? If yes, than I think it's a big drawback/regression and the OTA should be postponed until it is figured out.

Yes this is what i've heard unfortunately. The contact migration is already a huge task . There is already known regressions ( most minors ) ( thread here https://github.com/ubports/ubuntu-touch/issues/997 ). and i'm afraid they will not be fixed with that OTA, Or we would have to wait another few months.
I agree all known regressions should be fixed before release. Maybe there should be a warning or discussion about that.

jezek

@lduboeuf said in Contacts dissapereance on RC v146 (2022-09-29):

Maybe there should be a warning or discussion about that.

There should definitely be a discussion. And if there is a vote, I would definitely vote for postponing the OTA until this is fixed (there is no regular OTA schedule anyway). Even if it is for 2 another months. You can't do this with some warning on a page. Some people don't read forums. They update and then no contacts. Everybody will ask how to downgrade. As for me I won't update until this is resolved.

domubpkm

Personally, I'm still on RC 54 (OTA-23) but very customized: I just installed the PRs I needed and everything works fine. As it is, I can wait. I have always done this by skipping several RCs. It's just my way of doing it.

lduboeuf

@jezek at least the contact migration issue will be fixed, but not sync yet

arubislander

@jezek said in Contacts dissapereance on RC v146 (2022-09-29):

There should definitely be a discussion. And if there is a vote, I would definitely vote for postponing the OTA until this is fixed (there is no regular OTA schedule anyway). Even if it is for 2 another months. You can't do this with some warning on a page. Some people don't read forums. They update and then no contacts. Everybody will ask how to downgrade. As for me I won't update until this is resolved.

My understanding is that the issue of the disappearing contacts is being looked to be fixed before the OTA is released. Only the contact sync issue will pushed back to a future OTA.
So updating to OTA-24 should not cause the user loss of contacts once this issue is fixed.

domubpkm

@arubislander said in Contacts dissapereance on RC v146 (2022-09-29):

is being looked to be fixed before the OTA is released

Yes. An OTA without contacts is IMPOSSIBLE, or else I'm in the 'Twilight Zone'.

Lakotaubp

I believe what @arubislander says is correct, but I have asked in the Q&A group to try and find out for sure to clear this up.

flohack

Just for clarification what happens on the next OTA re contacts:

Software has been exchanged to replace the old contacts backend with a newer one
The old user data remains in your device
A new Db is created, and a migrator runs one time to move all contacts into the new Db (thats the current blocker, the migrator wont start on all devices due to a very difficult datetime issue in the recovery mode)
A flag file is created to not migrate more than once
Google contacts sync is disabled for now

Thats all. No contacts can be lost. In case anything goes really wrong the old database is there, and with another small emergency OTA we can help all users to re-migrate it for example. So we got most of scenarios covered.

I disagree to keep this pending until everything with sync is fixed. This means we cannot release another OTA, or any update to users. This means 16.04 is dead in the water (which it is already kinda) and we come closer to a cutoff date where everything must first work in 20.04 - imagine all of what we have queued now, approx 35 items, if one of them breaks in 20.04 which is very likely, we can again not release.

Also there is no point of saying that this is ok, simply put all hands on 20.04: It still does not have a release date, and most of all a lot of devices might not get upgraded if we cannot motivate the porters to do the necessary changes.

We need to be able to push 1, 2 or even 3 more OTAs before 20.04