How to sync contacts of a CalDav server successfully?

marc_aurel · 14 Dec 2018, 17:00

I followed the instructions of the documentation: https://docs.ubports.com/en/latest/userguide/advanceduse/dav.html

Unfortunately the synchronisation failed, althoug the log output of the script says, that the synchronisation was successfull. Maybe the problem is, that I haven't configured the calDav server in the script, because I use the system account settings of UBports.

Here is some logoutput:

evolution-contacts: database 'sogoadressen' (...) was created.
[INFO] addressbook: configuring datastore with sync mode 'two-way'
[INFO] calendar: configuring datastore with sync mode 'two-way'
[INFO] memo: configuring datastore with sync mode 'two-way'
[INFO] todo: configuring datastore with sync mode 'two-way'
...
[INFO] addressbook: not selected
[INFO] calendar: not selected
[INFO] memo: not selected
[INFO] sogoadressen: checking usability...
[INFO] sogoadressen: configuring datastore with sync mode 'None'
[INFO] sogokalender: not selected
[INFO] todo: not selected
...
[INFO] @default/addressbook: inactive
[INFO] @default/calendar: inactive
[INFO] @default/memo: inactive
[INFO] @default/sogokalender: inactive
[INFO] @default/todo: inactive
[INFO @sogoadressen] target side of local sync ready
[INFO @sogoadressen] @sogoadressen/addressbook: inactive
[INFO @sogoadressen] @sogoadressen/calendar: inactive
[INFO @sogoadressen] @sogoadressen/memo: inactive
[INFO @sogoadressen] @sogoadressen/todo: inactive
[INFO @sogoadressen] @sogoadressen/sogoadressen: using configured database=https://...
[INFO @sogoadressen] @sogoadressen/sogoadressen: starting first time sync from client (peer is server)
 [INFO] @default/sogoadressen: starting first time sync from client (peer is client)
 [INFO] creating complete data backup of datastore sogoadressen before sync (enabled with dumpData and needed for printChanges)
@default data changes to be applied during synchronization:
*** @default/sogoadressen ***
Comparison was impossible.

[INFO] @default/sogoadressen: deleting 0
[INFO] @default/sogoadressen: started
[INFO @sogoadressen] @sogoadressen/sogoadressen: started
[INFO] @default/sogoadressen: first time sync done successfully
[INFO @sogoadressen] @sogoadressen/sogoadressen: first time sync done successfully

Synchronization successful.

Changes applied during synchronization (@sogoadressen):
+---------------|-----------------------|-----------------------|-CON-+
|               |     @sogoadressen     |       @default        | FLI |
|        Source | NEW | MOD | DEL | ERR | NEW | MOD | DEL | ERR | CTS |
+---------------+-----+-----+-----+-----+-----+-----+-----+-----+-----+
|  sogoadressen |  0  |  0  |  0  |  0  |  0  |  0  |  0  |  0  |  0  |
|  refresh-from-local, 0 KB sent by client, 0 KB received             |
+---------------+-----+-----+-----+-----+-----+-----+-----+-----+-----+
|          start Fri Dec 14 17:36:12 2018, duration 0:02min           |
|               synchronization completed successfully                |
+---------------+-----+-----+-----+-----+-----+-----+-----+-----+-----+
[INFO] creating complete data backup after sync (enabled with dumpData and needed for printChanges)

Synchronization successful.

 Changes applied during synchronization:
 +---------------|-----------------------|-----------------------|-CON-+
 |               |       @default        |     @sogoadressen     | FLI |
 |        Source | NEW | MOD | DEL | ERR | NEW | MOD | DEL | ERR | CTS |
+---------------+-----+-----+-----+-----+-----+-----+-----+-----+-----+
|  sogoadressen |  0  |  0  |  0  |  0  |  0  |  0  |  0  |  0  |  0  |
|  refresh-from-remote, 0 KB sent by client, 0 KB received            |
|  item(s) in database backup: 0 before sync, 0 after it              |
+---------------+-----+-----+-----+-----+-----+-----+-----+-----+-----+
|          start Fri Dec 14 17:36:12 2018, duration 0:02min           |
|               synchronization completed successfully                |
+---------------+-----+-----+-----+-----+-----+-----+-----+-----+-----+

Data modified @default during synchronization:
*** @default/sogoadressen ***
Comparison was impossible.

Some ideas, what is wrong?

joe · 16 Dec 2018, 16:21

@marc_aurel

Yes this part needs to be filled in:

CAL_URL="https://..."                # add the caldav URL here
CONTACTS_URL="https://...."          # add the caldav URL here
USERNAME=""               # you know this one
PASSWORD=""               # lots of ******

# ----------------- [ Phone ] ----------------- #

marc_aurel · 16 Dec 2018, 16:37

@joe Thanks. Yes, I have done this except for the calendar, because I have already synchronized my calendar by using the system account. I assumed, that this would be a conflict. Could that be the reason for the failed synchronisation?

geekvine · 18 Dec 2018, 02:39

had the same issue and the response i received was "it works for everyone else" so i went back to unlegacy android for the time being.

marc_aurel · 18 Dec 2018, 15:09

@joe Unfortunately it doesn't help to configure the calendar url. It would be surprising, because the sysnchronsation processes are running independently.

Now I can see three empty addressbooks and calendars after three trials. I can't find a location of the databases in the shell script to remove them. Has someone an ideaa how to remove this trash?

Here is my log output with masked URLs:

database 'sogokalender' (38373199-f415-404c-b286-376bb080539c) was created.
[INFO] @default/5iqsercom5qk62i69evfgbqq529n2a: inactive
[INFO] @default/5joergwurzer: inactive
[INFO] @default/6contactbirthdays: inactive
[INFO] @default/6default: inactive
[INFO] @default/8182ffb55fd864c5a939a3c151c398: inactive
[INFO] @default/8default: inactive
[INFO] @default/8schulferien: inactive
[INFO] @default/addressbook: inactive
[INFO] @default/calendar: inactive
[INFO] @default/memo: inactive
[INFO] @default/sogoadressen: inactive
[INFO] @default/todo: inactive
[INFO @sogokalender] target side of local sync ready
[INFO @sogokalender] @sogokalender/addressbook: inactive
[INFO @sogokalender] @sogokalender/calendar: inactive
[INFO @sogokalender] @sogokalender/memo: inactive
[INFO @sogokalender] @sogokalender/todo: inactive
[INFO @sogokalender] @sogokalender/sogokalender: using configured database=https://............................
[INFO @sogokalender] @sogokalender/sogokalender: starting first time sync from client (peer is server)
[INFO] @default/sogokalender: starting slow sync from client (peer is client)
[INFO] creating complete data backup of datastore sogokalender before sync (enabled with dumpData and needed for printChanges)
@default data changes to be applied during synchronization:
*** @default/sogokalender ***
Comparison was impossible.

[INFO] @default/sogokalender: deleting 0
[INFO] @default/sogokalender: started
[INFO @sogokalender] @sogokalender/sogokalender: started
[INFO] @default/sogokalender: slow sync done successfully
[INFO @sogokalender] @sogokalender/sogokalender: first time sync done successfully

Synchronization successful.

Changes applied during synchronization (@sogokalender):
+---------------|-----------------------|-----------------------|-CON-+
|               |     @sogokalender     |       @default        | FLI |
|        Source | NEW | MOD | DEL | ERR | NEW | MOD | DEL | ERR | CTS |
+---------------+-----+-----+-----+-----+-----+-----+-----+-----+-----+
|  sogokalender |  0  |  0  |  0  |  0  |  0  |  0  |  0  |  0  |  0  |
|  refresh-from-local, 0 KB sent by client, 0 KB received             |
+---------------+-----+-----+-----+-----+-----+-----+-----+-----+-----+
|          start Tue Dec 18 15:34:57 2018, duration 0:02min           |
|               synchronization completed successfully                |
+---------------+-----+-----+-----+-----+-----+-----+-----+-----+-----+
[INFO] creating complete data backup after sync (enabled with dumpData and needed for printChanges)

Synchronization successful.

Changes applied during synchronization:
+---------------|-----------------------|-----------------------|-CON-+
|               |       @default        |     @sogokalender     | FLI |
|        Source | NEW | MOD | DEL | ERR | NEW | MOD | DEL | ERR | CTS |
+---------------+-----+-----+-----+-----+-----+-----+-----+-----+-----+
|  sogokalender |  0  |  0  |  0  |  0  |  0  |  0  |  0  |  0  |  0  |
|  refresh-from-remote, 0 KB sent by client, 0 KB received            |
|  item(s) in database backup: 0 before sync, 0 after it              |
+---------------+-----+-----+-----+-----+-----+-----+-----+-----+-----+
|          start Tue Dec 18 15:34:56 2018, duration 0:03min           |
|               synchronization completed successfully                |
+---------------+-----+-----+-----+-----+-----+-----+-----+-----+-----+

Data modified @default during synchronization:
*** @default/sogokalender ***
Comparison was impossible.

evolution-contacts: database 'sogoadressen' (7b258d1a-90eb-4ee4-a1d2-6b58cd1da6b1) was created.
[INFO] @default/5iqsercom5qk62i69evfgbqq529n2a: inactive
[INFO] @default/5joergwurzer: inactive
[INFO] @default/6contactbirthdays: inactive
[INFO] @default/6default: inactive
[INFO] @default/8182ffb55fd864c5a939a3c151c398: inactive
[INFO] @default/8default: inactive
[INFO] @default/8schulferien: inactive
[INFO] @default/addressbook: inactive
[INFO] @default/calendar: inactive
[INFO] @default/memo: inactive
[INFO] @default/sogokalender: inactive
[INFO] @default/todo: inactive
[INFO @sogoadressen] target side of local sync ready
[INFO @sogoadressen] @sogoadressen/addressbook: inactive
[INFO @sogoadressen] @sogoadressen/calendar: inactive
[INFO @sogoadressen] @sogoadressen/memo: inactive
[INFO @sogoadressen] @sogoadressen/todo: inactive
[INFO @sogoadressen] @sogoadressen/sogoadressen: using configured database=https://..........................
[INFO @sogoadressen] @sogoadressen/sogoadressen: starting first time sync from client (peer is server)
[INFO] @default/sogoadressen: starting slow sync from client (peer is client)
[INFO] creating complete data backup of datastore sogoadressen before sync (enabled with dumpData and needed for printChanges)
@default data changes to be applied during synchronization:
*** @default/sogoadressen ***
Comparison was impossible.

[INFO] @default/sogoadressen: deleting 0
[INFO] @default/sogoadressen: started
[INFO @sogoadressen] @sogoadressen/sogoadressen: started
[INFO] @default/sogoadressen: slow sync done successfully
[INFO @sogoadressen] @sogoadressen/sogoadressen: first time sync done successfully

Synchronization successful.

Changes applied during synchronization (@sogoadressen):
+---------------|-----------------------|-----------------------|-CON-+
|               |     @sogoadressen     |       @default        | FLI |
|        Source | NEW | MOD | DEL | ERR | NEW | MOD | DEL | ERR | CTS |
+---------------+-----+-----+-----+-----+-----+-----+-----+-----+-----+
|  sogoadressen |  0  |  0  |  0  |  0  |  0  |  0  |  0  |  0  |  0  |
|  refresh-from-local, 0 KB sent by client, 0 KB received             |
+---------------+-----+-----+-----+-----+-----+-----+-----+-----+-----+
|          start Tue Dec 18 15:35:06 2018, duration 0:01min           |
|               synchronization completed successfully                |
+---------------+-----+-----+-----+-----+-----+-----+-----+-----+-----+
[INFO] creating complete data backup after sync (enabled with dumpData and needed for printChanges)

Synchronization successful.

Changes applied during synchronization:
+---------------|-----------------------|-----------------------|-CON-+
|               |       @default        |     @sogoadressen     | FLI |
|        Source | NEW | MOD | DEL | ERR | NEW | MOD | DEL | ERR | CTS |
+---------------+-----+-----+-----+-----+-----+-----+-----+-----+-----+
|  sogoadressen |  0  |  0  |  0  |  0  |  0  |  0  |  0  |  0  |  0  |
|  refresh-from-remote, 0 KB sent by client, 0 KB received            |
|  item(s) in database backup: 0 before sync, 0 after it              |
+---------------+-----+-----+-----+-----+-----+-----+-----+-----+-----+
|          start Tue Dec 18 15:35:05 2018, duration 0:02min           |
|               synchronization completed successfully                |
+---------------+-----+-----+-----+-----+-----+-----+-----+-----+-----+

Data modified @default during synchronization:
*** @default/sogoadressen ***
Comparison was impossible.

no crontab for phablet
cron stop/waiting
cron start/running, process 16458

It seems, that the script is not correct or up to date.

I have tried an alternative by importing the addresses in the android addressbook app, but I don't find the shared file system, where I can copy the icf file.

marc_aurel · 20 Dec 2018, 09:17

Hello again. I managed to remove the databases:

syncevolution --remove-database backend=evolution-contacts database=sogoadressen
syncevolution --remove-database backend=evolution-calendar database=sogokalender

I realized, in the documentation for manual synchronisation, that the script might only work for owncloud. I'm using the Baikal server instead.

syncevolution --configure --template SyncEvolution_Client syncURL=local://@owncloud username= password= owncloud Calendar1
syncevolution --configure sync=two-way database=Calendar1 owncloud Calendar1

The Baikal server has a different URL path:

https://<my-domain>/baikal-flat/card.php/addressbooks/<my-account-name>
https://<my-domain>/baikal-flat/cal.php/principals/<my-account-name>

CalDav and CardDav synchronisation works perfect on macOS, iOS, Android, Lineage, Sailfish but not with the script for UBports. Setup for the calendar works randomly on UBporty by adding the CalDav account within the default calendar app.

joe · 22 Dec 2018, 16:12

Sorry for the non-reply.
I had only used that sync tutorial a long time ago. I think I only used the cardDav part as well. I am not currently using it, so I'm not sure if it still works.

The system calDav sync works with NextCloud now. So maybe you can remove those lines from the script to sync calendar.

Not sure if you saw the comments on github, but there's something about line 11.

"On line 11 you should also define CONTACTS_URL="https://..." otherwise on line 50 there are no contacts to import..."

This might be an updated script:
https://gist.github.com/vanyasem/379095d25ac350676fc70c42efe17c8c

Ellypsis · 22 Dec 2018, 18:16

Hi,
Here the script I use to sync on UT my carddav account on my baikal server :

#!/bin/bash
# This script is a draft combination of the script found at https://gist.github.com/tcarrondo
# It is more or less to remember what I have done to make it work for my Fairphone 2 with UBports ubuntu touch
# Combined by me: Sebastian Gallehr <sebastian@gallehr.de>
# Thanks to: Tiago Carrondo <tcarrondo@ubuntu.com>
# Thanks to: Romain Fluttaz <romain@botux.fr>
# Thanks to: Wayne Ward <info@wayneward.co.uk>
# Thanks to: Mitchell Reese <mitchell@curiouslegends.com.au>
# --------------- [ Server ] ---------------- #
#CAL_URL="https://domaine.ovh/baikal/html/cal.php/calendars/username/default/"      # insert the CalDAV URL here
CONTACTS_URL="http://domaine.ovh/baikal/html/card.php/addressbooks/username/default/" # insert the CardDAV URL here
USERNAME="username"                # your CalDAV/CardDAV username
PASSWORD="password"                # your CalDAV/CardDAV password

# ----------------- [ Phone ] ----------------- #
#CALENDAR_CONFIG_NAME="Baikal"   # I use "myCloud"
CONTACTS_CONFIG_NAME="Baikal"   # I use "myCloud"

#CALENDAR_NAME="personalcalendar"          # I use "personalcalendar"
#CALENDAR_VISUAL_NAME="BaikalCalendar"   # a nice name to show on the Calendar app like "OwnCalendar"

CONTACTS_NAME="personalcontacts"          # I use "personalcontacts"
CONTACTS_VISUAL_NAME="BaikalContact"   # a nice name to show on the Contacts app like "OwnContacts"

CRON_FREQUENCY="hourly"               # Sync frequency, I use "hourly"


export DBUS_SESSION_BUS_ADDRESS=$(ps -u phablet e | grep -Eo 'dbus-daemon.*address=unix:abstract=/tmp/dbus-[A-Za-z0-9]{10}' | tail -c35)

#CALENDAR

#Create Calendar
#syncevolution --create-database backend=evolution-calendar database=$CALENDAR_VISUAL_NAME
#Create Peer
#syncevolution --configure --template webdav username=$USERNAME password=$PASSWORD syncURL=$CAL_URL keyring=no target-config@$CALENDAR_CONFIG_NAME
#Create New Source
#syncevolution --configure backend=evolution-calendar database=$CALENDAR_VISUAL_NAME @default $CALENDAR_NAME
#Add remote database
#syncevolution --configure database=$CAL_URL backend=caldav target-config@$CALENDAR_CONFIG_NAME $CALENDAR_NAME
#Connect remote calendars with local databases
#syncevolution --configure --template SyncEvolution_Client syncURL=local://@$CALENDAR_CONFIG_NAME $CALENDAR_CONFIG_NAME $CALENDAR_NAME
#Add local database to the source
#syncevolution --configure sync=two-way database=$CALENDAR_VISUAL_NAME $CALENDAR_CONFIG_NAME $CALENDAR_NAME
#Start first sync
#syncevolution --sync refresh-from-remote $CALENDAR_CONFIG_NAME $CALENDAR_NAME

#CONTACT

#Create contact list
syncevolution --create-database backend=evolution-contacts database=$CONTACTS_VISUAL_NAME
#Create Peer
syncevolution --configure --template webdav username=$USERNAME password=$PASSWORD syncURL=$CONTACTS_URL keyring=no target-config@$CONTACTS_CONFIG_NAME
#Create New Source
syncevolution --configure backend=evolution-contacts database=$CONTACTS_VISUAL_NAME @default $CONTACTS_NAME
#Add remote database
syncevolution --configure database=$CONTACTS_URL backend=carddav target-config@$CONTACTS_CONFIG_NAME $CONTACTS_NAME
#Connect remote contact list with local databases
syncevolution --configure --template SyncEvolution_Client Sync=None syncURL=local://@$CONTACTS_CONFIG_NAME $CONTACTS_CONFIG_NAME $CONTACTS_NAME
#Add local database to the source
syncevolution --configure sync=two-way backend=evolution-contacts database=$CONTACTS_VISUAL_NAME $CONTACTS_CONFIG_NAME $CONTACTS_NAME
#Start first sync
syncevolution --sync refresh-from-remote $CONTACTS_CONFIG_NAME $CONTACTS_NAME

#Add Sync Cron job
#sudo mount / -o remount,rw
#COMMAND_LINE="export DISPLAY=:0.0 && export DBUS_SESSION_BUS_ADDRESS=$(ps -u phablet e | grep -Eo 'dbus-daemon.*address=unix:abstract=/tmp/dbus-[A-Za-z0-9]{10}' | tail -c35) && /usr/bin/syncevolution $CALENDAR_NAME && /usr/bin/syncevolution $CONTACTS_NAME"
#sudo sh -c "echo '$COMMAND_LINE' > /sbin/sogosync"
#sudo chmod +x /sbin/sogosync

#CRON_LINE="@$CRON_FREQUENCY /sbin/sogosync"
#(crontab -u phablet -r;) # only if no other cron jobs already exist in crontab
#(crontab -u phablet -l; echo "$CRON_LINE" ) | crontab -u phablet -
#sudo mount / -o remount,ro
#sudo service cron restart

The script comes from the UT wiki, with some minor modifications.
As you can see in the script, I don't use the crontab services. I prefer to sync manually.
Here is the bash_aliases I use to sync my contacts :
alias SyncContact="syncevolution baikal personalcontacts"
Don't forget to set your favorite address book in Contact/Preference, to automatically add your new contacts to the synced address book.

marc_aurel · 1 Oct 2019, 09:42

@ellypsis Thank your for your helpful post. The problem was probably, that didn't add default at the end of the URL, because I expected, that the synchronisation retrieves the available calendars and contacts. I never add the name of the addressbook or calender in system settings of any OS.

I have tested the updated script successfully. But I had to restart the phone and remove the created database and start the script again. For any reasons.