The MMS lost story

jezek

Failed MMS redownload progress report:

This week some work on propagating error details from nuntium to messaging-app was done, 4PR's were updated:

nuntium - Upon decoding mNotificationInd handle Expiry token (absolute/relative) and fill Expiry field as unix time. Propagate error description, expiry date and message size encoded as json to telepathy-ofono.
telepathy-ofono - forward the error message details in headers to history service.
history-service - save the error message status to messageStatus and details (encoded as json) to messageText field in db.
messaging-app - change error message detection, cause now the message can have a non empty text (with error details).
Note: The branch can't be compiled, I've tracked the error and created issue. After it's fixed (fix is trivial) I will rebase the branch it should pass the CI checks (I hope, localy with crossbuilder it worked).
Note: The expiry date could be wrongly encoded, haven't tested on real incoming messages. Some sources say the expiry date is encoded as Size + token + data, some Size + token + size + data. Majority and OpenMediaAssociation(?) documentation said it was the first case (s+t+d), so I've done it that way, but only tests in real world conditions will show. So keep an eye on the Expire field.

@lduboeuf Now is the time to ask for some help. Could you help with the messaging-app's UI part? Error details are saved in text field, encoded as json. Currently there are 3 fields:

"Description" (string) - just a text with error description.
"Expire" (string) - time in RFC3339 format, when the message expires at MMS provider.
"Size" (int) - size in bytes (?) of the message

I'm planing (as you suggested) to add a "CellularDataOn" (bool) field (have a better name) which could indicate if the cellular data was on/off at download time.

If any other error detail, that could be propagated from nuntium, comes to your mind, say it, I will try to add.

AppLee

@jezek said in The MMS lost story:

"CellularDataOn" (bool) field (have a better name)

Usually for boolean I use "isMyState" or "hasMyProperty", in this case I would have used hasCellularData or hasMobileData (the first one is my favorite as the toggle switch has the same label in the settings.

lduboeuf

@jezek said in The MMS lost story:

Failed MMS redownload progress report:

Note: The branch can't be compiled, I've tracked the error and created issue. After it's fixed (fix is trivial) I will rebase the branch it should pass the CI checks (I hope, localy with crossbuilder it worked).

fixed, you can rebase now

Note: The expiry date could be wrongly encoded, haven't tested on real incoming messages. Some sources say the expiry date is encoded as Size + token + data, some Size + token + size + data. Majority and OpenMediaAssociation(?) documentation said it was the first case (s+t+d), so I've done it that way, but only tests in real world conditions will show. So keep an eye on the Expire field.

the one defined here is not working ?: https://github.com/ubports/nuntium/blob/38940464d532a145412c47d8901c1a74fb662eff/mms/decoder.go#L405

@lduboeuf Now is the time to ask for some help. Could you help with the messaging-app's UI part? Error details are saved in text field, encoded as json. Currently there are 3 fields:

"Description" (string) - just a text with error description.

"Expire" (string) - time in RFC3339 format, when the message expires at MMS provider.

"Size" (int) - size in bytes (?) of the message

Yes for sure ~~, how i can commit to your PR ?~~

Edit: For the expiry date, i just had a case where Expire field is empty in json

lduboeuf

@jezek said in The MMS lost story:

"Description" (string) - just a text with error description.

Humm, where does come from error messages, are they normalized somewhere ?

jezek

@lduboeuf said in The MMS lost story:

fixed, you can rebase now

heh, no need to rebase, you push right to my fork's branch, just merged my local branch with origin. Honestly, I didn't do this kind of collaboration until this time. How do you push to another's PR?

@lduboeuf said in The MMS lost story:

the one defined here is not working ?: https://github.com/ubports/nuntium/blob/38940464d532a145412c47d8901c1a74fb662eff/mms/decoder.go#L405

There is a TODO near. The Expiry can come in 2 formats, depending on the token. If token means "absolute" then the following bytes are unix date. If token means "relative" the following bytes are # of seconds. The previous code always handled the Expiry as relative (but it didn't matter, cause it was nowhere used). If we want to use Expiry field, it has to be correctly encoded. So the token has to come in account. I had some doubts, but I recently browsed some other code and pages and now I know I have a bug in decoding absolute token. Will fix.

@lduboeuf said in The MMS lost story:

Edit: For the expiry date, i just had a case where Expire field is empty in json

If I recall correctly, the Expire should be empty only when no Expiry field comes in the mNotifiationInd (message notification) from provider, but could be a bug. Can you prowide nuntium logs for that event please?

jezek

@lduboeuf said in The MMS lost story:

@jezek said in The MMS lost story:

"Description" (string) - just a text with error description.

Humm, where does come from error messages, are they normalized somewhere ?

The error is just a string, that comes from error, that is passed to handleMRetrieveConfDownloadError function in nuntium. They are not translated. I don't know what you mean under normalized. Maybe I should send some error type.

lduboeuf

@jezek said in The MMS lost story:

Honestly, I didn't do this kind of collaboration until this time. How do you push to another's PR?

If you are the maintainer of a repo ( my case for messaging-app ), you can add commits to the PR since users allow to edit the PR ( there is a checkbox for that when opening a PR). So for that case, if i checkout your repo, the PR branch is editable by the maintainer.

@lduboeuf said in The MMS lost story:

the one defined here is not working ?: https://github.com/ubports/nuntium/blob/38940464d532a145412c47d8901c1a74fb662eff/mms/decoder.go#L405

There is a TODO near. The Expiry can come in 2 formats, depending on the token. If token means "absolute" then the following bytes are unix date. If token means "relative" the following bytes are # of seconds. The previous code always handled the Expiry as relative (but it didn't matter, cause it was nowhere used). If we want to use Expiry field, it has to be correctly encoded. So the token has to come in account. I had some doubts, but I recently browsed some other code and pages and now I know I have a bug in decoding absolute token. Will fix.

@lduboeuf said in The MMS lost story:

Edit: For the expiry date, i just had a case where Expire field is empty in json

Thanks for the explanation

If I recall correctly, the Expire should be empty only when no Expiry field comes in the mNotifiationInd (message notification) from provider, but could be a bug. Can you prowide nuntium logs for that event please?

Sorry, don't have my logs now, but as it was a MMS sent by myself to myself, could it be the issue ?

lduboeuf

@jezek said in The MMS lost story:

@lduboeuf said in The MMS lost story:

Humm, where does come from error messages, are they normalized somewhere ?

The error is just a string, that comes from error, that is passed to handleMRetrieveConfDownloadError function in nuntium. They are not translated. I don't know what you mean under normalized. Maybe I should send some error type.

Ok, but yes, it is preferable to have "normalized error" to be able to translate them later. e.g "x-mms-error-no-context", something like that.

jezek

@lduboeuf said in The MMS lost story:

Sorry, don't have my logs now, but as it was a MMS sent by myself to myself, could it be the issue ?

Oh, I completely forgot. I can send message to myself. Thanks for making me remember. Will test it that way.

@lduboeuf said in The MMS lost story:

Ok, but yes, it is preferable to have "normalized error" to be able to translate them later. e.g "x-mms-error-no-context", something like that.

Ok, will do it that way.

lduboeuf

@jezek said in The MMS lost story:

@lduboeuf said in The MMS lost story:

Sorry, don't have my logs now, but as it was a MMS sent by myself to myself, could it be the issue ?

Oh, I completely forgot. I can send message to myself. Thanks for making me remember. Will test it that way.

Yes sometimes you need to be quick switching cellular off

@lduboeuf said in The MMS lost story:

Ok, but yes, it is preferable to have "normalized error" to be able to translate them later. e.g "x-mms-error-no-context", something like that.

Ok, will do it that way.

Of course we still need to make it translatable on both messaging-app and telephony-service for the notification part.
Can i see somewhere all the possible error messages ?
I should ask also in UI forum for what to do

jezek

Failed MMS redownload progress report:

This week nuntium PR only got updates.

I've been working on the download error message reports.
@lduboeuf sorry for late reply about errors, I've been working on it. Here is the result.

The error message text in json now contains following fields:

Code [string] - contains one of following error code:
- x-ubports-nuntium-mms-error-unknown - unknown error, should not happen.
- x-ubports-nuntium-mms-error-activate-context - failed first contact with MMS service & context activation. Redownload is allowed if not expired.
- x-ubports-nuntium-mms-error-get-proxy - first contact was successful, but getting proxy failed. Very rare, should occur only with bad signal. Redownload is allowed if not expired.
- "x-ubports-nuntium-mms-error-download-content - context & proxy ok, but download failed. Very rare, should occur only with bad signal or maybe if message expires (needs investigation what happens if trying to redownload expired message). Redownload is allowed if not expired.
- x-ubports-nuntium-mms-error-storage - the downloaded message file can't be saved to storage. Should happen only if disk full, or bad permissions (almost never). Redownload is NOT allowed (maybe should be?).
- x-ubports-nuntium-mms-error-forward - everything went ok, but for some unexplained reason, can't forward the message to telepathy-ofono. But if that's the case, I'll be a miracle if the error message can be sent o telepathy-ofono, so should never happen. Redownload is NOT allowed.
Message [string] - raw error message caught in nuntium (just for debug purposes).
Expire [string, optional] - date-time in RFC3339 format, when the message expires in MMS service. If field not present, the Expire field was not sent by provider ~~(how do we handle this?)~~. Edit: However, if expiry time not provided by operator, we assume 7 days expiry time, so this field will not be empty ever (unless some bug).
Size [int, optional] - Size in bytes of message. If not present, size info was no sent by provider or size is 0.
MobileData [bool, optional] - if mobile data was enabled in the time of error handling. If not present, the mobile data property could not be determined.

Note: The PR checks failed. I found FAIL github.com/ubports/nuntium/mms [build failed] in jenkins logs, but localy the mms package can be build. I think it's because of go version jenkins is using (Setting up golang-1.6-src (1.6.2-0ubuntu5~16.04.4) ...), but I need at least 1.7 (or better 1.13). I don't know how to force required version. Should I rewrite code to be able to compile with 1.6? I'd rather use latest (possible) golang version.
Edit: I tried to change the debian/control to use go-1.13. The package gets installed, but it fails with dh_auto_build: go install -v failed to to execute: No such file or directory
Edit: PR build is successful, thanks to @lduboeuf

lduboeuf

@jezek said in The MMS lost story:

Failed MMS redownload progress report:

This week nuntium PR only got updates.

I've been working on the download error message reports.
@lduboeuf sorry for late reply about errors, I've been working on it. Here is the result.

The error message text in json now contains following fields:

Code [string] - contains one of following error code:

x-ubports-nuntium-error-unknown - unknown error, should not happen.

x-ubports-nuntium-mms-activate-context-error - failed first contact with MMS service & context activation. Redownload is allowed if not expired.

x-ubports-nuntium-mms-get-proxy-error - first contact was successful, but getting proxy failed. Very rare, should occur only with bad signal. Redownload is allowed if not expired.

"x-ubports-nuntium-mms-download-content-error - context & proxy ok, but download failed. Very rare, should occur only with bad signal or maybe if message expires (needs investigation what happens if trying to redownload expired message). Redownload is allowed if not expired.

x-ubports-nuntium-mms-storage-error - the downloaded message file can't be saved to storage. Should happen only if disk full, or bad permissions (almost never). Redownload is NOT allowed (maybe should be?).

x-ubports-nuntium-mms-forward-error - everything went ok, but for some unexplained reason, can't forward the message to telepathy-ofono. But if that's the case, I'll be a miracle if the error message can be sent o telepathy-ofono, so should never happen. Redownload is NOT allowed.

if possible, i would suggest having the same prefix for all error status, x-ubports-nuntium-mms-error-...

Message [string] - raw error message caught in nuntium (just for debug purposes).

Expire [string, optional] - date-time in RFC3339 format, when the message expires in MMS service. If field not present, the Expire field was not sent by provider (how do we handle this?).

not present is ok for me, don't see what is your question.

Size [int, optional] - Size in bytes of message. If not present, size info was no sent by provider or size is 0.

MobileData [bool, optional] - if mobile data was enabled in the time of error handling. If not present, the mobile data property could not be determined.

Note: The PR checks failed. I found FAIL github.com/ubports/nuntium/mms [build failed] in jenkins logs, but localy the mms package can be build. I think it's because of go version jenkins is using (Setting up golang-1.6-src (1.6.2-0ubuntu5~16.04.4) ...), but I need at least 1.7 (or better 1.13). I don't know how to force required version. Should I rewrite code to be able to compile with 1.6? I'd rather use latest (possible) golang version.
Edit: I tried to change the debian/control to use go-1.13. The package gets installed, but it fails with dh_auto_build: go install -v failed to to execute: No such file or directory

~~humm, changes in debian control file is not enough ?~~
[Edit]: Did some tries with debian control, so even by setting the PATH to /usr/lib/go-1.13/bin in rules (export PATH := /usr/lib/go-1.13/bin:$(PATH) ), i had no success, it didn't find go files ( but no more "no such files or directory" ), though. But i'm really a noob in that debian packaging thing.

[EDIT2]:
So finally got it working... but really this is just some copy/paste from everywhere....
For the last part, see https://bugs.launchpad.net/ubuntu/+source/dh-golang/+bug/1911911

in control->

...
golang-1.13-go,
...

in rules ->

...
export DH_GOLANG_INSTALL_ALL := 1
#https://bugs.launchpad.net/ubuntu/+source/dh-golang/+bug/1911911
export GOCACHE := ${CURDIR}/_build/go-build
export GO111MODULE = off
export GOPATH := $(CURDIR)/obj-$(DEB_BUILD_GNU_TYPE)/src/$(DH_GOPKG)
export PATH := /usr/lib/go-1.13/bin:$(PATH)

jezek

@lduboeuf said in The MMS lost story:

if possible, i would suggest having the same prefix for all error status, x-ubports-nuntium-mms-error-...

ok, will fix & notice you (hopefully today).

not present is ok for me, don't see what is your question.

if there is no expire information from operator, can we expect the message never expires, or expires after some default time (eg. 7days)?

[Edit]: Did some tries with debian control, so even by setting the PATH to /usr/lib/go-1.13/bin in rules (export PATH := /usr/lib/go-1.13/bin:$(PATH) ), i had no success, it didn't find go files ( but no more "no such files or directory" ), though. But i'm really a noob in that debian packaging thing.

I'm also virgin in the debian packaging thing. For me it is read internet + trial/error cycles for now. Will experiment. But our approach (with the path) could be a good one. If the go install command runs, but didn't find go files, the setting GOROOT and/or GOPATH may help. Will try (also can you ). If we can't solve it today, tomorrow I will comment out the incriminating code (it's just some testing thing), to be able to compile at least.
Edit: The moment I posted this, you edited your comment wit solution to the packaging problem. Thank you. (I've read the same bug on launchpad, but you were quicker by implementing )

lduboeuf

if there is no expire information from operator, can we expect the message never expires, or expires after some default time (eg. 7days)?

Yes, lets say 7 days

I'm also virgin in the debian packaging thing. For me it is read internet + trial/error cycles for now. Will experiment. But our approach (with the path) could be a good one. If the go install command runs, but didn't find go files, the setting GOROOT and/or GOPATH may help. Will try (also can you ). If we can't solve it today, tomorrow I will comment out the incriminating code (it's just some testing thing), to be able to compile at least.

Yes, see above, i've edited my comment and finally got it built...

jezek

@lduboeuf I've pushed commits to set default expiry time (7days), unify download error code and changes you suggested to be able to build a dep package (thank you again). I've updated the previous post to reflect current state.

lduboeuf

@jezek said in The MMS lost story:

@lduboeuf I've pushed commits to set default expiry time (7days),

sorry, need confirmation, but i see 15 days seems the default duration

unify download error code

also maybe @UniSuperBox @Flohack or someone from core teams can have a look on them

and changes you suggested to be able to build a dep package (thank you again).

glad it helped , thanks also for keeping that alive

lduboeuf

@jezek Is there anything still to be done in backend side ?

jezek

@lduboeuf said in The MMS lost story:

@jezek Is there anything still to be done in backend side ?

Yes, I have a bunch TODOs in nuntium (and other apps along the way to messaging-app), but the error message part will stay the same. Yesterday I trimmed some TODOs (loccaly, not pushed yet) and looked at what has to be done and how it affects the current error message format (and error codes) and I've come to conclusion, that it will be not affected (unless you want to change something).

Why do you ask?

lduboeuf

@jezek ok,nice. Was just asking if i need to do the UI part now
BTW we don't need to have all errors message handling at first.
And if you see there is too much work remaining, we can cut some parts to be able to deliver a solution quicker.

This is so awesome that we can redownload afterward without asking the user

jezek

@lduboeuf said in The MMS lost story:

@jezek ok,nice. Was just asking if i need to do the UI part now

Yes, you can start (if you want).

BTW we don't need to have all errors message handling at first.

That, I leave for you, sir.

And if you see there is too much work remaining, we can cut some parts to be able to deliver a solution quicker.

I have a plan to move some TODOs to github issue tracker to speed up release, but some need to be done (in my opinion).

This is so awesome that we can redownload afterward without asking the user

Yes, that is awesome. And in my opinion it should be switchable (on/off). Eg.: I have a prepaid card and I wouldn't like it if it would automatically download some MMS and use some credit.