Script for updating apps [help needed]

CiberSheep

I was collecting some of the regex and code I found here to make a script to update apps from Xenial to Focal.

I've put some of this in an unfinished sh:
https://gitlab.com/cibersheep/update-apps
MR are welcome to finish the process that are missing

Warning: It is not complete, I'm sure there's a better way of doing things and it is a good moment to do the same for Noble. Here's where I need your help.

Xenial to FocalUpdate to Morph

clickable.json to yaml
- this is pretty bad, is there a better way of doing this? I mean, with the default tool in the system, without an external tool
- it will leave a dirty yaml file that need to be edited by hand
Framework update
- I think this is ok
- I'm sure I forgot something here
Update to Morph
- Probably ok
Update imports and Ubuntu qml items
- It turned out pretty well

gpatel-fr

@CiberSheep said in Script for updating apps [help needed]:

it will leave a dirty yaml file that need to be edited by hand

I tried it on this project: https://github.com/august-alt/filemanager-app
and the result of your script could be loaded by the python yaml library, so what's wrong ? is the software used less lenient than the yaml python library, or is there more complicated clickable.json files ?

CiberSheep

@gpatel-fr said in Script for updating apps [help needed]:

or is there more complicated clickable.json files ?

Thanks for asking. Yep, when the json has nested elements it would fail?
For example applying the script to this json:

}
[...]
  "kill": "qmlscene",
  "dependencies_host" : [
	"qml-module-ubuntu-onlineaccounts2"
  ]
}

will leave a yml file similar to

[...]
  "kill": "qmlscene"
  "dependencies_host":
	"qml-module-ubuntu-onlineaccounts2"

and, I think, it should end with a list

"dependencies_host":
    - "qml-module-ubuntu-onlineaccounts2"

gpatel-fr

@CiberSheep said in Script for updating apps [help needed]:

with the default tool in the system, without an external tool

what's the 'system' in fact ? python as installed in 24.04 has json (of course, it's part of the 'batteries') and yaml is packaged too. So if the system is a phone under noble it has everything needed for a conversion.

If the job is done in a context where for some reason the yaml package is not present, there is always json. Having imported the json file a basic exporting 'by hand' should not be too difficult if one can afford to ignore handling the myriad of special cases that are certainly in the big yaml libraries.

This assumes that UT clickable.json files are always valid json.
If it can be false (comments springs to mind) a python solution will fail badly.

CiberSheep

@gpatel-fr said in Script for updating apps [help needed]:

@CiberSheep said in Script for updating apps [help needed]:

with the default tool in the system, without an external tool

what's the 'system' in fact ?

A Linux machine.

python as installed in 24.04 has json (of course, it's part of the 'batteries') and yaml is packaged too. So if the system is a phone under noble it has everything needed for a conversion.

I was thinking first a computer, but it is nice to think that UT could be the target. Is it ready now, though?

If the job is done in a context where for some reason the yaml package is not present, there is always json.
clickable.json is set as deprecated, if I remember that correctly

This assumes that UT clickable.json files are always valid json.

We are upgrading a working project. So, yes

CiberSheep

Ouh! actually it's a nice idea to use python... I have learn a trick now.

https://gitlab.com/cibersheep/update-apps/-/commit/2f625ff90ccfa94fb605512c6fede0c00199cf7f

This assumes pytho3 is installed in the system

gpatel-fr

@CiberSheep said in Script for updating apps [help needed]:

This assumes pytho3 is installed in the system

for most linux systems (ie fedora and debian derivatives and quite a few others) this is not a very adventurous assertion, however yaml is not included in standard python. If yaml is not a requirement pip installed automatically by running the script, this will fall back to search and replace and fail silently if the script is too complicated for the simple search and replace.

Personally I'd do a test to check if python is installed, if yes, test if we have yaml, if yes, run something like you did, if python but no yaml, run a script such as this one:

#! /usr/bin/env python3

import json
import os
import sys
import traceback
from collections import OrderedDict

def do_convert(p_item, offset, output_file):
    def outval(v):
        if isinstance(v, str):
            return "'" + v + "'"
        else:
            return str(v)

    if isinstance(p_item, dict):
        for k, v in p_item.items():
            if isinstance(v, list):
                if len(v) == 0:
                    output_file.write(offset + k + ': []\n')
                else:
                    output_file.write(offset + k + ':\n')
                    do_convert(v, offset.replace('-', ' ') + '- ', output_file)
            elif isinstance(v, dict):
                output_file.write(offset + k + ':\n')
                do_convert(v, offset.replace('-', ' ') + '  ', output_file)
            else:
                output_file.write(offset + k + ": " + outval(v) + "\n")
            offset = offset.replace('-', ' ')
    elif isinstance(p_item, list):
        for k in p_item:
            if isinstance(k, list):
                do_convert(k, offset, output_file)
            elif isinstance(k, dict):
                do_convert(k, offset, output_file)
            else:
                output_file.write(offset + outval(k) + "\n")

if __name__ == '__main__':
    try:
        j_string=open(sys.argv[1]).read()
        p_dict=json.loads(j_string, object_pairs_hook=OrderedDict)
        output_file = open('.'.join([os.path.splitext(sys.argv[1])[0], 'yml']), 'w')
        offset = ''
        do_convert(p_dict, offset, output_file)
        output_file.close()
        sys.exit(0)
    except Exception as e:
        print(e)
        traceback.print_exc()
        sys.exit(1)

that seems (not heavily tested...) to do a decent job converting more complex cases that the tinyl clickable.json files that I have seen and if not, print a warning and run the bash search and replace (that will still run fine in most cases)

CiberSheep

@gpatel-fr nice! Thanks.
I'll test your code