UBports Robot Logo UBports Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    • Register
    • Login

    How-To : Offline Unav maps

    Scheduled Pinned Locked Moved Support
    14 Posts 6 Posters 2.6k Views 7 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
      Reply
      • Reply as topic
      Log in to reply
      This topic has been deleted. Only users with topic management privileges can see it.
      • E Offline
        ernest
        last edited by ernest

        A quick how-to on getting offline maps for Unav.

        Recently i needed to update my offline map and everytime it's quit a mess to setup an OpenstreetMap local server, lost of hours lost on that. I recently found a docker instance perfect for our needs.

        What we need in this how-to

        • A linux computer (of-course) with disk space
        • Snap of Jtiledownloader installed
        • docker.io installed

        Optional

        • A nextcloud instance
        • desktop nextcloud sync software
        • Ubsync.

        First of all : setting up tile server.

        Not much to say, except follow the work of this great Github repository :
        https://github.com/Overv/openstreetmap-tile-server

        docker volume create openstreetmap-data
        
        git clone https://github.com/Overv/openstreetmap-tile-server.git
        
        cd openstreetmap-tile-server/
        

        What i've done :
        docker volume create openstreetmap-data
        download the appropriate pbf there : http://download.geofabrik.de/

        docker run -v /absolute/path/to/luxembourg.osm.pbf:/data.osm.pbf -v openstreetmap-data:/var/lib/postgresql/10/main overv/openstreetmap-tile-server import
        

        If lucky enough then

        docker run -p 80:80 -v openstreetmap-data:/var/lib/postgresql/10/main -d overv/openstreetmap-tile-server run
        

        Std output :

        root@T450s:/home/ernest# docker run -v /home/ernest/Documents/texas-latest.osm.pbf:/data.osm.pbf -v openstreetmap-data:/var/lib/postgresql/10/main overv/openstreetmap-tile-server import
        Unable to find image 'overv/openstreetmap-tile-server:latest' locally
        latest: Pulling from overv/openstreetmap-tile-server
        898c46f3b1a1: Pull complete 
        63366dfa0a50: Pull complete 
        041d4cd74a92: Pull complete 
        6e1bee0f8701: Pull complete 
        b3d49b2dd661: Pull complete 
        7f7fd75ed7c3: Pull complete 
        ee5c945e49f2: Pull complete 
        808e9c0fdf3b: Pull complete 
        d780496bb933: Pull complete 
        52ddc01564c5: Pull complete 
        98df30b28f52: Pull complete 
        6806a290c5cc: Pull complete 
        fe5da1ed6f5d: Pull complete 
        c2b03f1243c3: Pull complete 
        892f03c6da75: Pull complete 
        60740ad09b89: Pull complete 
        510cf260d67d: Pull complete 
        e5cf11ef9535: Pull complete 
        327759077133: Pull complete 
        29fd64d01acc: Pull complete 
        6529fd3cf6e4: Pull complete 
        7b31de690592: Pull complete 
        42ed3f4d774a: Pull complete 
        8c1ac07d887f: Pull complete 
        9a8a920e2d77: Pull complete 
        e8e3c66eb441: Pull complete 
        8bcaa423208c: Pull complete 
        a64871ac5a50: Pull complete 
        Digest: sha256:f8c58ef99819eb73adfaee91401b7f23a878f29122fae285734912657f5082da
        Status: Downloaded newer image for overv/openstreetmap-tile-server:latest
        + '[' 1 -ne 1 ']'
        + '[' import = import ']'
        + CreatePostgressqlConfig
        + cp /etc/postgresql/10/main/postgresql.custom.conf.tmpl /etc/postgresql/10/main/postgresql.custom.conf
        + sudo -u postgres echo 'autovacuum = on'
        + cat /etc/postgresql/10/main/postgresql.custom.conf
        # Suggested minimal settings from
        # https://ircama.github.io/osm-carto-tutorials/tile-server-ubuntu/
        
        shared_buffers = 128MB
        min_wal_size = 1GB
        max_wal_size = 2GB
        maintenance_work_mem = 256MB
        
        # Suggested settings from
        # https://github.com/openstreetmap/chef/blob/master/roles/tile.rb#L38-L45
        
        max_connections = 250
        temp_buffers = 32MB
        work_mem = 128MB
        wal_buffers = 1024kB
        wal_writer_delay = 500ms
        commit_delay = 10000
        # checkpoint_segments = 60 # unrecognized in psql 10.7.1
        max_wal_size = 2880MB
        random_page_cost = 1.1
        track_activity_query_size = 16384
        autovacuum_vacuum_scale_factor = 0.05
        autovacuum_analyze_scale_factor = 0.02
        autovacuum = on
        + service postgresql start
         * Starting PostgreSQL 10 database server
           ...done.
        + sudo -u postgres createuser renderer
        + sudo -u postgres createdb -E UTF8 -O renderer gis
        + sudo -u postgres psql -d gis -c 'CREATE EXTENSION postgis;'
        CREATE EXTENSION
        + sudo -u postgres psql -d gis -c 'CREATE EXTENSION hstore;'
        CREATE EXTENSION
        + sudo -u postgres psql -d gis -c 'ALTER TABLE geometry_columns OWNER TO renderer;'
        ALTER TABLE
        + sudo -u postgres psql -d gis -c 'ALTER TABLE spatial_ref_sys OWNER TO renderer;'
        ALTER TABLE
        + '[' '!' -f /data.osm.pbf ']'
        + sudo -u renderer osm2pgsql -d gis --create --slim -G --hstore --tag-transform-script /home/renderer/src/openstreetmap-carto/openstreetmap-carto.lua -C 2048 --number-processes 4 -S /home/renderer/src/openstreetmap-carto/openstreetmap-carto.style /data.osm.pbf
        osm2pgsql version 0.96.0 (64 bit id space)
        
        Allocating memory for dense node cache
        Allocating dense node cache in one big chunk
        Allocating memory for sparse node cache
        Sharing dense sparse
        Node-cache: cache=2048MB, maxblocks=32768*65536, allocation method=11
        Mid: pgsql, cache=2048
        Setting up table: planet_osm_nodes
        Setting up table: planet_osm_ways
        Setting up table: planet_osm_rels
        Using lua based tag processing pipeline with script /home/renderer/src/openstreetmap-carto/openstreetmap-carto.lua
        Using projection SRS 3857 (Spherical Mercator)
        Setting up table: planet_osm_point
        Setting up table: planet_osm_line
        Setting up table: planet_osm_polygon
        Setting up table: planet_osm_roads
        
        Reading in file: /data.osm.pbf
        Using PBF parser.
        Processing: Node(44188k 237.6k/s) Way(4610k 12.36k/s) Relation(26670 952.50/s)  parse time: 587s
        Node stats: total(44188377), max(6434488285) in 186s
        Way stats: total(4610012), max(686484151) in 373s
        Relation stats: total(26922), max(9523915) in 28s
        Sorting data and creating indexes for planet_osm_point
        Sorting data and creating indexes for planet_osm_line
        Sorting data and creating indexes for planet_osm_polygon
        Sorting data and creating indexes for planet_osm_roads
        Copying planet_osm_roads to cluster by geometry finished
        Creating geometry index on planet_osm_roads
        Copying planet_osm_point to cluster by geometry finished
        Creating geometry index on planet_osm_point
        Creating osm_id index on planet_osm_roads
        Creating indexes on planet_osm_roads finished
        All indexes on planet_osm_roads created in 20s
        Completed planet_osm_roads
        Stopping table: planet_osm_nodes
        Stopped table: planet_osm_nodes in 0s
        Stopping table: planet_osm_ways
        Building index on table: planet_osm_ways
        Creating osm_id index on planet_osm_point
        Creating indexes on planet_osm_point finished
        All indexes on planet_osm_point created in 75s
        Completed planet_osm_point
        Stopping table: planet_osm_rels
        Building index on table: planet_osm_rels
        Stopped table: planet_osm_rels in 4s
        Copying planet_osm_line to cluster by geometry finished
        Creating geometry index on planet_osm_line
        Creating osm_id index on planet_osm_line
        Creating indexes on planet_osm_line finished
        All indexes on planet_osm_line created in 218s
        Completed planet_osm_line
        Copying planet_osm_polygon to cluster by geometry finished
        Creating geometry index on planet_osm_polygon
        Creating osm_id index on planet_osm_polygon
        Creating indexes on planet_osm_polygon finished
        All indexes on planet_osm_polygon created in 344s
        Completed planet_osm_polygon
        Stopped table: planet_osm_ways in 385s
        
        Osm2pgsql took 993s overall
        node cache: stored: 44188377(100.00%), storage efficiency: 57.43% (dense blocks: 3386, sparse nodes: 24604874), hit rate: 100.00%
        + service postgresql stop
         * Stopping PostgreSQL 10 database server
           ...done.
        + exit 0
        root@T450s:/home/ernest# sudo docker run -p 80:80 -v openstreetmap-data:/var/lib/postgresql/10/main -d overv/openstreetmap-tile-server run
        
        

        and open in your browser : http://localhost/ done you have a tile server running.
        8414b7e9-a729-4dd4-bd1e-f3a7ea8cd23e-image.png

        I'm not sure how long the docker instance last and may need to be regenerated each time, fine for me I'm using it every 6 month. Please see the github info, if you want to have an docker instance lasting.

        Second step : downloading the tile, for that you need Jtiledownloader

        See : http://unav.me/offline/ and the snap store
        a191d9e2-2ca7-4800-aa70-36e2f098256c-image.png
        f2fb655c-e491-4e00-a266-0888ab5bece1-image.png
        The configuration is pretty simple : as server enter http://localhost:80/tile. then open "slippy map chooser" you should see the map.
        Then select the area of interest (defined with left click), set tile zoom level to 10,11,12,13,14,15,16,17 and click download and wait.
        56f8c977-2c75-4cea-b2e8-ab139068b2ad-image.png

        Third step :

        transfert to the phone... this is up to you, define as the step 6 there : http://unav.me/offline/
        I'm suggesting to zip the folder and extract it in the phone.
        Personnally, i'm synchronizing my Jtiledownload folder archive to my nextcloud instance and then this instance to my utouch folder, it's longer but "automatic" update.

        I hope it will help to reduce your data consumption 🙂

        Restart from scratch

        Following thise link to cleanup docker image, container and volume
        https://www.digitalocean.com/community/tutorials/how-to-remove-docker-images-containers-and-volumes

        R 1 Reply Last reply Reply Quote 7
        • R Offline
          ricardo65 @ernest
          last edited by

          @ernest Good evening, I'm trying to install unav offline the first step and nothing comes out. I could have a hand. I have it installed, but only my city since the download by jtiledownloader takes a lot and it takes a long time to download it.

          1 Reply Last reply Reply Quote 0
          • E Offline
            ernest
            last edited by

            I've added two steps to clone the git repository.

            Nothing happens because you were missing the git most likely.

            1 Reply Last reply Reply Quote 0
            • R Offline
              ricardo65
              last edited by

              I want to install the docker and I get this:Captura de pantalla de 2019-05-22 23-28-49.png

              1 Reply Last reply Reply Quote 0
              • R Offline
                ricardo65
                last edited by

                Could you help me install the one in Spain?

                1 Reply Last reply Reply Quote 0
                • G Offline
                  Giiba
                  last edited by Giiba

                  I recently found that foxtrotgps runs fine in a Libertine shell, and more importantly the files are compatible with uNav.

                  This can all be done on the phone, or just using foxtrot on a pc.

                  I think the better question is why does uNav not cache downloaded tiles? If that happened most users would be fine downloading tiles on-the-go; the current arrangement eats mobile data unnecessarily.

                  G 1 Reply Last reply Reply Quote 0
                  • G Offline
                    guru @Giiba
                    last edited by

                    I download Openstreetmap tiles in advance for my interested regions (Munich and Havana) with the resolution 11-18 to my laptop and update them there from time to time, tar+SCP copy them to the SD of my BQ E4.5, start a small local Python web server in the phone and have a patched uNav app which ask this local webserver and not the Internet for the tiles. I can post the exact details if someone is interested.

                    Matthias

                    T G 2 Replies Last reply Reply Quote 1
                    • T Offline
                      trainailleur @guru
                      last edited by

                      @guru said in How-To : Offline Unav maps:

                      I download Openstreetmap tiles in advance for my interested regions (Munich and Havana) with the resolution 11-18 to my laptop and update them there from time to time, tar+SCP copy them to the SD of my BQ E4.5, start a small local Python web server in the phone and have a patched uNav app which ask this local webserver and not the Internet for the tiles. I can post the exact details if someone is interested.

                      Please do. This sounds extremely useful.

                      G 1 Reply Last reply Reply Quote 0
                      • G Offline
                        Giiba @guru
                        last edited by

                        I'm curious if there's a reason to use a web server vs a symlink to the storage location...

                        ie.
                        I symlinked foxtrotgps to download to ~/Downloads/maps
                        and symlinked uNav to read from ~/Downloads/maps/OSM

                        This seems simpler with no overhead to me, but I've no idea what the advantages of a tile server might be.

                        G 1 Reply Last reply Reply Quote 1
                        • G Offline
                          guru @trainailleur
                          last edited by

                          OK, here comes the first part of the howto:

                          How To Use OSM Tiles Offline with uNav
                          guru@unixarea.de

                          The problem

                          The app for UbuntuTouch "uNav" works only with Internet connection when presenting maps and locations. Sometimes Internet is not reliable or over data mobile even expensive and the idea to workaround is having the maps' tiles prefetched and stored in the Ubuntu phone. Here is the tool chain so solve this.

                          1. Prefetching OSM tiles for a given area of interest and

                          There is a C-written tool 'osmtiles' to be fetched an compiled
                          on your Linux or FreeBSD workstation: http://www.millions.ca/~stacy/osmtiles.tgz
                          Compilation is simple (see its README):

                              tar xzf osmtiles.tgz
                              cd osmtiles
                              cc -o osmtiles osmtiles.c -lm
                          

                          The tool 'osmtiles' allows to make a list of all OSM tiles of a given GPS coordinates of an area, for example:

                               osmtiles -z 9 -Z 18                \
                                  23.178555 -82.462692            \
                                  23.050039 -82.288628            \
                                  > havana.txt 
                          

                          -z 9 and -Z 18 give the nivel of the tiles, from 9 to 18. The above region (23.178555 -82.462692) x (23.050039 -82.288628) are the GPS coordinates of a rectangle of the capital of Cuba, Havana. The output of the tool is a list file like this:

                              9/138/222.png
                              10/277/444.png
                              11/554/888.png
                              11/554/889.png
                              11/555/888.png
                              11/555/889.png
                              12/1109/1776.png
                              ...
                          

                          Some ~18.000 lines, i.e. tiles to fetch. This list is stored for fetch (and later updates) in a file, let's say havana.txt. I have a bunch of such files for the places I visit.

                          Based on this list I have a shell script which in principle does for any of the files in the list:

                              while read name ; do
                                  fetch the MD5 sum from the server for the file http://tile.openstreetmap.org/$name
                          	if the MD5 is different from what I have
                          	    fetch the file http://tile.openstreetmap.org/$name
                              done < havana.txt
                          

                          This way, later on updates, only modified tiles will be fetched.
                          I can share this script, no problem, if you promise not to blame
                          me, but send patches for it.

                          Next step is making, again based on the file of the place to visit, a tar archive:

                              cd osm
                              tar --files-from=../Havana.lst -czf ../Havana.tgz
                              ls -lh ../Havana.tgz
                              -rw-r--r--  1 guru  wheel    93M 24 may.  16:58 ../Havana.tgz
                          

                          i.e. such a file is around 100 MByte in size.

                          Move these file(s) to the phone with SCP or ADB and unpack them on the SD card to some directory which will later be served by a small Python web server.

                          1. The modifications in the phone

                          (TO BE CONTINUED)

                          1 Reply Last reply Reply Quote 1
                          • G Offline
                            guru @Giiba
                            last edited by

                            @Giiba said in How-To : Offline Unav maps:

                            I'm curious if there's a reason to use a web server vs a symlink to the storage location...

                            ie.
                            I symlinked foxtrotgps to download to ~/Downloads/maps
                            and symlinked uNav to read from ~/Downloads/maps/OSM

                            This seems simpler with no overhead to me, but I've no idea what the advantages of a tile server might be.

                            The big advantage is that the change in uNav is one single line: substituting the URL of the original web server by http://localhost:8000 and al is done once you have the tiles fetched.

                            And, fetching tiles with foxtrotgps, i.e. by hand when visiting the place with foxtrotgps is not a solution for the problem when you need tiles and have no Internet connection.

                            1 Reply Last reply Reply Quote 0
                            • G Offline
                              guru
                              last edited by

                              Part 2:

                              1. The modifications in the phone

                              We start with copy over the file Havana.tgz to the phone and unpack it there:

                                  scp -p Havana.tar.gz phablet@192.168.2.102:.
                                  ssh phablet@192.168.2.102
                              
                                  ls /media/phablet/
                                  9CC0-D6CE
                                  mkdir /media/phablet/9CC0-D6CE/Maps
                                  cd /media/phablet/9CC0-D6CE/Maps
                                  tar xzf ~/Havana.tar.gz
                              

                              Note: the dir 9CC0-D6CE may vary depending on the SD type. Change this to the name in your phone and later in the start script for the Python web server too. Unpacking the tar archive should give:

                                  ls /media/phablet/9CC0-D6CE/Maps/
                                  osm
                                  ls /media/phablet/9CC0-D6CE/Maps/osm
                                  10  11  12  13  14  15  16  17  18  9
                              
                              

                              i.e. the tiles in their sub-dirs 9 ... 18.

                              To start our special uNav-osm app we need a new desktop entry as ~/.local/share/applications/unav-osm.desktop which must contain the following lines:

                                  [Desktop Entry]
                                  Name=unav-osm
                                  Type=Application
                                  Exec=/usr/bin/nohup /home/phablet/unav.sh
                                  Icon=/userdata/system-data/opt/click.ubuntu.com/navigator.costales/current/icon.png
                                  Terminal=false
                                  X-Ubuntu-Touch=true
                              

                              As you see it will not execute the uNav app directly, but a small shell script /home/phablet/unav.sh. This script starts upfront the Python web service and then the uNav app itself. It has the folling few lines.

                                  #!/bin/sh
                                  #
                                  cd /media/phablet/9CC0-D6CE/Maps
                                  python3 -m http.server 8888 &
                                  #
                                  cd /userdata/system-data/opt/click.ubuntu.com/navigator.costales/current
                                  qmlscene %u qml/Main.qml > /dev/null 2> /dev/null
                              

                              Create this script /home/phablet/unav.sh with vi and make it executable:

                                  vi /home/phablet/unav.sh
                                  chmod 0755 /home/phablet/unav.sh
                              

                              Last action is to make a small modification in uNav itself. At the time of writing it is the version 2.3 and the fix must be done in this file: /userdata/system-data/opt/click.ubuntu.com/navigator.costales/current/nav/index.html
                              What to change is best visible in this diff:

                                  cd /userdata/system-data/opt/click.ubuntu.com/navigator.costales/current/nav
                                  diff index.html index.html.orig
                              210c210
                              < 		source: new ol.source.OSM({url: 'http://localhost:8888/osm/{z}/{x}/{y}.png'}),
                              ---
                              > 		source: new ol.source.OSM({url: 'http://{a-c}.basemaps.cartocdn.com/rastertiles/voyager/{z}/{x}/{y}.png'}),
                              
                              

                              The file must be modified as root and it's a good idea to make a copy (like I did):

                                  sudo su
                                  cd /userdata/system-data/opt/click.ubuntu.com/navigator.costales/current/nav
                                  cp -p index.html index.html.orig
                                  vi index.html
                              

                              Note: the URL http://{a-c}.basemaps.cartocdn.com/rastertiles/... appears three times in the file. Change only the first location for /voyager/.... because this is also
                              what you want to pick-up in the Settings menu of uNav for the value:

                                  Mode
                                  Online
                              
                                  Online style
                                  Carto Voyager        <---------------
                              

                              That's it as changes.

                              One final note: You have only offline tiles for small places and not the world between. So, you can't slide over the world, for example from Munich to Havana. If you leave your location of offline tiles you will see only a white map in uNav.
                              The way to move is configure in Settings some other Online style, for example Mapbox. Or configure locations as Favourites to jump over the Ocean.

                              For questions just ping me or write me to guru@unixarea.de

                              Matthias

                              R 1 Reply Last reply Reply Quote 2
                              • R Offline
                                ricardo65 @guru
                                last edited by

                                @guru good, I've seen your explanation but I see it very complicated I in this of many commands I'm lost, my question would be: could do all Spain or otherwise as would be done by parts.I currently according to the site I want to go under me tiles by jtiledownloader and then I pass them to the cell phone but it is a huge job, if you could give me a hand I would thank you.

                                1 Reply Last reply Reply Quote 0
                                • joniusJ Offline
                                  jonius
                                  last edited by

                                  Just for the record: Offline Maping is possible with Pure Maps by installing OMS Scout Server.

                                  1 Reply Last reply Reply Quote 0
                                  • First post
                                    Last post