@lakotaubp Thanks.

The link was for GPS, not WIFI.

I did some digging, and turns out that without active mobile network card in the phone, AGPS will use a long time to get a fix.
This is because AGPS use cell tower fix to determine rough location, and then get help from internet to calculate a more accurate position. (Simple explanation, read more here: https://en.wikipedia.org/wiki/Assisted_GPS)

I put my phone card in the Nexus 6P, and even inside it took under 1 min for the unit to have a fix and position.

So now I only have WIFI to sort out. 🙂

Guys did you see, I got a working port now ^^