Geo Web Crawler

General description: This project is an ongoing project for crawling and indexing places that are hard to be found!

Contributor: Kazem Jahanbakhsh

Implementation period: March 2012 - present

geo crawler world

Main Idea:
In the last decade, we have seen how several companies such as Google, Microsoft, and Yahoo use web crawler to search and index web graph. The result of this effort is several serach engine for information retrieval. We have also witnessed how companies started using map services in order to make places in cities searchable. The basic idea is to drive a car equipped with special cameras and gps to capture images from streets and then use these images for creaing street maps.

These days everybody carries a smartphone or an advanced camera (equipped with gps). Thus, people who are equipped with advanced digital devices can be considered as mobile sensors. These people volunteerly visit and explore different locations in cities and share their collceted data with others. One example is Flickr website where people share their taken photos with other people publicly. Thus, we can think of people as crawlers who explore different locations for free. Using this available information, we can answer several location-based search queries which other map services such as Google map are not able to answer.

To give a few examples, suppose you want to show the locations of "drinking fountains", "sculptures", or "location of sites with graffiti" in a city on a map. If we use Google map to answer these questions, we may not find the things that we are looking for. However, we can use Flickr or other similar websites to answer these questions. Here, I have used Flickr API to search locations (lat/lon) of all "outdoor sculptures" in New York City. I have shown the final results of my search on a Google map below. This example illustrates one type of query which Google map fails to find the answer whereas people are able asnwer it by sharing their collected photos.

You can follow me here on Twitter Follow @kjahanbakhsh