Abstract—Abstract: There are hundreds of new web pages that are added daily to web directories. Web crawlers are developing over the same time of web pages growing up rapidly. Thus, the need for an efficient web crawler that deals with most of the web pages. Most of the web crawlers do not have the ability to visit and parse pages using URLs. In this study, a new web crawler algorithm has been developed using the priority queue. URLs, in crawled web pages, have been divided into inter domain links and intra domain links. The algorithm sets weight to these hyperlinks according to the type of links and stores links in the priority queue. Experimental results show that the developed algorithm gives a well crawled performance against unreached crawled web pages. In addition, the developed algorithm has a good capability to eliminate duplicated URLs.

Index Terms—Key words: Web crawler, page importance, link priority, priority queue.

Computer Engineering Department, Gazi University, Ankara, Turkey.

Cite:Mohammed Rashad Baker, M. Ali Akcayol, "Priority Queue Based Estimation of Importance of Web Pages for Web Crawlers," International Journal of Computer and Electrical Engineering vol. 9, no. 1, pp. 330-342, 2017.