Volume 9 Number 1 (Jun. 2017)
Home > Archive > 2017 > Volume 9 Number 1 (Jun. 2017) >
IJCEE 2017 Vol.9(1):330-342 ISSN: 1793-8163
DOI: 10.17706/IJCEE.2017.9.1.330-342

Priority Queue Based Estimation of Importance of Web Pages for Web Crawlers

Mohammed Rashad Baker, M. Ali Akcayol
Abstract—Abstract: There are hundreds of new web pages that are added daily to web directories. Web crawlers are developing over the same time of web pages growing up rapidly. Thus, the need for an efficient web crawler that deals with most of the web pages. Most of the web crawlers do not have the ability to visit and parse pages using URLs. In this study, a new web crawler algorithm has been developed using the priority queue. URLs, in crawled web pages, have been divided into inter domain links and intra domain links. The algorithm sets weight to these hyperlinks according to the type of links and stores links in the priority queue. Experimental results show that the developed algorithm gives a well crawled performance against unreached crawled web pages. In addition, the developed algorithm has a good capability to eliminate duplicated URLs.

Index Terms—Key words: Web crawler, page importance, link priority, priority queue.

Computer Engineering Department, Gazi University, Ankara, Turkey.

Cite:Mohammed Rashad Baker, M. Ali Akcayol, "Priority Queue Based Estimation of Importance of Web Pages for Web Crawlers," International Journal of Computer and Electrical Engineering vol. 9, no. 1, pp. 330-342, 2017.

General Information

ISSN: 1793-8163 (Print)
Abbreviated Title: Int. J. Comput. Electr. Eng.
Frequency: Quarterly
Editor-in-Chief: Prof. Yucong Duan
Abstracting/ Indexing: INSPEC, Ulrich's Periodicals Directory, Google Scholar, EBSCO, ProQuest, and Electronic Journals Library
E-mail: ijcee@iap.org

What's New

  • Jun 03, 2019 News!

    IJCEE Vol. 9, No. 2 - Vol. 10, No. 2 have been indexed by EI (Inspec) Inspec, created by the Institution of Engineering and Tech.!   [Click]

  • May 13, 2020 News!

    IJCEE Vol 12, No 2 is available online now   [Click]

  • Mar 04, 2020 News!

    IJCEE Vol 12, No 1 is available online now   [Click]

  • Dec 11, 2019 News!

    The dois of published papers in Vol 11, No 4 have been validated by Crossref

  • Oct 11, 2019 News!

    IJCEE Vol 11, No 4 is available online now   [Click]

  • Read more>>