Volume 9 Number 1(Jun. 2017)
Home > Archive > 2017 > Volume 9 Number 1(Jun. 2017) >
IJCEE 2017 Vol.9(1):330-342 ISSN: 1793-8163
DOI: 10.17706/IJCEE.2017.9.1.330-342

Priority Queue Based Estimation of Importance of Web Pages for Web Crawlers

Mohammed Rashad Baker, M. Ali Akcayol
Abstract—Abstract: There are hundreds of new web pages that are added daily to web directories. Web crawlers are developing over the same time of web pages growing up rapidly. Thus, the need for an efficient web crawler that deals with most of the web pages. Most of the web crawlers do not have the ability to visit and parse pages using URLs. In this study, a new web crawler algorithm has been developed using the priority queue. URLs, in crawled web pages, have been divided into inter domain links and intra domain links. The algorithm sets weight to these hyperlinks according to the type of links and stores links in the priority queue. Experimental results show that the developed algorithm gives a well crawled performance against unreached crawled web pages. In addition, the developed algorithm has a good capability to eliminate duplicated URLs.

Index Terms—Key words: Web crawler, page importance, link priority, priority queue.

Computer Engineering Department, Gazi University, Ankara, Turkey.

Cite:Mohammed Rashad Baker, M. Ali Akcayol, "Priority Queue Based Estimation of Importance of Web Pages for Web Crawlers," International Journal of Computer and Electrical Engineering vol. 9, no. 1, pp. 330-342, 2017.

General Information

ISSN: 1793-8163
Frequency: Semiyearly
Editor-in-Chief: Prof. Yucong Duan
Abstracting/ Indexing: EI (INSPEC, IET), Ulrich's Periodicals Directory, Google Scholar, EBSCO, Engineering & Technology Digital Library, ProQuest, and Electronic Journals Library
E-mail: ijcee@iap.org

What's New

  • Jul 27, 2017 News!

    IJCEE Vol. 9, No. 1 is available online now.   [Click]

  • Jun 29, 2017 News!

    IJCEE Vol. 7, No. 6 has been indexed by EI (Inspec) Inspec, created by the Institution of Engineering and Tech.!   [Click]

  • Jun 29, 2017 News!

    IJCEE Vol. 7, No. 5 has been indexed by EI (Inspec) Inspec, created by the Institution of Engineering and Tech.!   [Click]

  • Jun 29, 2017 News!

    IJCEE Vol. 7, No. 4 has been indexed by EI (Inspec) Inspec, created by the Institution of Engineering and Tech.!   [Click]

  • Jun 29, 2017 News!

    IJCEE Vol. 7, No. 1 has been indexed by EI (Inspec) Inspec, created by the Institution of Engineering and Tech.!   [Click]

  • Read more>>