Fractional pagerank crawler: Prioritizing URLs efficiently for crawling important pages early

Md Hijbul Alam; JongWoo Ha; Sang-Geun Lee

doi:10.1007/978-3-642-00887-0_52

Fractional pagerank crawler: Prioritizing URLs efficiently for crawling important pages early

Md Hijbul Alam, JongWoo Ha, Sang-Geun Lee

Center for Artificial Intelligence Research

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

4 Citations (Scopus)

Abstract

Crawling important pages early is a well studied problem. However, the availability of di.erent types of framework for publishing web content greatly increases the number of web pages. Therefore, the crawler should be fast enough to prioritize and download the important pages. As the importance of a page is not known before or during its download, the crawler needs a great deal of time to approximate the importance to prioritize the download of the web pages. In this research, we propose Fractional PageRank crawlers that prioritize the downloaded pages for the purpose of discovering important URLs early during the crawl. Our experiments demonstrate that they improve the running time dramatically while crawling the important pages early.

Original language	English
Title of host publication	Database Systems for Advanced Applications - 14th International Conference, DASFAA 2009, Proceedings
Pages	590-594
Number of pages	5
DOIs	https://doi.org/10.1007/978-3-642-00887-0_52
Publication status	Published - 2009
Event	14th International Conference on Database Systems for Advanced Applications, DASFAA 2009 - Brisbane, QLD, Australia Duration: 2009 Apr 21 → 2009 Apr 23

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	5463
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Other

Other	14th International Conference on Database Systems for Advanced Applications, DASFAA 2009
Country/Territory	Australia
City	Brisbane, QLD
Period	09/4/21 → 09/4/23

ASJC Scopus subject areas

Theoretical Computer Science
Computer Science(all)

Access to Document

10.1007/978-3-642-00887-0_52

Cite this

Alam, M. H., Ha, J., & Lee, S-G. (2009). Fractional pagerank crawler: Prioritizing URLs efficiently for crawling important pages early. In Database Systems for Advanced Applications - 14th International Conference, DASFAA 2009, Proceedings (pp. 590-594). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 5463). https://doi.org/10.1007/978-3-642-00887-0_52

Fractional pagerank crawler: Prioritizing URLs efficiently for crawling important pages early. / Alam, Md Hijbul; Ha, JongWoo; Lee, Sang-Geun.
Database Systems for Advanced Applications - 14th International Conference, DASFAA 2009, Proceedings. 2009. p. 590-594 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 5463).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Alam, MH, Ha, J & Lee, S-G 2009, Fractional pagerank crawler: Prioritizing URLs efficiently for crawling important pages early. in Database Systems for Advanced Applications - 14th International Conference, DASFAA 2009, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 5463, pp. 590-594, 14th International Conference on Database Systems for Advanced Applications, DASFAA 2009, Brisbane, QLD, Australia, 09/4/21. https://doi.org/10.1007/978-3-642-00887-0_52

Alam MH, Ha J, Lee S-G. Fractional pagerank crawler: Prioritizing URLs efficiently for crawling important pages early. In Database Systems for Advanced Applications - 14th International Conference, DASFAA 2009, Proceedings. 2009. p. 590-594. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-642-00887-0_52

Alam, Md Hijbul ; Ha, JongWoo ; Lee, Sang-Geun. / Fractional pagerank crawler : Prioritizing URLs efficiently for crawling important pages early. Database Systems for Advanced Applications - 14th International Conference, DASFAA 2009, Proceedings. 2009. pp. 590-594 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{4b9820078877434997caa13374112044,

title = "Fractional pagerank crawler: Prioritizing URLs efficiently for crawling important pages early",

abstract = "Crawling important pages early is a well studied problem. However, the availability of di.erent types of framework for publishing web content greatly increases the number of web pages. Therefore, the crawler should be fast enough to prioritize and download the important pages. As the importance of a page is not known before or during its download, the crawler needs a great deal of time to approximate the importance to prioritize the download of the web pages. In this research, we propose Fractional PageRank crawlers that prioritize the downloaded pages for the purpose of discovering important URLs early during the crawl. Our experiments demonstrate that they improve the running time dramatically while crawling the important pages early.",

author = "Alam, {Md Hijbul} and JongWoo Ha and Sang-Geun Lee",

year = "2009",

doi = "10.1007/978-3-642-00887-0_52",

language = "English",

isbn = "9783642008863",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

pages = "590--594",

booktitle = "Database Systems for Advanced Applications - 14th International Conference, DASFAA 2009, Proceedings",

note = "14th International Conference on Database Systems for Advanced Applications, DASFAA 2009 ; Conference date: 21-04-2009 Through 23-04-2009",

}

TY - GEN

T1 - Fractional pagerank crawler

T2 - 14th International Conference on Database Systems for Advanced Applications, DASFAA 2009

AU - Alam, Md Hijbul

AU - Ha, JongWoo

AU - Lee, Sang-Geun

PY - 2009

Y1 - 2009

N2 - Crawling important pages early is a well studied problem. However, the availability of di.erent types of framework for publishing web content greatly increases the number of web pages. Therefore, the crawler should be fast enough to prioritize and download the important pages. As the importance of a page is not known before or during its download, the crawler needs a great deal of time to approximate the importance to prioritize the download of the web pages. In this research, we propose Fractional PageRank crawlers that prioritize the downloaded pages for the purpose of discovering important URLs early during the crawl. Our experiments demonstrate that they improve the running time dramatically while crawling the important pages early.

AB - Crawling important pages early is a well studied problem. However, the availability of di.erent types of framework for publishing web content greatly increases the number of web pages. Therefore, the crawler should be fast enough to prioritize and download the important pages. As the importance of a page is not known before or during its download, the crawler needs a great deal of time to approximate the importance to prioritize the download of the web pages. In this research, we propose Fractional PageRank crawlers that prioritize the downloaded pages for the purpose of discovering important URLs early during the crawl. Our experiments demonstrate that they improve the running time dramatically while crawling the important pages early.

UR - http://www.scopus.com/inward/record.url?scp=67650099403&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=67650099403&partnerID=8YFLogxK

U2 - 10.1007/978-3-642-00887-0_52

DO - 10.1007/978-3-642-00887-0_52

M3 - Conference contribution

AN - SCOPUS:67650099403

SN - 9783642008863

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 590

EP - 594

BT - Database Systems for Advanced Applications - 14th International Conference, DASFAA 2009, Proceedings

Y2 - 21 April 2009 through 23 April 2009

ER -

Fractional pagerank crawler: Prioritizing URLs efficiently for crawling important pages early

Abstract

Publication series

Other

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this