Web Crawler

Author

Michael Schmidt, Governors State University

Publication Date

Fall 2015

Document Type

Project Summary

Degree Name

Master of Science

Department

Computer Science

First Advisor

Soon-Ok Park, Ph.D.

Second Advisor

(Clare) Xueqing Tang, Ph.D.

Third Advisor

Neng-Shin Chen, M.S.

Abstract

A web crawler is a piece of code that travels the Internet and collects data from various web pages, also known as web scraping. Some web crawlers are autonomous and require no instructions once started. This project will focus on a user driven web crawler where user input will direct where the crawler goes and how the collected data is analyzed. Web scraping replaces the need for manual data entry and more easily reveals trends among data collected. It can also aggregate information from multiple sources into one central location. While this application provides three specific examples of web crawling/scraping, it could be easily altered to better suit additional markets and/or needs.

Comments

GSU logo has been redacted from the title page by OPUS staff.

Recommended Citation

Schmidt, Michael, "Web Crawler" (2015). All Capstone Projects. 148.
https://opus.govst.edu/capstones/148