2017-03-24Konferenzveröffentlichung DOI: 10.18452/1447
A Generic News Crawler and Extractor
The amount of news published and read online has increased tremendously in recent years, making news data an interesting resource for many research disciplines, such as the social sciences and linguistics. However, large scale collection of news data is cumbersome due to a lack of generic tools for crawling and extracting such data. We present news-please, a generic, multilanguage, open-source crawler and extractor for news that works out-of-thebox for a large variety of news websites. Our system allows crawling arbitrary news websites and extracting the major elements of news articles on those websites, i.e., title, lead paragraph, main content, publication date, author, and main image. Compared to existing tools, news-please features full website extraction requiring only the root URL.
Dateien zu dieser Publikation
Is Part Of Series: Everything Changes, Everything Stays theSame? Understanding Information Spaces. Proceedings of the 15th InternationalSymposium of Information Science (ISI 2017), isi2017, 13.03.2017 - 15.03.2017, Berlin, pp 218-223