Web scraping, like other things, is part skill, part investigation. Websites are built in a myriad of different ways, some are very simple, others are complex dynamic beasts. In the vast vast world of the Internet, correctly done, small referral fees can add up very quickly into handsome bottom lines. These companies make profit by getting a small referral fee for driving a customer to a particular website. Web scraping is heavily used by companies involved for example in the price and product comparison business. Web scraping is carried out for a wide variety of reasons, but mostly because the data is not available through easier means. The job of web-scraping is to go under the hood, and extract data from websites using code automation, so that we can get it into a format we can work with. Normally, it is mashed up and mixed in a blend of CSS and HTML. The problem with data in websites however is that generally, the data is not presented to us in an easy to get at manner. however, there is a huge volume of data out there that's not available to us in these nice easily parsable formats and a lot of that data resides and is presented to us via websites. When we think of different sources of data, we generally think about structured or semi-structured data presented to us in SQL, Web-services, CSV, etc.

Part three - Web scraping with C# - point and scrape!.

Part one - How to web scrape using C# (this article).

This article is part one of a four part series. Download sample client - no EXE - 28.6 KB.

Streetsmart edge api