How to use Python and Selenium to scrape websites

Web scraping has been used to extract data from websites almost from the time the World Wide Web was born. In the early days, scraping was mainly done on static pages – those with known elements, tags, and data. More recently, however, advanced technologies in web development have made the task a bit more difficult. In this article, we’ll explore how we might go about scraping data in the case that new technology and other factors prevent standard scraping. Traditional data scraping As most websites produce pages meant for human readability rather than automated reading, web scraping mainly consisted of…

This story continues at The Next Web