Web scraping is a fundamental skill that is extremely useful for data collection and automating tasks. The following examples will show how we scrape sites such as wrapbootstrap and themeforest to populate the HTML/CSS Theme Templates page. We will be using jsoup for DOM parsing and OkHttp for HTTP. Although jsoup is capable of handling HTTP for us we prefer to stick with OkHttp in case we need anything more complex than a simple GET request, such as special headers and cookies. Why learn two libraries when one will do?
We like to start simple, so we are only gathering four fields’ title, URL, image URL, and the number of downloads, if available.