Scrape your Competitor's Websites with Advanced Web Scraper

 In this post, we will go over the details of our latest project Advanced Web Scraper for H&M Germany Knime Workflow.

   This workflow, connects to the H&M Germany website and gets the product category and sub category information, as well as all the product page URLs and price information to aggregate at certain hierarchies. 

   This template can be used for other retailers. However, since the design of the website will be different than this, there has to be some changes required. 

Click here to Download this workflow from Official Knime Page

   For each category in H&M website, we will replicate the steps. However, when website gets UI updates, there might be some code changes required as well. 

   First, we will use Webpage Retriever and Xpath nodes to connect to the website and get the product categories.  Then, we will do some transformation to get the price and format the data for our reports.

   We will also calculate the product count at each prive level per category to see how are the prices distributed per categories.  After these transformations, data will be ready to sent to Power BI for further reporting.

By using metanodes, we can wrap and run all of these with just one click and the data will be ready for our PowerBI dataset. 

We can also export results to Excel like below. 

If you liked this project, please don't forget to share and leave a comment below!

Share: