To scrape or not to scrape?

Have you ever wondered why you need to click on all the traffic lights in a series of images before you can access a website? In many cases this is to make sure you are an actual person and not an automated software looking to ‘scrape data’. 

Web scraping has become an increasingly popular practice in recent years. In this blog we look at what web scraping is, the legal position in the UK and the dangers associated with it. 

What is data scraping?

Data scrapers use software to automatically collect, or “scrape”, publicly available data from online sources, for example from websites like LinkedIn. They extract, aggregate and combine data from various sources and store it. This is usually invisible to everyone else besides the scraper.

The collected data can be used for many purposes, including contact scraping of email addresses to compile a mailing list, web scraping of the underlying code, competitor monitoring by collecting price information of competitors, or reputation monitoring of comments made on social media platforms or review sites. It is also often used for screen scraping, which allows the scraper to emulate a human end-user to extract publicly available data from websites on a large scale.

The legal position in the UK

While web scraping is itself not illegal, there are certain instances where it may become illegal depending on how and what the scraper does with the data gathered. 

The two most common claims that can be brought against data scrapers are breach of contract and IP infringement (specifically, database right infringement). Depending on the precise circumstances, it is possible that a data scraper could also infringe copyright or trade mark rights, data protection legislation and/or contravene the Computer Misuse Act 1990.

Breach of contract

To have a justified breach of contract claim, the owner of the website in question has to show that its terms and conditions of use are enforceable and have been breached. It is now very common for website terms and conditions to expressly prohibit data scraping (or equivalent activities). 

While there is no clear precedent on whether website terms form a binding contracts in the UK, in a 2015 case in the European Court of Justice has held that website operators can prohibit ‘screen scraping’ of unprotected data via their website terms and conditions, in a case involving Ryanair. This means that website operators can set contractual restrictions that prohibit other businesses from ‘scraping’ information from their sites if they cannot otherwise rely on intellectual property rights giving them protection against unauthorised use of that data. 

Database right infringement

Determining whether there is also a database right infringement claim is also highly specific to the facts of each case. This will depend on: 

  • the type and volume of data that is being extracted;
  • the frequency with which the data is being extracted; and
  • the level of investment that was required to develop the database from which the data is being extracted.

The scraped data may comprise copyrighted work and therefore can amount to a copyright infringement or if it falls into the category of trade secrets for the purposes of the Trade Secrets Directive, this can provide the scraped website with another course of action.

Web Scraping and Data Protection

In the US a recent case involving LinkedIn, considered whether scraping publicly available information from social media websites constitutes a violation of the terms of use and privacy statements of the website. It was held that this is not illegal, despite the LinkedIn’s claims that this violates user privacy.

However, the position in Europe is very different considering we need to apply GDPR standards. Here it is unlikely that the data captured via standard scraping or other data-mining techniques is compatible with GDPR restrictions.

The Polish Supervisory Authority, the Polish equivalent of the Information Commissioner’s Office, has already imposed a fine on a company that processed contact data obtained from publicly available sources without informing the individuals concerned.

Ultimately, there is no simple yes or no answer to this question as the legality of web scraping depends on the exact circumstances of each case.

If you are interested in web scraping, to be on the safe side, you should always follow the terms and conditions of each of the websites. If they state that they do not allow web scraping, then we would recommend following this. Alternatively, you may be able to get permission to perform your business activities by contacting the website operators. 

While scraping can have some beneficial uses, such as aggregating prices of products or services for ease of comparison, it can also be misused, especially when it is used to access and store personal data.  

Ask a lawyer for advice on web scraping.

Janet Nikova
Latest posts by Janet Nikova (see all)