Web crawling and data protection: CNIL has issued a 180,000 EUR fine against a provider of automobile insurance policies for failure to adequately protect data in violation of GDPR, specifically citing disallowing web crawling as a way to protect personal data from wrongful access.

In particular the company :
  1. sent usernames and passwords in cleartext
  2. allowed users to access other users accounts
  3. allowed users’ accounts to be accessible by the general public when entering a URL or changing the last numbers in a URL

The compromised information included copies of driver’s licenses, registration cards, bank identification records and documents to determine whether a person had been subject to a license withdrawal or hit-and-run.

Key takeaways:
  • Don’t send passwords in cleartext.
  • Adopt a strong password policy.
  • Ensure access controls to information are limited and accurate.
  • Use a “robot.txt” or other means to disallow SEO and crawling by search engines of internal web pages containing sensitive information.