How to: Use custom web scraping with Home Assistant

Introduction

I get asked about this integration a lot, how does my condo know what the TSA wait times are!?! Well in this post I will show you how to setup a scrape integration to pull the data from a webpage into Home Assistant.

TL;DR - A quick how to of importing custom data into Home Assistant which I use for TSA wait times and more!

Requirements

Personal Home Assistant installation

Using Web Scraping

Disclaimer: Some websites frown upon this as you are not visiting their website and they are unable to generate advertisement revenue using this methodology so please use this sparingly.

In this example I will setup a TSA wait time sensor so that we will know how long the TSA wait time is. With this information I can add it to my travel time, or keep a historical view of it, or whatever else your mind desires with this data. But for my needs I just want to know how much time I should add to my commute when I go to the airport.

TSA Wait time website

I found this website (https://www.tsawaittimes.com) that aggregates quite a few of the TSA wait times for several airports around the United States. You can also check to see if there is one specific for your airport by googling 'TSA wait time <fill in your airport>' and seeing what comes back. I did find that my local airport had it's own website for this data as well.

In Home Assistant

1) We need to add our Home Assistant - Scrape Integration sensor. And pretty much that's it.
Note: If you want to have action in an automation you will need to turn this value into a number. For instance if you want to trigger a light to go off when you need to leave you will need to add a sensor that will regular expression the minutes and turn that into a value. I might add this functionality later and update this blog post but for now the information is enough.

Source Code for my TSA integration

- platform: scrape
  resource: !secret tsa_airport_information_url
  select: !secret tsa_selector
  name: TSA wait time

Breaking down these secret's:

resource: the website that you want to scrape
example: https://www.tsawaittimes.com/security-wait-times/LAX/Los-Angeles-International
select: the value you want to read
example: "body > div.container > div.card.card-body.toptop.bigbottom.p-4 > div > div:nth-child(2) > div.card.toptop > div > p.center.bold"

How to figure out the selector?

This is probably the most complicated part of web scraping so I will go slow. The value that needs to go here is what is known as the CSS (Cascading Style Sheet) selector. If you are trying to generate this value by yourself here is a reference, but there is a much easier way to do it.

1) Navigate to the webpage that you want to visit
Example: https://www.tsawaittimes.com/security-wait-times/LAX/Los-Angeles-International

2) Right click on the text or element you want to pull into Home Assistant
Example: 7 minutes and 36 seconds*

And click on Inspect (all browsers should have this)

3) A window / side panel should popup that looks like this

4) All you need to do now is right click on the highlighted row, all browsers will be similar, go to copy, then copy selector (also CSS selector in some browsers)

5) Now in your clipboard you should have the following:
This is the selector value

body > div.container > div.card.card-body.toptop.bigbottom.p-4 > div > div:nth-child(2) > div.card.toptop > div > p.center.bold

6) In Home Assistant I surround it with double quotes but that's it, this will now scrape the value (and if you want to scrape it every so often look at the option scan_interval for the component)

Home Assistant sensor

Once you reload your Home Assistant, you will see a new sensor on the development page. Now you can use this sensor for whatever you like!

Conclusion

Sometimes there isn't an already built integration in Home Assistant and you have to resort to desperate measures. I use this for reading the status of a website that I use frequently. The examples on the integration page I find are also pretty useful.

Getting the price of energy from your provider's webpage
Getting the latest podcast title
Reading the status that is displayed on a particular website

When I put my house on vacation mode it will push me a notification, it will put the travel time to my airport as well as the TSA wait time so that I have adequate planning time to get to the airport.