F# and Home Assistant helping take the bins out on time

In this post I’m going to demonstrate how I use F# to retrieve data from a website (HTML), parse and expose it to Home Assistant which allows me to receive weekly notifictions on when to take my bins out for collection by my council.

This isn’t going to be a waste of time! (expect more bin puns)

The Problem

I’ve not got the best memory… Every week (on a Thursday) my local council (Coventry City Council) empty our (and our street’s) bins into a refuse truck. There are 3 bins for the house:

Blue = Recycling (cardboard, glass and certain plastics)
Green = General Waste (food waste and anything not recyclable)
Brown = Garden Waste (leaves, cuttings and grass etc.)

The council usually alternates between picking up the Blue & Brown one week, and the Green bin the next. Over Christmas the bins collected may change, sometimes randomly, and I usually (like I’m sure others do across the UK) wait for a neighbour to put their bin(s) out, copy them, and hope they checked and put out the correct bin.

This isn’t a great solution, it’s a house of cards waiting to fall. More than twice in recent times, the street has copied a neighbour who got it wrong. Resulting in none of our (incorrect) bins getting collected, and having to wait another week for the emptying of a bin that’s close to overflowing.

Unfortunately, there’s no recourse - you can’t lambaste your neighbour for getting it wrong when you didn’t check yourself - that’s trashy behaviour

YES……what I’ve described is a very lazy attitude to waste management. However, to find out which bin is being collected, on the Coventy City Council website above, is a pain. I’ll show a broad outline of the process/clicks below:

Go to website
Click burger menu and select “Environment and Waste”
On the new page, select “Rubbish and Recycling”
New page, Enter your street name and click “Search”
New page, select the street name you just entered
New page, click “Find out which bin will be collected when”
Success! (and another new page) Now search the page for the next upcoming date

It’s a bit of a faff, and sure, I could bookmark the page it takes me to. However, half of the problem is the fact that I also have to remember to check the website. You can order a printable version of the bin timetable from the council, but that’s not useful when the binmen strike and the whole schedule goes out of the window when they next come back.

My current solution is to set a weekly alarm for the evening before, to check/copy a neighbour and take the (hopefully) correct bin out.

I don’t want this as a long-term solution…..I refuse

The Ideal Solution

The ideal solution would be to have this information right in front of me, on my phone. Even better, would be to get notifications the night before about what bin is due to be collected.

Luckily, I’ve been running Home Assistant for over 3 years now. My wife and I both have the App on our phones, we’ve got lights, plugs and lots of devices in our house all hooked up to it. We’ve got about 10 automations at the moment which take care of things like turning off the lights at bedtime for example.

We use the Home Assistant frequently and it would be great to have this information front and centre. With Home Assistant, this is all possible. Let’s get to building it…

The Data Source

First off we’re going to need to get the data from somewhere. Unfortunately the Coventry City Council doesn’t have any JSON APIs that would allow me to easily retrieve this data.

I’ve been learning and experimenting with F# lately and I’m aware of a very powerful HTML parsing library, so let’s use that

F# to extract the data

Using the NuGet Package FSharp.Data it’s easy to download HTML from a URL and start working with the data on the page.

The page I’m working with looks like this: https://www.coventry.gov.uk/binsthursdayb

Using the F# code below, we download the HTML and start sifting through it to extract the data we need.

You can pop the below into a F# Project and call the function retrieveBinDates, print that out to the console and you’ll see the data nicely structured

namespace CoventryBinCollections

open System
open System.Globalization
open FSharp.Data

module CoventryBinDates =
    type BinType = Blue | Green | Brown | BlueAndBrown

    /// <summary>Holds a Date and the type of bin</summary>
    type BinCollection = {
        date: DateOnly
        bins: BinType
    }
    
    /// <summary>Holds a year and the HTML nodes containing collection dates for it</summary>
    type YearAndCollections = { Year: int; Collections: HtmlNode seq }
    
    /// <summary>
    /// Determines the "BinType" from the given string of text (directly from the Html) 
    /// </summary>
    /// <param name="htmlStr">The string of text, directly from the Coventry City Council Website</param>
    let private getBinType (htmlStr: string) =
        match htmlStr with
        | htmlStr when htmlStr.Contains("blue") && htmlStr.Contains("brown") -> Some BinType.BlueAndBrown
        | htmlStr when htmlStr.Contains("blue") -> Some BinType.Blue
        | htmlStr when htmlStr.Contains("green") -> Some BinType.Green
        | htmlStr when htmlStr.Contains("brown") -> Some BinType.Brown
        | _ -> None
    
    /// <summary>Returns a list of all Bin Dates (and their Bin Types) from the Coventry City Council
    /// website.</summary>
    /// <returns>The all bin dates available on the Coventry Council Website.</returns>
    let retrieveBinDates =
      let results = HtmlDocument.Load("https://www.coventry.gov.uk/binsthursdayb")
      
      results.Descendants ["div"]
      // This element is a nice identifiable ancestor of the dates 
      |> Seq.filter (fun x -> x.HasClass "widget-content")
      // Get the years and the child dates for that year
      |> Seq.map (fun x ->
                          let yearText = (x.Descendants ["h2"]|> Seq.head)
                                          .InnerText()
                                          .Trim()
                          let yearNum = Convert.ToInt32 (yearText.Substring(yearText.IndexOf(" ")))
                          { Year = yearNum; Collections = x.Descendants ["li"] }
      )
      |> Seq.collect (fun x ->
                          x.Collections
                          |> Seq.map (fun y ->
                                        let dateText = $"{y.InnerText().Split(':')[0]} {x.Year}"
                                        let date = DateOnly.Parse(dateText)
                                        let binType = getBinType (y.InnerText())
                                        {| date = date; binsOption = binType |}
                          )
                          |> Seq.filter (fun x -> x.binsOption.IsSome)
                          |> Seq.map (fun x -> { date = x.date; bins = x.binsOption.Value })
      )
      |> Seq.sortBy (fun x -> x.date)
      |> Seq.toList

F# to expose the data

Ok, we’ve got the essential data now, cleaned and in a good format. It’s simply a sequence of a date and a BinType (which can be one of: Blue, Green, Brown and BlueAndBrown).

Working ahead slightly, I looked into how Home Assistant might consume the data I’m finding. Home Assistant can be given a REST API endpoint and is more than happy working with the JSON data returned.

So now, we’ll have to write a simple F# web program to host our data, ready for Home Assistant.

I did look into running an F# script on a timer and possibly publishing data to an MQTT broker. However that got very complicated very quickly.

Our needs for this web server are incredibly simple, a single endpoint which returns the “Next bin”. Home Assistant can be configured to poll this endpoint and then we can show that data on the dashboard. I’m going to keep this API on my private home network, running on a Raspberry Pi, so I don’t need to concern myself with the complexity of building something for the public internet (certificates, vulnerabilities, DDoS etc).

I like to keep things simple, so I’ll be using Giraffe which is a simple Micro-framework which can be plugged into ASP.NET Core, which I’m very familiar with.

Creating a new F# Web Application and pasting the following code into Program.fs, gives us a web server with one endpoint (and come caching - so we’re not hitting the Coventry City Council website on every request) at the application root (http://localhost:5197 - your port number will likely be different).

open System
open Microsoft.AspNetCore.Builder
open Microsoft.AspNetCore.Http
open Microsoft.Extensions.Caching.Memory
open Microsoft.Extensions.DependencyInjection
open Giraffe
open FSharp.Data

type BinType = Blue | Green | Brown | BlueAndBrown

/// <summary>Holds a Date and the type of bin</summary>
type BinCollection = {
    date: DateOnly
    bins: BinType
}

/// <summary>Holds a year and the HTML nodes containing collection dates for it</summary>
type YearAndCollections = { Year: int; Collections: HtmlNode seq }

/// <summary>
/// Determines the "BinType" from the given string of text (directly from the Html) 
/// </summary>
/// <param name="htmlStr">The string of text, directly from the Coventry City Council Website</param>
let private getBinType (htmlStr: string) =
    match htmlStr with
    | htmlStr when htmlStr.Contains("blue") && htmlStr.Contains("brown") -> Some BinType.BlueAndBrown
    | htmlStr when htmlStr.Contains("blue") -> Some BinType.Blue
    | htmlStr when htmlStr.Contains("green") -> Some BinType.Green
    | htmlStr when htmlStr.Contains("brown") -> Some BinType.Brown
    | _ -> None

/// <summary>Returns a list of all Bin Dates (and their Bin Types) from the Coventry City Council
/// website.</summary>
/// <returns>The all bin dates available on the Coventry Council Website.</returns>
let retrieveBinDates =
  let results = HtmlDocument.Load("https://www.coventry.gov.uk/binsthursdayb")
  
  results.Descendants ["div"]
  |> Seq.filter (fun x -> x.HasClass "widget-content")
  |> Seq.map (fun x ->
                      let yearText = (x.Descendants ["h2"]|> Seq.head)
                                      .InnerText()
                                      .Trim()
                      let yearNum = Convert.ToInt32 (yearText.Substring(yearText.IndexOf(" ")))
                      { Year = yearNum; Collections = x.Descendants ["li"] }
  )
  |> Seq.collect (fun x ->
                      x.Collections
                      |> Seq.map (fun y ->
                                    let dateText = $"{y.InnerText().Split(':')[0]} {x.Year}"
                                    let date = DateOnly.Parse(dateText)
                                    let binType = getBinType (y.InnerText())
                                    {| date = date; binsOption = binType |}
                      )
                      |> Seq.filter (fun x -> x.binsOption.IsSome)
                      |> Seq.map (fun x -> { date = x.date; bins = x.binsOption.Value })
  )
  |> Seq.sortBy (fun x -> x.date)
  |> Seq.toList

type ApiResponse = {
  displayStr: string
  dateStr: string
  original: BinCollection
  notificationDate: DateOnly
}

/// <summary>
/// An Http Handler which simply returns the Bin Dates.
/// </summary>
/// <param name="next">The next function in the pipeline</param>
/// <param name="ctx">The HttpContext</param>
let getNextBinCollectionHandler =
  fun (next : HttpFunc) (ctx : HttpContext) ->
        task {
            let now = DateTime.Now
            
            let cache = ctx.GetService<IMemoryCache>()
            let cacheResult = cache.GetOrCreate("AllBinDates", fun x ->
                                              // Cache the result from the HTML extraction for 10 days)
                                              x.AbsoluteExpiration <- now.AddDays(10)
                                              retrieveBinDates)
                              |> List.filter(fun x -> x.date >= DateOnly.FromDateTime(now))
                              
            let nextBin =
              match cacheResult with
              (* The first item in the list is today, however we've passed 12pm (usually when bins are collected
               * So, just take the first item of the rest of the list *)
              | head :: tail when head.date.Day = now.Day && now.Hour >= 12 -> tail.Head
              (* It's either bin day (before 12) or the first item in the list is in the future *)
              | _ -> cacheResult.Head
            
            let response = {
              displayStr = match nextBin.bins with
                           | BlueAndBrown -> "Blue & Brown"
                           | Blue -> "Blue (recycling)"
                           | Green -> "Green (waste)"
                           | Brown -> "Brown (garden)"
              dateStr = nextBin.date.ToString("ddd dd MMMM")
              original = nextBin
              // Notify us to take the bins out, the day before (usually done in the Evening)
              notificationDate = nextBin.date.AddDays(-1)
            }
            
            return! json response next ctx
        }

let webApp =
    choose [ route "/" >=> getNextBinCollectionHandler ]

let builder = WebApplication.CreateBuilder()
builder.Services.AddGiraffe() |> ignore
builder.Services.AddMemoryCache() |> ignore

let app = builder.Build()
app.UseGiraffe webApp
app.Run()

The hard part of extracting and mapping the data into a format we can use has been done.

When run, the above code will output the following JSON data when we hit the root URL.

{
    "displayStr": "Blue & Brown",
    "dateStr": "Thu 15 February",
    "original": {
        "date": "2024-02-15",
        "bins": {
            "case": "BlueAndBrown"
        }
    },
    "notificationDate": "2024-02-14"
}

Home Assistant to consume the data

As briefly mentioned above, Home Assistant has the ability to consume data from a REST endpoint. Home Assistant uses “sensors” to consume data, and there is a REST Sensor which is built-in specifically for this purpose.

REST sensors have a whole host of features, allowing us to even talk with an endpoint which requires Authentication. As my “Bin API” is hosted inside my private network, there’s no Auth.

I’ve dockerised my application (which is outside the scope of this post) and hosted it on my Raspberry Pi with an address/port of http://10.0.0.87:5501 on my local network.

We have to configure Home Assistant so it knows where to look, how often and how to extract data. Opening Home Assistant’s configuration.yml now looks like this (I’ve commented it inline to help explain different pieces).

rest:
  - resource: "http://10.0.0.87:5501"
    # 6 hours
    scan_interval: 21600
    headers:
      Content-Type: application/json
    sensor:
        # A Simple string concatenation to show Bin Type and Date (e.g. "Blue on Wed 10 October) 
      - name: "Bin Collection Display"
        unique_id: "bin_collection_display"
        value_template: "{{ value_json.displayStr }} on {{ value_json.dateStr }}"
        # The Bin collection date (e.g. 2024-01-01)
      - name: "Bin Collection Date"
        unique_id: "bin_collection_date"
        value_template: "{{ value_json.original.date }}"
        # The Bin type string (e.g. "Blue")
      - name: "Bin Collection Type"
        unique_id: "bin_collection_type"
        value_template: "{{ value_json.displayStr }}"
        # What date to send out a notification to Home Assistant Users (essentially is always the day before the bin collection date)
      - name: "Bin Collection Notification Date"
        unique_id: "bin_collection_notification_date"
        value_template: "{{ value_json.notificationDate }}"

Note… I’m using the rest config section above. It is possible to just create a number of sensors, each with their own endpoints/scan intervals - however, for my needs that’s wasteful and might give conflicting values if different sensors are called at different times and displayed on my dashboard.

Saving the configuration file above and restarting Home Assistant, navigating to the Entites page will look something like this (I’ve changed some Icons to help distinguish from my other sensors).

Screen grab showing newly created entities in Home Assistant

Home Assistant to display the data

Now we’ve got the data in Home Assistant we need to display it. I’ve created a simple “Card” on my Dashboard with the following settings

Home Assistant Card configuration to display which bin to put out next

Which looks like this on the dashboard

Home Assistant Dashboard Card showing which bin to put out next

Fantastic! I’ve now got my next bin collection showing up in Home Assistant…..on my phone! (Cool ’eh?)

Running out of bin jokes now, so I might have to start re-cycling old ones

Home Assistant to notify

We’ve got our web server exposing a notification date, now let’s get Home Assistant using it to send notifications to my Android device.

To do this, we’re going to create a Home Assistant automation.

For me, the visual designer is a bit clunky and doesn’t lay out all the information I want to see in one go. I’ll post the YAML below for this automation, commented where needed.

alias: Bin Notification
description: Notifies phones in house to take the bins out
trigger:
    # Run this Automation every day at 5pm
  - platform: time
    at: "17:00:00"
condition:
    # Only perform the action below when today's date matches the "notificationDate" returned from the REST endpoint
  - condition: template
    value_template: >-
            {{ states('sensor.bin_collection_notification_date') == now().strftime('%Y-%m-%d') }}
action:
  - device_id: **the_id_home_assistant_has_given_my_phone**
    domain: mobile_app
    type: notify
    message: "{{ states('sensor.bin_collection_type') }}"
    title: Bin Day Tomorrow
    alias: Send Notification to Ryan
    data:
      car_ui: true
      persistent: true
      tag: bin
mode: single

It’s as simple as that, Home Assistant will run that automation every day at 5pm, and only send me a mobile notification if it’s the day before Bin Day. I have an always-on WireGuard VPN setup for my phone - linked to my home network, so even when I’m out of the house, I’ll get the notification to put the bin out!

Conclusion

Thanks for staying this long and reading about how I’ve optimised by waste collection routine and added in a bit of tech to ensure it gets done correctly.

I’ve had this running for a few weeks now and it’s been faultless. I’ve deleted my weekly alarm and no longer have to rely on my neighbours and follow their lead. I’m now a waste trail-blazer and to my neighbours…..you can rely on me!

I love little learning excercises like this. I’ve learned some more about Home Assistant in this process, and found out how easy it is to configure it to poll REST endpoints! I’m racking my brains to try and think up more data sources I can pull into Home Assistant and further automate chores!

If you’ve got any wacky/interesting automations - I’d love to hear about them and they might give me some more inspiration for my next one.

Thanks for reading!

Apologies for the bin puns, in retrospect I shouldn’t have included them…………..they were rubbish!