Recently I was working on a project that involved "scraping" information from a website. A did a bit of research for available libraries that provide the toolset to do these efficiently.

I found Html Agility Pack, created by Simon Mourier. It's a parser that transforms any real world HTML into valid XML, from which you can then extract information by using XPath.

I also had to use a FormProcessor add-on for Html Agility Pack to simulate login process.

These tools proved to be very useful.