Alternative to parsing

Sometimes the task arises to understand what content is in the list of urls, Screaming Frog is already blocked in many places, importHTML in Google Tables falls off after a few dozen results, and A-Parser not everyone will want to buy and understand.

Example of a task: we have collected a group report on competitors' pages and need to understand what topics are there, or we are looking for link donors and want to know the topics. So a couple of years ago I came up with the idea of just collecting data from the output for quick analysis. Interestingly, until now I have not seen such a tool anywhere, where you can give a list of urls and get the title from the output.

Principle of operation:

• In the file query.txt we put the list of pages
• The script picks up a batch of 50 urls and checks the first result in the output in 10 threads
• Saves as it parses to the result.xlsx file
• When errors occur, it makes 2 more attempts to retrieve data

The price of parsing is equal to the cost of accessing Yandex - 20 rubles/1000 requests, no blocking, proxies and bypasses. Minuses - only part of the text is in the Title of the output.

Example of implementation and full description on Github⬇️, you can do the rest yourself.