Using a Custom Scrapy Extension to Log to a DB

The Scrapy crawl stat logs are useful for tracking and monitoring the performance of a spider. If you want to keep them longer rather than just see the console printout, you can have them written to a database.