Software Engineer with Python, Web Scraping, and RDBMS Expertise Needed

We are looking for a Python software engineer with lots of web scraping experience to help us scrape and create a large-scale database. We will use a variety of APIs, scraping tools, and need complex parsing to crawl the website pages, identify the correct pages, and then scrap the data. We will be collecting both regular numeric, text, and other data types and also want to find and download PDFs, images, and more (unstructured data). We will need to visit and crawl 70,000+ web sites and find pages with data by category. This project will create multiple SQL databases: one database with structured information and another with free-form unstructured information. The database with structured information will get some data through API calls and other data by scraping website pages. Because this will evolve into a complex system we want someone with no less than five years experience programming and ideally 10+ years who understands enterprise systems, servers and architecture, not just scraping. We are building an industrial strength, custom scrapping system that takes data from thousands of source websites and routes it into multiple SQL RDBMS. The ideal candidate will have experience with multiple scrapers like Scrapy, Selenium, Beautiful Soup, and Requests. Since we have a list of thousands of different websites to scrape, and each of these websites have a different structure, we need to build a custom-made way to search, find, and select the correct pages and then scrape the information we need (i.e. resumes, tables, graphics). We want to build a way to scrape pages automatically by looking for keywords on the hundreds of thousands of pages we will scrape. There may be a list of 50 categories of information we will search for in time at these URLs. And it may make sense to use or call different scraping tools and build custom ones. And this will be set up to also compare and detect changes in the data fields (versus the last visit/scrape) and flag these changes. The process scraping unstructured information will find and scrape web pages to extract a wide variety of information like project roadmaps, organization mission statements, investor information, team biographies/resumes, financial data, etc. This could evolve into an ongoing project either part-time or full-time after we build the prototype/MVP product.