Skip to content

too many request #1

@mauricioPallares

Description

@mauricioPallares

i need scrap 100.000 url with scrapy but it take much time finish it, at least 5 hour by 9000 request, but if i use

process = CrawlerProcess(get_project_settings())
process.crawl(DataamazonSpider, skus=list_of_sku_1)
process.crawl(DataamazonSpider, skus=list_of_sku_2)
process.start()

i have the error 429 too many request

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions