Zyte-funded
90h 20m
Zyte-funded
90h 20m
Personal
1h 20m
Community-funded
0m
Although I don't have any expectations about it nor plan to actively promote it, I'm open to accepting donations and committing a corresponding amount of time per month to the free software projects listed below.
I live in Spain and, for legal reasons, to accept recurring donations I must register as self-employed (autónomo), which entails minimum costs I don't want to take on until I have reasonable assurance that I won't, at minimum, lose money each month by accepting donations.
That's the minimum I consider sufficient to make it unlikely I'll lose money each month. It's calculated by adding the estimated monthly fixed costs of being self-employed (social security contributions, taxes and administrative costs) and dividing that total by the monthly time I would pledge to community work. As living costs (for example housing) rise, minutes per € will fall each year.
I will claim pledges and start dedicating 2.99 minutes / € per month to the projects below.
Once I claim the pledges, I will start accepting requests to create Liberapay teams associated with specific project groups or individual projects listed below. That way, you can donate to a specific group or project, and I will spend the corresponding time working on that specific group or project.
I don't plan to offer that option at first. But if you're interested, contact me and we'll talk.
I'd be very surprised, but the plan is:
51h 15m
41h 55m
In addition to my usual code reviews, and triaging the most recent and 1-month-old pull requests and issues:
I worked on a new throttling
API that deprecates
AutoThrottle in favor of a new throttler component built into the
engine, with configurable backoff and rampup, per-request delays, and
support for honoring throttling feedback from target websites (via
robots.txt and response headers).
I resumed my work on the infamous 2013 CVE. I had analyzed the problem in depth in February 2024, proposed a multi-part solution, and released a draft of the first part. I now have finished the draft and plan to fix the issue once and for all.
I updated the proposal to use xtractmime in Scrapy.
I created a middleware to detect request metadata duplication.
In the documentation:
I increased test coverage.
2h 15m
I extended the URL benchmark tests to cover uncached
performance and prepared
the adoption of can_ada for URL
parsing, which fixes several
issues and in passing improves the performance of some URL APIs by up
to 30%.
I also reviewed pull requests.
2h 5m
I reviewed pull requests.
1h 35m
1h 25m
I fixed a parsing error and released a new version with the change. I also reviewed pull requests.
1h 20m
40m
39h 5m
37h 50m
I created a pull request to make it possible to use proxy mode natively, instead of having to use scrapy-zyte-smartproxy, and so that, when using automatic request parameter mapping, it is used automatically when possible, as it should offer better performance (less latency and traffic). But it is a complex change, so the changes might not be integrated for a couple of years.
I made traces larger than what Zyte API allows be removed, with a warning.
I made sessions managed by the plugin react correctly to the spider's close signal.
I proposed properly handling session limit and session creation errors.
In the documentation:
I proposed a workaround to prevent certain tests from failing randomly.
1h 15m
I started working on a reorganization of the documentation, starting with the index and the contributing page.
1h 20m
1h 20m
I continued working on the tutorial and the related features.