{
  "feed_url": "https://gbilder.com/tags/web-scraping/feed.json",
  "home_page_url": "https://gbilder.com/tags/web-scraping/",
  "items": [
    {
      "content_html": "\n  \u003cdiv class=\"e-content\"\u003e\n\n  \u003cdiv class=\"citation\"\u003e\n    \u003cdiv class=\"citation-header\"\u003e\n      \u003cstrong\u003eCite as:\u003c/strong\u003e\n      \u003cbutton class=\"copy-icon\" onclick=\"copyCitation()\" aria-label=\"Copy citation\"\u003e\n        \u003cimg src=\"https://gbilder.com/icons/copy.svg\" alt=\"Copy citation\" width=\"20\" height=\"20\"\u003e\n      \u003c/button\u003e\n    \u003c/div\u003e\n    \u003cp id=\"citation-text\"\u003e\n      \n        \u003ca href=\"https://orcid.org/0000-0003-1315-5960\" target=\"_blank\" rel=\"noopener noreferrer\"\u003eGeoffrey Bilder\u003cimg src=\"https://gbilder.com/ORCID-iD_icon_vector.svg\" alt=\"ORCID iD icon\" width=\"16\" height=\"16\" style=\"margin-left:4px;vertical-align:text-bottom;\"\u003e\n\n          \u003c/a\u003e\n\n        \n      \n      \n        (2025, April 2).\n      \n      \u003ci\u003eBots Behaving Badly\u003c/i\u003e.\n      Retrieved from \u003ca href=\"https://doi.org/10.59347/w366z-egx85\"\u003e\n      https://doi.org/10.59347/w366z-egx85\n      \u003c/a\u003e\n    \u003c/p\u003e\n  \u003c/div\u003e\n\n\n\u003cp\u003eI\u0026rsquo;ve seen an uptick in people I respect concluding that the massive increase in badly-behaved web-scraping \u003ca href=\"https://en.wikipedia.org/wiki/Internet_bot\"\u003ebots\u003c/a\u003e is intentional and just another example of sociopathic AI companies ripping-up long-respected web conventions for the sake of profit. Here are two recent examples:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"https://go-to-hellman.blogspot.com/2025/03/ai-bots-are-destroying-open-access.html\"\u003eAI bots are destroying Open Access\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://www.marginalia.nu/log/a_115_rude_crawler/\"\u003eImproved ways to operate a rude crawler\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eI am \u003ca href=\"https://gbilder.com/blog/2024/07/ai-update/\"\u003eno fan of AI companies\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003eBut I don’t think this explains what\u0026rsquo;s really happening.\u003c/p\u003e\n\u003cp\u003eThe problem still has its roots in AI, but for different reasons—  reasons that are possibly more concerning than the idea that IA companies are doing this stuff on purpose.\u003c/p\u003e\n\u003cp\u003eInstead, I suspect the \u003cstrong\u003ereal\u003c/strong\u003e problem is people \u003cem\u003eusing AI coding tools\u003c/em\u003e that generate, buggy, and hyper-aggressive bots.\u003c/p\u003e\n\u003cp\u003eAnd I bet that the vast majority of the developers behind these bots have no idea that they are causing problems.\u003c/p\u003e\n\u003cp\u003eThis would consistent with what I used to see when trying to manage \u003ca href=\"https://api.crossref.org\"\u003ea popular API\u003c/a\u003e even \u003cem\u003ebefore\u003c/em\u003e AI tools became ubiquitous.\u003c/p\u003e\n\u003cp\u003eWe \u003cem\u003econstantly\u003c/em\u003e had problems with bots— but only twice could the problems be attributed to a bot that was deliberately designed to be aggressive and evade detection.\u003c/p\u003e\n\u003cp\u003eAll the other times it was some naive programmer (often a researcher)  who had built a buggy bot or who simply didn’t know about bot conventions like \u003ca href=\"https://en.wikipedia.org/wiki/Robots.txt\"\u003e\u003ccode\u003erobots.txt\u003c/code\u003e\u003c/a\u003e, error/retry backoffs, \u003ca href=\"https://en.wikipedia.org/wiki/Rate_limiting\"\u003erate-limiting\u003c/a\u003e, \u003ca href=\"https://en.wikipedia.org/wiki/Connection_pool\"\u003econnection pools\u003c/a\u003e, \u003ca href=\"https://en.wikipedia.org/wiki/User-Agent_header\"\u003euser-agent\u003c/a\u003e ettiquette, etc..\u003c/p\u003e\n\u003cp\u003eWe also had a constant problem with developers who didn’t know how \u003ca href=\"https://www.merge.dev/blog/cursor-pagination\"\u003ecursors\u003c/a\u003e worked in our API because they simply hadn’t read the docs. This meant that on any given day we would see dozens of bots just endlessly requesting the next page of empty results from a cursor that had long ago stopped delivering them data.\u003csup id=\"fnref:1\"\u003e\u003ca href=\"#fn:1\" class=\"footnote-ref\" role=\"doc-noteref\"\u003e1\u003c/a\u003e\u003c/sup\u003e\u003c/p\u003e\n\u003cp\u003eWhenever we managed to get in touch with the developers, they would be genuinely surprised, profusely apologetic, and they’d immediately shut off or fix their code.\u003c/p\u003e\n\u003cp\u003eSometimes we’d find that said developers had \u003cem\u003eentirely forgotten they’d left the bot running\u003c/em\u003e and they’d sheepishly admit that the project it was built for had ended months or years ago!\u003c/p\u003e\n\u003cp\u003eAnd a lot of this code was public.\u003c/p\u003e\n\u003cp\u003eAnd AIs were trained on it.\u003c/p\u003e\n\u003cp\u003eAnd so AI-generated code is replicating all of these problems.\u003c/p\u003e\n\u003cp\u003eI routinely test new AI coding tools with a set of problems that I know are easy to get wrong. Web scraping is one of these things.\u003c/p\u003e\n\u003cp\u003eAnd all the tools I’ve tried make at least a few of the  same mistakes humans make— they ignore \u003ca href=\"https://en.wikipedia.org/wiki/Robots.txt\"\u003e\u003ccode\u003erobots.txt\u003c/code\u003e\u003c/a\u003e, they don’t use \u003ca href=\"https://en.wikipedia.org/wiki/Connection_pool\"\u003econnection pooling\u003c/a\u003e, they don’t honor \u003ca href=\"https://en.wikipedia.org/wiki/Rate_limiting\"\u003erate limits\u003c/a\u003e, they don’t set correct \u003ca href=\"https://en.wikipedia.org/wiki/User-Agent_header\"\u003euser-agent\u003c/a\u003e headers.\u003c/p\u003e\n\u003cp\u003eI also test to see if the AI tools will generate code that properly handles cursors with the Crossref API.\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eOnly once\u003c/em\u003e have I seen an LLM produce the correct code the first time.  And even then, it was a fluke and I haven’t been able to get that particular LLM to do it again since.\u003c/p\u003e\n\u003cp\u003eAnd, of course, increasingly AI\u0026rsquo;s are also trained on code that includes mechanisms to evade being blocked— and so, in addition to being buggy, they are also effectively trained to be more aggressive and stealthy.\u003c/p\u003e\n\u003cp\u003eAnd whereas most new programmers are justifiably reluctant to delve into the intricacies of creating concurrent apps, AI tools seem to have no such inhibitions and generate code that spawns dozens of requests at that same time.\u003c/p\u003e\n\u003cp\u003eBut most of the the people using these AI tools don’t know this. All they know is that the AI tools allow them to easily build a bot to scrape data from the web- something that seemed too daunting for them to do before.\u003c/p\u003e\n\u003cp\u003eThey are \u0026ldquo;\u003ca href=\"https://en.wikipedia.org/wiki/Vibe_coding\"\u003evibe coding\u003c/a\u003e.\u0026rdquo;\u003c/p\u003e\n\u003cp\u003eAnd so now we have a gigantic influx of new \u0026ldquo;developers\u0026rdquo; building new tools to make use of data on the internet. And these tools are based on highly concurrent, hyper-aggressive, and extremely buggy AI-generated code.\u003c/p\u003e\n\u003cp\u003eThis may sound like I’m splitting hairs— I mean, the problem is still related to AI, right? And this behavior, — whether deliberate or not — is causing a lot of collateral damage.\u003c/p\u003e\n\u003cp\u003eYes, but I think this underscores what I suspect is the most realistic threat from AI — not that we will all soon be replaced by AGI, but that more and more of our systems are going to become infested with subtly buggy data and code that has unintentional, unpredictable, and damaging downstream effects.\u003c/p\u003e\n\u003cp\u003eOnce we all realize what is happening, we\u0026rsquo;re going to spend the next decade trying to excise this stuff. And we\u0026rsquo;re not going to be able to use AI tools to help us do it.\u003c/p\u003e\n\u003ch2 id=\"example-code-generated-by-claude-37-on-2025-03-24\"\u003eExample code generated by Claude 3.7 on 2025-03-24\u003c/h2\u003e\n\u003cp\u003ePrompt: \u003ccode\u003eGenerate a Python function that uses the Crossref API to get all the records that were indexed on 2024-01-01.\u003c/code\u003e\u003c/p\u003e\n\n    \u003cdiv class=\"admonition danger\"\u003e\n      \u003cdiv class=\"admonition-header\"\u003e\u003csvg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 512 512\"\u003e\u003cpath d=\"M256 32c14.2 0 27.3 7.5 34.5 19.8l216 368c7.3 12.4 7.3 27.7 .2 40.1S486.3 480 472 480L40 480c-14.3 0-27.6-7.7-34.7-20.1s-7-27.8 .2-40.1l216-368C228.7 39.5 241.8 32 256 32zm0 128c-13.3 0-24 10.7-24 24l0 112c0 13.3 10.7 24 24 24s24-10.7 24-24l0-112c0-13.3-10.7-24-24-24zm32 224a32 32 0 1 0 -64 0 32 32 0 1 0 64 0z\"/\u003e\u003c/svg\u003e\n        \u003cspan\u003eBroken code below\u003c/span\u003e\n      \u003c/div\u003e\n      \u003cdiv class=\"admonition-content\"\u003e\n        \u003cp\u003eThankfully the code below will only generate one request and fail emediately with a \u003ccode\u003e400\u003c/code\u003e error because it has used the wrong \u003ccode\u003efilter\u003c/code\u003e parameters. But \u003cem\u003ehad\u003c/em\u003e it created a correct request, the script would have never exited and would have kept requesting the same URL from the Crossref API forever because it has made an erroneous assumption about how \u003ccode\u003ecursors\u003c/code\u003e work in the API. Also note that, though it at least includes user-agent information, it has simplistic rate-limiting and no mechanisms for backing-off when it encounters errors.\u003c/p\u003e\n      \u003c/div\u003e\n    \u003c/div\u003e\u003cp\u003eResults:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kn\"\u003eimport\u003c/span\u003e \u003cspan class=\"nn\"\u003erequests\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kn\"\u003eimport\u003c/span\u003e \u003cspan class=\"nn\"\u003etime\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kn\"\u003efrom\u003c/span\u003e \u003cspan class=\"nn\"\u003edatetime\u003c/span\u003e \u003cspan class=\"kn\"\u003eimport\u003c/span\u003e \u003cspan class=\"n\"\u003edatetime\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003edef\u003c/span\u003e \u003cspan class=\"nf\"\u003eget_crossref_records_by_indexed_date\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003edate_str\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003eemail\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"kc\"\u003eNone\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003erows_per_page\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"mi\"\u003e1000\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003emax_records\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"kc\"\u003eNone\u003c/span\u003e\u003cspan class=\"p\"\u003e):\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"s2\"\u003e\u0026#34;\u0026#34;\u0026#34;\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s2\"\u003e    Retrieve records indexed by Crossref on a specific date using the Crossref API.\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s2\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s2\"\u003e    Args:\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s2\"\u003e        date_str (str): Date in YYYY-MM-DD format (e.g., \u0026#39;2024-01-01\u0026#39;)\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s2\"\u003e        email (str, optional): Email address for polite API use\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s2\"\u003e        rows_per_page (int, optional): Number of records per page (default 1000, max 1000)\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s2\"\u003e        max_records (int, optional): Maximum number of records to retrieve (None for all)\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s2\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s2\"\u003e    Returns:\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s2\"\u003e        list: List of Crossref record dictionaries\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s2\"\u003e    \u0026#34;\u0026#34;\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"c1\"\u003e# Base URL for Crossref API\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"n\"\u003ebase_url\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;https://api.crossref.org/works\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"c1\"\u003e# Validate date format\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"k\"\u003etry\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"n\"\u003edatetime\u003c/span\u003e\u003cspan class=\"o\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003estrptime\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003edate_str\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"s1\"\u003e\u0026#39;%Y-%m-\u003c/span\u003e\u003cspan class=\"si\"\u003e%d\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"k\"\u003eexcept\u003c/span\u003e \u003cspan class=\"ne\"\u003eValueError\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"k\"\u003eraise\u003c/span\u003e \u003cspan class=\"ne\"\u003eValueError\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;Incorrect date format, should be YYYY-MM-DD\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"c1\"\u003e# Construct filter for indexed date\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"n\"\u003efilter_param\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"sa\"\u003ef\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;indexed:\u003c/span\u003e\u003cspan class=\"si\"\u003e{\u003c/span\u003e\u003cspan class=\"n\"\u003edate_str\u003c/span\u003e\u003cspan class=\"si\"\u003e}\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"c1\"\u003e# Set up headers with user agent (best practice for Crossref API)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"n\"\u003eheaders\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"s1\"\u003e\u0026#39;User-Agent\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e \u003cspan class=\"sa\"\u003ef\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;PythonCrossrefClient/1.0 (Python Script; \u003c/span\u003e\u003cspan class=\"si\"\u003e{\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u0026#34;\u003c/span\u003e \u003cspan class=\"k\"\u003eif\u003c/span\u003e \u003cspan class=\"n\"\u003eemail\u003c/span\u003e \u003cspan class=\"ow\"\u003eis\u003c/span\u003e \u003cspan class=\"kc\"\u003eNone\u003c/span\u003e \u003cspan class=\"k\"\u003eelse\u003c/span\u003e \u003cspan class=\"n\"\u003eemail\u003c/span\u003e\u003cspan class=\"si\"\u003e}\u003c/span\u003e\u003cspan class=\"s1\"\u003e)\u0026#39;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"c1\"\u003e# Parameters for the API request\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"n\"\u003eparams\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"s1\"\u003e\u0026#39;filter\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e \u003cspan class=\"n\"\u003efilter_param\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"s1\"\u003e\u0026#39;rows\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e \u003cspan class=\"n\"\u003erows_per_page\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"s1\"\u003e\u0026#39;cursor\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e \u003cspan class=\"s1\"\u003e\u0026#39;*\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e  \u003cspan class=\"c1\"\u003e# Start cursor\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"s1\"\u003e\u0026#39;sort\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e \u003cspan class=\"s1\"\u003e\u0026#39;indexed\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e  \u003cspan class=\"c1\"\u003e# Sort by indexed date\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"s1\"\u003e\u0026#39;order\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e \u003cspan class=\"s1\"\u003e\u0026#39;asc\u0026#39;\u003c/span\u003e  \u003cspan class=\"c1\"\u003e# Ascending order\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"n\"\u003eall_records\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"p\"\u003e[]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"n\"\u003epage_count\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"mi\"\u003e0\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"k\"\u003ewhile\u003c/span\u003e \u003cspan class=\"kc\"\u003eTrue\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"c1\"\u003e# Make the API request\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"n\"\u003eresponse\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003erequests\u003c/span\u003e\u003cspan class=\"o\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eget\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003ebase_url\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003eparams\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"n\"\u003eparams\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003eheaders\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"n\"\u003eheaders\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"c1\"\u003e# Check for successful response\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"k\"\u003eif\u003c/span\u003e \u003cspan class=\"n\"\u003eresponse\u003c/span\u003e\u003cspan class=\"o\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003estatus_code\u003c/span\u003e \u003cspan class=\"o\"\u003e!=\u003c/span\u003e \u003cspan class=\"mi\"\u003e200\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"nb\"\u003eprint\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"sa\"\u003ef\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;Error: API request failed with status code \u003c/span\u003e\u003cspan class=\"si\"\u003e{\u003c/span\u003e\u003cspan class=\"n\"\u003eresponse\u003c/span\u003e\u003cspan class=\"o\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003estatus_code\u003c/span\u003e\u003cspan class=\"si\"\u003e}\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"nb\"\u003eprint\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"sa\"\u003ef\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;Response text: \u003c/span\u003e\u003cspan class=\"si\"\u003e{\u003c/span\u003e\u003cspan class=\"n\"\u003eresponse\u003c/span\u003e\u003cspan class=\"o\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003etext\u003c/span\u003e\u003cspan class=\"si\"\u003e}\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"k\"\u003ebreak\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"c1\"\u003e# Parse the JSON response\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"n\"\u003edata\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003eresponse\u003c/span\u003e\u003cspan class=\"o\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003ejson\u003c/span\u003e\u003cspan class=\"p\"\u003e()\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"c1\"\u003e# Get the records from the current page\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"n\"\u003eitems\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003edata\u003c/span\u003e\u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;message\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e][\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;items\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"c1\"\u003e# Add items to our collection\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"n\"\u003eall_records\u003c/span\u003e\u003cspan class=\"o\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eextend\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eitems\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"n\"\u003epage_count\u003c/span\u003e \u003cspan class=\"o\"\u003e+=\u003c/span\u003e \u003cspan class=\"mi\"\u003e1\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"nb\"\u003eprint\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"sa\"\u003ef\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;Retrieved page \u003c/span\u003e\u003cspan class=\"si\"\u003e{\u003c/span\u003e\u003cspan class=\"n\"\u003epage_count\u003c/span\u003e\u003cspan class=\"si\"\u003e}\u003c/span\u003e\u003cspan class=\"s2\"\u003e, \u003c/span\u003e\u003cspan class=\"si\"\u003e{\u003c/span\u003e\u003cspan class=\"nb\"\u003elen\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eitems\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"si\"\u003e}\u003c/span\u003e\u003cspan class=\"s2\"\u003e records. Total: \u003c/span\u003e\u003cspan class=\"si\"\u003e{\u003c/span\u003e\u003cspan class=\"nb\"\u003elen\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eall_records\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"si\"\u003e}\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"c1\"\u003e# Check if we\u0026#39;ve reached the maximum records limit\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"k\"\u003eif\u003c/span\u003e \u003cspan class=\"n\"\u003emax_records\u003c/span\u003e \u003cspan class=\"ow\"\u003eis\u003c/span\u003e \u003cspan class=\"ow\"\u003enot\u003c/span\u003e \u003cspan class=\"kc\"\u003eNone\u003c/span\u003e \u003cspan class=\"ow\"\u003eand\u003c/span\u003e \u003cspan class=\"nb\"\u003elen\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eall_records\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e \u003cspan class=\"o\"\u003e\u0026gt;=\u003c/span\u003e \u003cspan class=\"n\"\u003emax_records\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"n\"\u003eall_records\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003eall_records\u003c/span\u003e\u003cspan class=\"p\"\u003e[:\u003c/span\u003e\u003cspan class=\"n\"\u003emax_records\u003c/span\u003e\u003cspan class=\"p\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"k\"\u003ebreak\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"c1\"\u003e# Get the next cursor from the response\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"n\"\u003enext_cursor\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003edata\u003c/span\u003e\u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;message\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e]\u003c/span\u003e\u003cspan class=\"o\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eget\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;next-cursor\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"c1\"\u003e# If there\u0026#39;s no next cursor, we\u0026#39;ve reached the end\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"k\"\u003eif\u003c/span\u003e \u003cspan class=\"ow\"\u003enot\u003c/span\u003e \u003cspan class=\"n\"\u003enext_cursor\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e  \u003cspan class=\"c1\"\u003e# \u0026lt;---- Argh- NO! This is wrong!\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"k\"\u003ebreak\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"c1\"\u003e# Update the cursor parameter for the next request\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"n\"\u003eparams\u003c/span\u003e\u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;cursor\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e]\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003enext_cursor\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"c1\"\u003e# Be nice to the API with a small delay between requests\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"n\"\u003etime\u003c/span\u003e\u003cspan class=\"o\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003esleep\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"mi\"\u003e1\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"nb\"\u003eprint\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"sa\"\u003ef\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;Total records retrieved: \u003c/span\u003e\u003cspan class=\"si\"\u003e{\u003c/span\u003e\u003cspan class=\"nb\"\u003elen\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eall_records\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"si\"\u003e}\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"k\"\u003ereturn\u003c/span\u003e \u003cspan class=\"n\"\u003eall_records\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Example usage\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eif\u003c/span\u003e \u003cspan class=\"vm\"\u003e__name__\u003c/span\u003e \u003cspan class=\"o\"\u003e==\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;__main__\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"c1\"\u003e# Get records indexed on January 1, 2024\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"c1\"\u003e# Add your email for polite API use\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"n\"\u003erecords\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003eget_crossref_records_by_indexed_date\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;2024-01-01\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003eemail\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;your.email@example.com\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"c1\"\u003e# Print some information about the first few records\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"k\"\u003efor\u003c/span\u003e \u003cspan class=\"n\"\u003ei\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003erecord\u003c/span\u003e \u003cspan class=\"ow\"\u003ein\u003c/span\u003e \u003cspan class=\"nb\"\u003eenumerate\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003erecords\u003c/span\u003e\u003cspan class=\"p\"\u003e[:\u003c/span\u003e\u003cspan class=\"mi\"\u003e5\u003c/span\u003e\u003cspan class=\"p\"\u003e]):\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"nb\"\u003eprint\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"sa\"\u003ef\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"se\"\u003e\\n\u003c/span\u003e\u003cspan class=\"s2\"\u003eRecord \u003c/span\u003e\u003cspan class=\"si\"\u003e{\u003c/span\u003e\u003cspan class=\"n\"\u003ei\u003c/span\u003e\u003cspan class=\"o\"\u003e+\u003c/span\u003e\u003cspan class=\"mi\"\u003e1\u003c/span\u003e\u003cspan class=\"si\"\u003e}\u003c/span\u003e\u003cspan class=\"s2\"\u003e:\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"nb\"\u003eprint\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"sa\"\u003ef\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;DOI: \u003c/span\u003e\u003cspan class=\"si\"\u003e{\u003c/span\u003e\u003cspan class=\"n\"\u003erecord\u003c/span\u003e\u003cspan class=\"o\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eget\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;DOI\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"si\"\u003e}\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"nb\"\u003eprint\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"sa\"\u003ef\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;Title: \u003c/span\u003e\u003cspan class=\"si\"\u003e{\u003c/span\u003e\u003cspan class=\"n\"\u003erecord\u003c/span\u003e\u003cspan class=\"o\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eget\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;title\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;No title\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e])[\u003c/span\u003e\u003cspan class=\"mi\"\u003e0\u003c/span\u003e\u003cspan class=\"p\"\u003e]\u003c/span\u003e\u003cspan class=\"si\"\u003e}\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"nb\"\u003eprint\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"sa\"\u003ef\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;Type: \u003c/span\u003e\u003cspan class=\"si\"\u003e{\u003c/span\u003e\u003cspan class=\"n\"\u003erecord\u003c/span\u003e\u003cspan class=\"o\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eget\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;type\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"si\"\u003e}\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"nb\"\u003eprint\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"sa\"\u003ef\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;Indexed: \u003c/span\u003e\u003cspan class=\"si\"\u003e{\u003c/span\u003e\u003cspan class=\"n\"\u003erecord\u003c/span\u003e\u003cspan class=\"o\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eget\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;indexed\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"p\"\u003e{})\u003c/span\u003e\u003cspan class=\"o\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eget\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;date-time\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"si\"\u003e}\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cdiv class=\"footnotes\" role=\"doc-endnotes\"\u003e\n\u003chr\u003e\n\u003col\u003e\n\u003cli id=\"fn:1\"\u003e\n\u003cp\u003eYes, we could could have fixed this in the API. Don\u0026rsquo;t get me started.\u0026#160;\u003ca href=\"#fnref:1\" class=\"footnote-backref\" role=\"doc-backlink\"\u003e\u0026#x21a9;\u0026#xfe0e;\u003c/a\u003e\u003c/p\u003e\n\u003c/li\u003e\n\u003c/ol\u003e\n\u003c/div\u003e\n\u003c/div\u003e\n",
      "date_published": "2025-04-02T00:00:00Z",
      "id": "https://gbilder.com/posts/2025-04-02-bots-behaving-badly/",
      "tags": [
        "web-scraping",
        "bots",
        "AI"
      ],
      "url": "https://gbilder.com/posts/2025-04-02-bots-behaving-badly/"
    }
  ],
  "language": "en",
  "title": "#web-scraping · Louche Cannon",
  "version": "https://jsonfeed.org/version/1.1"
}