2mo ago

GitHub is introducing rate limits for unauthenticated pulls, API calls, and web access

Updated rate limits for unauthenticated requests - GitHub Changelog

An update from GitHub: https://github.com/orgs/community/discussions/159123#discussioncomment-13148279

The rates are here: https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api?apiVersion=2022-11-28

60 req/hour for unauthenticated users
5000 req/hour for authenticated - personal
15000 req/hour for authenticated - enterprise org

137 comments

Probably getting hammered by ai scrapers
- you mean, doin' what microsoft and their ai 'partners' do to others?
  Also, as Microsoft appears to have recognized scraping for AI training as a problem, are you seizing your own scraping activities on public code and the larger web or is this a case of double standards?
  
  Yeah but they're allowed to do it because they have brazillions of dollars.
- The funny thing is that rate limits won't help them with genai scrapers
- Everything seems to be. There was a period where you could kinda have a sane experience browsing over a VPN or otherwise using a cloud service IP range endpoint but especially the past 6 months or so things have gotten worse exponentially by the week. Everything is moving behind cloudflare or other systems
If Microsoft knows how to do one thing well, it’s killing a successful product.
- I came here looking for this comment. They bought the service to destroy it. It's kind of their thing.
  
  Github has literally never been doing better. What are you talking about??
- RIP Skype
  
  we could have had bob or clippy instead of 'cortana' or 'copilot'
Github is owned by Microsoft, so don't worry, it's going to get worse
60 req/hour for unauthenticated users
That's low enough that it may cause problems for a lot of infrastructure. Like, I'm pretty sure that the MELPA emacs package repository builds out of git, and a lot of that is on github.
- That’s low enough that it may cause problems for a lot of infrastructure.
  Likely the point. If you need more, get an API key.
  
  Or just make authenticated requests. I'd expect that to be well within with capabilities of anyone using MELPA, and 5000 requests per hour shouldn't pose any difficulty considering MELPA only has about 6000 total packages.
- Do you think any infrastructure is pulling that often while unauthenticated? It seems like an easy fix either way (in my admittedly non devops opinion)
  
  It's gonna be problematic in particular for organisations with larger offices. If you've got hundreds of devs/sysadmins under the same public IP address, those 60 requests/hour are shared between them.
  Basically, I expect unauthenticated pulls to not anymore be possible at my day job, which means repos hosted on GitHub become a pain.
  
  If I’m using Ansible or something to pull images it might get that high.
  Of course the fix is to pull it once and copy the files over, but I could see this breaking prod for folks who didn’t write it that way in the first place
- I didn't think of that - also for nvim you typically pull plugins from git repositories
I see the "just create an account" and "just login" crowd have joined the discussion. Some people will defend a monopolist no matter what. If github introduced ID checks à la Google or required a Microsoft account to login, they'd just shrug and go "create a Microsoft account then, stop bitching". They don't realise they are being boiled and don't care. Consoomer behaviour.
Anti Commercial-AI license
- Or we just realize that GitHub without logging in is a service we are getting for free. And when there's something free, there's someone trying to exploit it. Using GitHub while logged in is also free and has none of these limits, while allowing them to much easier block exploiters.
  
  I would like to remind you that you are arguing for a monopolist. I'd agree with you if it were for a startup or mid-sized company that had lots of competition and was providing a good product being abused by competitors or users. But Github has a quasi-monopoly, is owned by a monopolist that is part of the reason other websites are being bombarded by requests (aka, they are part of the problem), and you are sitting here arguing that more people should join the monopoly because of an issue they created.
  Can you see the flaws in reasoning in your statements?
  Anti Commercial-AI license
I honestly don't really see the problem here. This seems to mostly be targeting scrapers.
For unauthenticated users you are limited to public data only and 60 requests per hour, or 30k if you're using Git LFS. And for authenticated users it's 60k/hr.
What could you possibly be doing besides scraping that would hit those limits?
- You might behind a shared IP with NAT or CG-NAT that shares that limit with others, or might be fetching files from raw.githubusercontent.com as part of an update system that doesn't have access to browser credentials, or Git cloning over https:// to avoid having to unlock your SSH key every time, or cloning a Git repo with submodules that separately issue requests. An hour is a long time. Imagine if you let uBlock Origin update filter lists, then you git clone something with a few modules, and so does your coworker and now you're blocked for an entire hour.
- 60 requests per hour per IP could easily be hit from say, uBlock origin updating filter lists in a household with 5-10 devices.
- I hit those many times when signed out just scrolling through the code. The front end must be sending off tonnes of background requests
  
  This doesn't include any requests from the website itself
No no, no no no no, no no no no, no no there's no limit
https://forgejo.org/
- Until there will be.
  I think people are grossly underestimating the sheer size and significance of the issue at hand. Forgejo will very likely eventually get to the same point Github is at right now, and will have to employ some of the same safeguards.
  
  Except Forgejo is open source and you can run your own instance of it. I do, and it's great.
- Dude, this is cool!
  
  It works really well too. I have an instance.
- No, no limits, we'll reach for the skyyyy
- Amazon's AI crawler is making my git server unstable
  End of the day someone still has to pay for those requests
LOL!!!! RIP GitHub
EDIT: trying to compile any projects from source that use git submodules will be interesting. eg ROCm has more than 60 submodules to pull in 💀
- The Go module system pulls dependencies from their sources. This should be interesting.
  Even if you host your project on a different provider, many libraries are on github. All those unauthenticated Arch users trying to install Go-based software that pulls dependencies from github.
  How does the Rust module system work? How does pip?
  
  already not looking forward to the next updates on a few systems.
  
  For Rust, as I understand, crates.io hosts a copy of the source code. It is possible to specify a Git repository directly as a dependency, but apparently, you cannot do that if you publish to crates.io.
  So, it will cause pain for some devs, but the ecosystem at large shouldn't implode.
  
  Compiling any larger go application would hit this limit almost immediately. For example, podman is written in go and has around 70 dependencies, or about 200 when including transitive dependencies. Not all the depends are hosted on GitHub, but the vast majority are. That means that with a limit of 60 request per hour it would take you 3 hours to build podman on a new machine.
Crazy how many people think this is okay, yet left Reddit cause of their API shenanigans. GitHub is already halfway to requiring signing in to view anything like Twitter (X).
- They make you sign in to use search, on code anyways.
  
  Which i hate so much anytime i want to quickly look for something
- It's not the same making API costs unbearable for a social media user and limiting the rate non-logged-in users.
  You can still use GitHub without being logged in. You can still use GitHub without almost any limit on a free account.
  You cannot even use reddit on a third party app with an account with reddit gold.
!codeberg@programming.dev
- Codeberg has used way stricter rate limiting since pretty much forever. Nice thought, but Codeberg will not solve this problem, like at all.
  
  What? I have never seen a rate limiting screen on codeberg. Ever. If I click too much on github I get rate limited. It happens so frequently, I use https://sourcegraph.com/search when I have to navigate a repository's code.
  Anti Commercial-AI license
Just browsing GitHub I've got this limit
- i've hit it many times so far.. even as quick as the second page view (first internal link clicked) after more than a day or two since the last visit (yes, even with cleaned browser data or private window).
  it's fucking stupid how quick they are to throw up a roadblock.
- Then login.
- Just browse authenticated, you won't have that issue.
  
  that is not an acceptable 'solution' and opens up an entirely different and more significant can o' worms instead.
Wow so surprising, never saw this coming, this is my surprised face. :-l
The enshittification begins (continues?)...
- just now? :)
Open source repositories should rely on p2p. Torrenting repos is the way I think.
Not only for this. At any point m$ could take down your repo if they or their investors don't like it.
I wonder if it would already exist and if it could work with git?
- Git is p2p and distributed from day 1. Github is just a convenient website. If Microsoft takes down your repo, just upload to another system. Nothing but convenience will be lost.
  
  Not entirely true. You lose tickets and PRs in that scenario.
  
  Look into https://radicle.xyz/
- The project's official repo should probably exist in a single location so that there is an authoritative version. At that point p2p is only necessary if traffic for the source code is getting too expensive for the project.
  Personally I think the source hut model is closest to the ideal set up for OSS projects. Though I use Codeberg for my personal stuff because I'm cheap and lazy
  
  I'm wary of external dependencies. They are cool now, but will they be cool in the future? Will they even exist?
  One thing I think p2p excels is resiliance. People be still using eDonkey even if it's abandoned.
  A repo signature should deal with "fake copies". It's true we have the problem that BitTorrent protocol is not though for updating files, so a different protocol would be needed. I don't even know how possible/practical it is. It's true that any big project should probably host their own remote repo, and copy it on other platforms as needed. Github only repos was always a dangerous practice.
- https://blog.tangled.sh/intro
  See also: https://fossil-scm.org/
- Torrenting doesn't deal well with updating files.
  And you have another problem: how do you handle bad actors spamming the download ?
  That's probably why github does that.
  
  That's true. I didn't think of that.
  IPFS supposedly works fine with updating shares. But I don't want to get closer to that project as they had fallen into cryptoscam territory.
  I'm currently reading about "radicle" let's see what the propose.
  I don't get the bad actors spamming the download. Like downloading too much? Torrent leechers?
  EDIT: Just finished by search sbout radicle. They of course have relations with a cryptomscam. Obviously..... ;_; why this keep happening?
- https://radicle.xyz/
  
  I've been reading about it. But at some point I found that the parent organization run a crypto scam. Supposedly is not embedded into the protocol but they also said that the token is used to give rewards withing the protocol. That just made me wary of them.
  Though the protocol did seen interesting. It's MIT licensed I think so I suppose it could just be forked into something crypto free.
This going to fuck over obtanium?
Its always blocked me from searching in firefox when I'm logged out for some reason.
THIS is why I clone all my commonly used Repos to my personal gitea instance.
- I recently switched my instance from gitea to forgejo because everyone said to do it and it was easy to do.
  
  What were the benefits
- That's actually kind of an interesting idea.
  Is there a reasonable way that I could host my own ui that will keep various repos. I care about cloned and always up to date automatically?
  
  Afict, you should be able to follow the instructions for migrating the repo and it will clone it to your instance and track for updates. It's been a minute since I've read up on it though
Good thing I moved all my repos from git[lab|hub] to Codeberg recently.
- did your project manager or client ask for you to move there?
Good thing git is “federated” by default.
- & then you have fossil which is github in a box
Maybe charge OpenAI for scrapes instead of screwing over your actual customers.
I have a question: why do lemmy dev keep using microsoft github?
- Yeah, shoulda use https://gitflic.ru/
Probably because of AI agents. This is why we can’t have nice things.
That' just how the cookie crumbles.
is authenticated like when you use a private key with git clone? stupid question i know
also this might be terrible if you subscribe to filter lists on raw github in ublock or adguard
- is authenticated like when you use a private key with git clone
  Yes
  also this might be terrible if you subscribe to filter lists on raw github in ublock or adguard
  Yes exactly why this is actually quite problematic. There's a lot of HTTPS Git pull remotes around and random software that uses raw.githubusercontent.com to fetch data. All of that is now subject to the 60 req/hr limit and not all of it will be easy to fix.
RIP yocto builds
This is specific to the GH REST API I think, not operations like doing a git clone to copy a repo to local machine, etc.
- These changes will apply to operations like cloning repositories over HTTPS, anonymously interacting with our REST APIs, and downloading files from raw.githubusercontent.com.
  
  downloading files from raw.githubusercontent.com
  oh fuck, this is going to break stuff.
- These changes will apply to operations like cloning repositories over HTTPS, anonymously interacting with our REST APIs, and downloading files from raw.githubusercontent.com.
The numbers actually seem reasonable...
- Not at all if you're a software developer, which is the whole point of the service. Automated requests from their own tools can easily punch through this building a large project even one time.
- ....
  60 requests
  Per hour
  How is that reasonable??
  You can hit the limits by just browsing GitHub for 15 minutes.
  
  Without login

137 comments