Rate limiting has a reputation as an infrastructure chore. Someone notices the API is getting hammered, someone adds a limit at the gateway, a number gets picked because it sounds safe, and everyone moves on. The number is almost always round, almost always arbitrary, and almost always chosen to protect the servers.
That framing is where it goes wrong. Protecting the servers is the easy part. The hard part is deciding who gets slowed down, when, and what happens to them when they do. Those are not infrastructure questions. They are product questions wearing an infrastructure hat.
A limit is a sentence you pass on a user
Every rate limit is a rule about how much a person is allowed to do before you tell them no. Written that way, it is obviously a product decision. You are deciding what normal use looks like, and by extension you are deciding who counts as abnormal.
The trouble is that your most engaged users and your abusers can look identical from the server's point of view. Both make a lot of requests. If you set the limit purely to stop the abuser, you will catch the power user too, and the power user is the one you least want to annoy. They are the person who loves the product enough to push on it. A limit tuned only for defense treats enthusiasm as an attack.
The numbers encode a belief about behavior
When you say one hundred requests per minute, you are making a claim. You are saying that a real person, using this the way we intended, will not exceed one hundred requests in a minute. If you have never looked at how people actually use the thing, that claim is a guess, and you are enforcing a guess on real humans.
So before setting a limit I want to know what the top few percent of legitimate users actually do. Not the average, the average is useless here. The edge. The person who uses it hardest without doing anything wrong. Your limit should sit comfortably above them, not on top of them. If you do not know that number, you are not ready to set the limit, you are just ready to pick one.
What happens at the edge is the whole experience
The moment someone hits the limit is a moment in your product, and it usually gets designed by nobody. The default is a bare error with a status code and no explanation, which reads as the product breaking rather than the product having an opinion.
There is a lot of room to do better, and all of it is a product choice. Do you tell them when they can try again. Do you slow them down gradually instead of slamming a door, so heavy use feels like friction rather than a wall. Do you treat a signed-in customer differently from an anonymous one. Do you let a paying user do more than a free one, which turns the limit from a punishment into a feature you can sell. None of that is about servers. It is about what you want the person on the other end to feel.
Different actions deserve different limits
Treating every endpoint the same is the tell that a limit was set by infrastructure and not product. Reading data is cheap and safe, so it can be generous. Sending a password reset email, or anything that costs real money or can be weaponized, should be tight. A single number across everything means you are either too loose on the dangerous actions or too strict on the harmless ones. Usually both at once.
The right limit follows the meaning of the action, not just its cost to the machine. That mapping, from what an action means to how freely it should be allowed, is product work. It cannot be derived from CPU graphs.
Where this leaves me
I still put the enforcement at the infrastructure layer, because that is the correct place for it to live. The gateway is where you can actually stop a request cheaply. What I do not do is let the infrastructure layer decide the numbers. Those come from looking at real behavior, deciding who you are willing to inconvenience, and choosing what the no feels like when it arrives. Do that part well and the infrastructure is just plumbing. Skip it and the plumbing quietly makes decisions about your users that nobody signed off on.