The choice between an LLM gateway and an SDK is usually argued on setup effort, which is the least important variable in the decision.
Setup effort is real, and gateways generally win it. But it is a one-time cost, and the four architectures available differ on things you live with permanently: what they can see, where they sit when they fail, whose security review they trigger, and which questions they can still answer in eighteen months. Those pull in different directions, and a team that optimizes only for the first one tends to rediscover the others later.
Disclosure, because this article compares approaches we have a stake in: Bear Lumen makes an SDK. We have tried to write the strongest version of every case here, including the ones that cut against us, and to cite primary sources so you can check. Where a claim comes from a vendor with an interest, it is labelled.
The four architectures
| Where it sits | What it observes | Code change | |
|---|---|---|---|
| Gateway / proxy | In the request path, between you and the provider | Anything that transits it: model, tokens, latency, plus whatever headers you attach | Change a base URL. Header propagation if you want attribution |
| In-app SDK | In your process | Whatever your code tells it, including work that never leaves the process | Per call site, or one client wrap |
| OTel auto-instrumentation | In your process, patched at import | Provider client calls, automatically | One init call per service |
| eBPF sensor | In the kernel, on hosts you control | Traffic and syscalls, per process and container | None, where it can run |
Most comparisons treat this as gateway versus SDK. That framing hides the most useful option and obscures where the real line falls.
The floor: token and cost capture is solved, for everyone
Every one of these four gets you tokens, model, provider, and a dollar figure. This is not a differentiator and you should not let anyone sell it to you as one.
The point most often missed is that OpenTelemetry's GenAI auto-instrumentation gets you this with no per-call-site code. It patches the provider client library at import, so one init() per service captures every model call underneath it. That collapses the usual dichotomy: you can have in-process telemetry with roughly gateway-grade deployment effort.
Two things about that layer are worth knowing before you build on it.
There is no cost attribute in the standard. The OTel GenAI semantic conventions define gen_ai.usage.input_tokens and gen_ai.usage.output_tokens, and nothing for dollars. Every cost number you see in every tool in this category is that tool applying its own price table to token counts. Those tables are proprietary, unversioned across vendors, and only as correct as the vendor's last update, which matters more than it sounds given how often providers reprice. Two tools can watch the same traffic and disagree about what it cost.
Those token attributes are still marked "Development," not Stable. Attribute names can change. A cost pipeline built directly on them carries real maintenance risk.
So the floor is commoditized and slightly shakier than it looks. The decision lives above it.
The ceiling: what requires your application to speak
Here is the claim you will hear, and it is wrong as usually stated: gateways cannot do business attribution.
They can. If your app propagates Helicone-Property-* or x-portkey-metadata or LiteLLM tags, per-customer and per-feature attribution works fine through a proxy. This is standard practice and dismissing it is the fastest way to lose the argument. The accurate version is narrower and more interesting: a gateway also needs your app to propagate identity, it is just a smaller and more centralized change, typically one middleware rather than N call sites.
The capability differences that do exist are uneven, and specific. Five gateways, checked against their own documentation:
| Custom dimensions | Session grouping | Nested spans, gateway only | Outcome after the fact | |
|---|---|---|---|---|
| Portkey | Any number of keys, string values to 128 chars | Yes | Yes, real span_id / parent_span_id headers | Yes, feedback API, integer -10 to 10, binds to trace |
| Helicone | Arbitrary key/values, limits not documented | Yes, path-encoded tree | Path strings, not span IDs | Yes, score API, integers and booleans, 10-minute processing delay, binds to request ID |
| LiteLLM | Tags, metadata | Only by forwarding to a downstream tool | No | No. Spend logs are immutable telemetry |
| Cloudflare AI Gateway | Maximum 5 entries | No | No, OTel export only | None documented |
| OpenRouter | None beyond user | No | No | No, cost lookup is read-only |
So two of the five support outcome attribution through the gateway path, and the ceiling is lower than an SDK's: integer and boolean scores rather than floats, categories, or free text, and binding at trace or request level rather than to an individual span.
Portkey's tracing is the strongest of the five: a real parent-child span tree through plain HTTP headers, no SDK, documented as working from cURL. If your mental model is "proxies only see flat request lists," it is out of date.
One clarification, since it comes up: Langfuse is not a gateway and is absent from that table for that reason. Its OpenAI drop-in is an import-level SDK swap, so the request goes directly to the provider and telemetry is sent out of band. Its maintainers have said publicly they do not offer a proxy, citing critical-path and maintenance concerns. Read that as an interested party arguing for its own architecture, but the reasoning is stated plainly enough to evaluate.
But note what every post-hoc design has in common. To attach an outcome, your application must capture the ID the gateway returns, persist it, and make a second API call later. That is the same integration burden an SDK imposes, arriving through a different door. "Zero integration" is true for cost and stops being true the moment you want to know whether the work succeeded.
The gap that is structural rather than a product decision
A gateway sees requests that pass through it. Your retrieval step, your reranker, your tool calls, your validation pass, and your business logic never make a request to a model provider, so a proxy cannot see them, in principle and not as a roadmap gap. As one vendor-neutral comparison puts it, proxies are "blind to internal reasoning, prompt templating logic, or local vector retrieval that happens before the API call."
For a RAG pipeline where retrieval is a meaningful share of latency and cost, that is a large blind spot. For a thin wrapper over a chat completion, it is nothing at all. Which one you are determines how much this paragraph matters to you.
Decision one: the technical case
Latency is mostly a non-argument
Published figures contradict each other, and almost every one is vendor-authored. LiteLLM reports 2ms median and 13ms p99 overhead for itself. Kong, a competitor, publishes a benchmark putting LiteLLM an order of magnitude behind. Portkey markets sub-1ms while community reports run 20 to 40ms with features enabled.
They are not comparable, because nearly all of them measure a mock upstream:
"Almost every one of these benchmarks clocks proxy forwarding against a mock upstream, which removes the one variable that dominates real requests: the model provider's own response time."
Set against a real inference call of 500ms to several seconds, a 2ms hop is 0.1% and a pessimistic 40ms hop is 2%. Per-request latency alone is not a sound basis for this decision in either direction. No independent, real-upstream, streaming-aware benchmark of these gateways appears to exist publicly, so treat every number above as directional.
The risk that does not show up in those benchmarks is the throughput ceiling. A production report against LiteLLM measured roughly 16 requests per second direct versus 9 through the proxy, a 44% reduction, at 500 concurrent requests. That is a single unresolved issue report rather than a benchmark, so weight it accordingly, but it illustrates the category: added latency per request and capacity under load are different failure modes, and gateways are marketed on the first.
Failure modes differ in kind, not degree
A proxy failing takes down your product. An SDK failing takes down your data, quietly.
The proxy side is well understood and has real mitigations: client-side fallback to a direct provider call, circuit breakers, self-hosting in your own VPC. One limit is easy to miss: transparent failover works cleanly before a stream starts, but a mid-stream failure after tokens have shipped has to be handled by your application. For chat UX, that is most requests.
The SDK side is less discussed and deserves more weight than it usually gets. Async telemetry is at-most-once. OpenTelemetry's batch processor defaults to a 2048-span queue, and when production outpaces export, spans are dropped with no error and no warning in most configurations. Langfuse documents that short-lived processes must flush explicitly or lose the buffer, which makes serverless a live hazard.
For cost attribution specifically, that is arguably the worse failure. A missing dashboard is obvious. A dashboard built on quietly lossy telemetry is wrong while looking fine, and you will make a pricing decision on it.
The most honest source on this tradeoff is a vendor selling both. Helicone's proxy-versus-async doc says of the async path that customers get "confidence that if we are going down or if there is a network issue that it will not affect their application," while the proxy path buys "gateway tools such as caching, rate limiting, API key management, threat detection." They pick neither for you.
The polyglot argument, which is the gateway's strongest
An SDK costs you once per language runtime. A gateway is language-agnostic by construction.
If you run a Python backend, Node edge functions, a Go service, and some n8n automations, the SDK path means four integrations, four release cadences, four sets of maintainers, and four opportunities to fall behind. The gateway path means one deployment that sees all of it regardless of origin.
This is the single best reason to choose a gateway and it has nothing to do with setup effort. It is an ongoing-maintenance argument, and it gets stronger as the org gets more heterogeneous.
Coverage is not the same shape for each
A gateway covers what routes through it, and misses the developer who called OpenAI directly, the vendored library with its own client, and the third-party SaaS doing AI on your behalf. An SDK covers what you wrapped. An eBPF sensor covers hosts you control, which excludes Lambda, Fargate, Cloud Run, and most managed platforms outright: privileged containers are not supported on Fargate, and eBPF needs to run on the host.
Each approach has an uninstrumented shadow. The useful question is whether yours is a shape you can see.
Decision two: the security case
| Prompts leave your boundary | Who holds provider keys | Privilege required | |
|---|---|---|---|
| Gateway, managed | Yes, in full | The gateway | None |
| Gateway, self-hosted | No | Your infrastructure | None |
| SDK / OTel | Only if content capture is enabled, and it is opt-in | Your app | Application-level |
| eBPF sensor | Read in-kernel; egress varies | Your app | CAP_BPF or CAP_SYS_ADMIN, privileged DaemonSet |
Three things worth knowing.
Self-hosting neutralizes most of the security case against gateways. LiteLLM, Portkey, and Kong can all run in your VPC. Portkey's self-hosted split keeps "all prompt content and LLM responses within your network" with only operational metrics reaching the control plane. What self-hosting does not remove: the gateway still holds provider keys, still sits in the request path, and is now your operational burden.
The OTel path has the best default privacy posture, and this is underrated. Message content is gated behind an explicit opt-in setting. A proxy sees full payloads by necessity, because it cannot function otherwise.
The eBPF question nobody answers in public. LLM API traffic is TLS. A kernel sensor cannot read token counts off the wire without getting inside that. The academic literature is explicit about how this is done: AgentSight describes attaching uprobes to SSL_read and SSL_write in crypto libraries to intercept decrypted traffic, and Pixie documents the same technique along with its fragility across OpenSSL, BoringSSL, and statically-linked binaries. Commercial vendors selling kernel-level LLM cost attribution generally do not mention TLS, uprobes, or decryption anywhere in their materials. That is a fair and specific question to ask on a call, and both possible answers matter: uprobe interception is a heavy security-review item, while reconciling against provider bills instead means your data is only as timely and granular as the billing export.
Either way, a privileged kernel agent is a serious review. There are documented container-escape CVEs and published cross-container attack research in this space. None of that makes eBPF bad technology. It means the marginal cost is near zero if you already run Cilium or Falco, and it is a large first step if you run no kernel agent today.
Decision three: the product case
This is the one that gets skipped, and it usually decides the outcome.
Who owns the number? A gateway is platform-team infrastructure. An SDK is product-team code. Whichever team owns cost accountability should probably own the mechanism, because the other one will deprioritize it.
What question are you actually answering? Chargeback and showback want completeness across an estate, and tolerate coarse dimensions. Pricing wants cost per unit of delivered value joined to what a customer pays, and tolerates gaps. These are different products and it is fine to want only one.
How reversible is it? A gateway is a config change, so switching is cheap. An SDK is code, which is stickier, but the semantics you defined are yours and portable. Neither is a trap; they are different kinds of commitment.
And the one people regret: history is not backfillable. Cost data is only useful in aggregate over time. If you eventually need cost per resolved outcome and your current tool cannot record outcomes, you do not get a head start when you switch. Your outcome time series begins on the day you cut over. For a pricing decision that wants a year of seasonality, that is most of the value, and no migration recovers it.
That argument cuts both ways, and honesty requires saying so. If you never need outcome data, instrumenting for it was wasted effort.
The framework: two questions, four answers
The variable most people reach for is company size. It is the wrong axis. A 2,000-person company with a central AI platform team and a gateway is the easiest integration in the world, because there is one place to put the hook. A 60-person company with twelve services each calling OpenAI directly is the hardest, at any stage.
Two independent variables actually decide this:
Is there a chokepoint? One place where model calls converge. This sets your integration cost, and it affects the SDK path far more than the gateway path.
Are your semantics legible? Does the code already know what a unit of work is and whether it succeeded? This determines whether outcome attribution is possible at all, for any tool.
| Semantics legible | Semantics absent | |
|---|---|---|
| Chokepoint exists | Setup is cheap either way, so choose on capability. The interesting quadrant | Cost attribution is instant either way. Outcome tracking needs domain work first, and no vendor does that work for you |
| No chokepoint | Gateway or sensor. Both are indifferent to code layout, which is exactly the problem you have | A gateway gets you shallow data for near-zero effort. That is the right first step |
Note the asymmetry in the bottom row, because it is the thing SDK advocates tend to skip: an eBPF sensor's cost is flat regardless of how your code is organized, and an SDK's scales with sprawl. In a badly organized codebase, the zero-code approaches are not just easier, they may be the only ones that ship.
Practical shorthand
Choose a gateway when you run multiple language runtimes; you want failover, semantic caching, key management, or routing anyway (production semantic-cache hit rates of 20 to 45% pay for the thing before attribution enters the argument); you have no chokepoint; or you need coverage fast across teams you do not control.
Choose an SDK when your cost question is semantic (per outcome, per agent step, per unit of value); significant work happens outside model calls (retrieval, ranking, tools); you need margin joined to revenue; you run serverless or managed platforms; or a deep-packet-inspection agent will not clear your security review.
Choose OTel auto-instrumentation when you want in-process capture with minimal integration and are willing to add business dimensions yourself. It is the most underrated option in this list.
Choose an eBPF sensor when you run your own fleet, a large share of spend is infrastructure rather than tokens, you cannot change the code, or you already operate a privileged kernel agent.
And run more than one if it fits. A gateway for request-level cost and routing, plus an SDK for application context, feeding one view, is an increasingly common pattern. These are not mutually exclusive and treating them as a religious choice is a mistake.
Does AI-assisted coding change the math?
It is a fair question in 2026. "No code changes" was compelling when instrumenting forty call sites meant weeks of tedium. If an agent does it in an afternoon, the axis has moved.
The evidence is genuinely mixed, and the strongest study points the other way. METR's randomized trial put sixteen experienced developers on 246 tasks in repositories averaging over a million lines, the closest published analogue to instrumenting a mature codebase. They were 19% slower with AI, and afterward still believed they had been 20% faster. METR is careful that this is a snapshot of early-2025 capability and does not generalize broadly, and they now label it historical.
Against that, GitHub's enterprise research found 12.9 to 21.8% more pull requests per week, and DORA 2025 finds AI adoption correlates positively with delivery throughput while also correlating with higher instability and more rework.
The synthesis that survives both: writing the code was never the expensive part. GitClear's analysis of 211 million changed lines found high-adoption teams merging 98% more pull requests with review time up 91%. Generation got cheaper and review became the bottleneck. Adding an SDK across N services still costs N reviews, N deploys, and one dependency-version coordination, and none of those got faster.
There is a subtler problem specific to instrumentation. It has to be semantically correct: the right customer ID, the right feature tag, the right span parent. That is a correctness class an agent cannot verify locally and a reviewer cannot easily eyeball, because wrong-but-plausible attribution produces a dashboard that looks entirely fine and is wrong. Given only 29% of developers now trust AI output accuracy, down from 40% the year before, that caution seems widely shared.
So the calculus has shifted toward the SDK, less than "the agent writes it now" implies.
A note on vendor durability
One thing that does not appear on any comparison table. Helicone, one of the two gateways with real post-hoc outcome support, was acquired by Mintlify in March 2026 and is in maintenance mode: security updates, new models, and bug fixes continue, with no new roadmap.
That is a fact about one vendor, not an argument about gateways, and every category has consolidation. But if a specific capability is why you are choosing a tool, it is worth checking whether that capability is still being built.
What to actually ask
Whichever direction you lean, these questions separate the architectures faster than a feature matrix:
- What fraction of my variable cost is model tokens versus infrastructure? If it is mostly infrastructure, an LLM-layer tool is solving someone else's problem.
- How many language runtimes make model calls? Multiply that by the SDK integration cost.
- Does meaningful work happen outside model calls? If yes, a proxy cannot see it.
- Will I need to know whether the unit of work succeeded? If yes, ask exactly how the outcome gets attached, and who stores the ID.
- What happens to my data when the collector fails? Loud or silent.
- Who signs off on the security review, and have they seen this shape of agent before?
- Am I building a cost center or a price? They are not the same artifact and they need different data.
The last one is the one that decides it. A backward-looking allocation that lands spend on a team tolerates coarse dimensions and rewards completeness. A forward-looking price needs cost per unit of delivered value, and that fact has to be written down by something that knows what a unit is.
No architecture converts one into the other after the fact.