Groq Review

The number that put Groq on the map: 1,000 tokens per second on GPT OSS 20B at $0.075 per million input tokens. That's not a benchmark slide — Groq published it. For real-time conversational apps or coding assistants where perceived speed is the product, that gap versus GPU inference is meaningful.

Two things give me pause. One: Groq's self-serve catalog is limited to open-weight models — GPT OSS, Qwen, Whisper, and its own Compound systems, with Llama now behind an enterprise contract. If your roadmap requires GPT-4-class capabilities, you're still paying OpenAI. Two: they've raised $350 million at a $3.5 billion valuation on top of $650 million in June, which is modest capital for a company planning to go from 54 to 200-plus megawatts by 2027. Silicon and data center build-outs burn cash fast.

The OpenAI-compatible API is the smartest thing they've done. Two lines of code to switch base URL and API key means you can pilot without rewriting anything. The Batch API at 50% lower cost with no rate limit impact also signals they're building for production workloads, not just demos.

The board question I'd anticipate in 18 months: what happens if Groq raises prices or narrows the model catalog? The switching cost stays low, which is your answer. Pilot it for latency-sensitive workloads. Don't retire your fallback inference provider.

Early on: 1,000 tokens per second on GPT OSS 20B at $0.075 per million input tokens. Those two numbers together are what makes Groq architecturally interesting rather than just competitively cheap. The LPU isn't a GPU with better drivers — it's purpose-built for sequential, memory-bandwidth-bound inference workloads, which means the speed advantage is structural, not a tuning artifact. If you're building real-time conversational infrastructure or a latency-sensitive coding assistant, that gap versus standard GPU-backed providers like Together AI or Fireworks AI is meaningful at scale.

The two-line migration path — swap base_url, swap API key — is the right integration story. It means your engineering team isn't committing to a proprietary SDK or a novel abstraction layer; the lock-in lives in the routing decision, not the codebase. The Batch API at 50% cost reduction with a 24-hour to 7-day window adds an async workload layer that makes this viable beyond just interactive use cases.

The strategic risk I'd flag to any team considering a 3-year dependency: Groq's self-serve catalog is third-party open-weight models — GPT OSS, Qwen, Whisper — plus its own Compound systems. It has already narrowed once: Llama is enterprise-only now, so a licensing shift or a lost hosting agreement can move a model behind a sales call without warning. The enterprise tier mentions fine-tuned model support and on-premises deployment, but no public pricing and no documented SLA that I could find.

If we adopt this at scale, in 3 years we have either a durable cost and latency advantage over GPU-incumbent providers, or we've built latency assumptions into our product contracts that a catalog change can break. That's not a reason to avoid Groq — it's a reason to abstract the model routing layer in your own infrastructure before you depend on it.

Token rates are public and specific. GPT OSS 20B runs $0.075 input / $0.30 output per million tokens. GPT OSS 120B runs $0.15 / $0.60. Qwen3.8-27B output hits $4.00/M at the high end, while the Llama models read 'Contact Sales' instead of a price. Batch API cuts that 50% for async workloads. Compare to Together AI or Fireworks AI — Groq's rates are competitive, and the OpenAI-compatible endpoint means migration cost approaches zero.

A team running 500M output tokens monthly on GPT OSS 120B pays roughly $300/month — $3,600/year. Scale to 2B tokens by year 3 with typical growth, and you're at $1,200/month. No seat multiplier. No SSO surcharge. The math stays clean as long as you stay pay-as-you-go.

The enterprise tier is where visibility collapses. On-prem deployments, fine-tuned models, dedicated support — all require contacting sales. No published rates, no contract terms, no auto-renewal window disclosed. For procurement teams, that's a negotiation blind spot. Also: no published overage policy for rate-limit breaches on the free tier. That's a forecasting gap, not a dealbreaker, but budget owners should note it.

Two-line migration to swap from OpenAI is the right call. Change base_url, swap the API key, done. No new SDK to learn, no new request schema, no adapter layer. That's the kind of decision that signals someone on the team actually debugs integration tickets. The JavaScript and Python SDK paths both leverage the existing OpenAI client library rather than shipping a proprietary wrapper — good sign for maintainability.

The speed numbers are where Groq separates from Together AI and Fireworks AI on paper. 500 tokens/sec for GPT OSS 120B versus 1,000 for the 20B variant. For a real-time conversational agent or a streaming coding assistant, that gap changes UX behavior, not just benchmark charts. The Batch API at 50% cost reduction with a separate processing queue also tells me someone thought about the pipeline use case, not just the chat demo.

The friction that accumulates is model catalog instability. The docs indicate the catalog is subject to change, and for any production service pinned to a specific model version, that's a rotation risk. Batch and Flex processing are gated behind the Developer plan, so the tier you prototype on isn't the tier your pipeline runs on. No support email listed publicly either — enterprise inquiries go through a separate access page, which is fine until it isn't.

Prompt caching exists, no extra fee for the feature itself, only charges on cache hits. At $0.0375/M cached input tokens for GPT OSS 20B, the math still works for high-repetition prompts. The 24-to-7-day batch processing window is wide — plan your pipeline scheduling accordingly or jobs sit longer than expected.

The number that keeps jumping out is 1,000 tokens per second for GPT OSS 20B. That's not marketing rounding. That's a meaningful gap over GPU-based providers like Together AI or Fireworks AI running the same model. For anything real-time — a voice agent, a coding assistant, a chat interface where someone's watching a cursor blink — that gap is felt by the person on the other end. That's a real thing.

The OpenAI-compatible API is genuinely low-friction. Two lines of code to switch your base URL and key. Developers already burned by migration tax from one provider to another will appreciate that Groq isn't asking you to relearn anything. Published rates go as low as $0.03 per million input tokens, which is cheap enough that you almost don't have to think about it at small scale. The batch API at 50% off is a smart feature for anyone running async pipelines.

Here's the quieter concern: the scored dimensions are mostly API-side gaps dressed as UX gaps. GroqCloud is the console, but there's no support email listed publicly, the website evidence shows a Next.js frontend with Google Analytics, and the mobile story appears to be basically nothing. For a developer tool that's web-only, that's not scandalous — but it's also not a product team that's sweating the daily-use details hard.

Three months in, you're either locked into this for latency reasons or you've diversified. The model catalog changes, the docs indicate no fine-tuning on the standard tier, and enterprise customization requires a separate conversation entirely. Speed is the product. Everything else is still catching up.

Three things I notice before reading the docs. One: 'the premier neocloud for fast inference' is the H1 — confident, but that's a category claim, not a capability. Two: no support email listed anywhere. Three: the Llama models and the enterprise tier both say 'Contact Sales' instead of a price. That's normal for enterprise. Still a flag.

The speed numbers are specific enough to take seriously. Early on: 1,000 tokens/second on GPT OSS 20B at $0.075/M input tokens. That's not marketing copy — that's a number I can benchmark against. The OpenAI-compatible API is the right move: two-line migration lowers switching cost, which cuts both ways. Easy in, easy out. Exit portability here is genuinely good — if Groq disappears, you're back on Together AI or Fireworks AI with a base_url change.

Long-term is where I hedge hard. Custom silicon is a generational bet. It worked for Google's TPUs behind a massive moat. Groq is building that moat in public, though it has stopped fighting one rival: it's an NVIDIA Cloud Partner now, and LPX racks pair 256 LPUs to NVIDIA's Vera Rubin rather than replacing it. A $350 million Series A at a $3.5 billion valuation is on the site. Changelog exists — that's something. But I'd want to see 18 months of model catalog stability before calling this a safe infrastructure dependency.