Groq Review

The number that matters: 840 tokens per second on Llama 3.1 8B at $0.05 per million input tokens. That's not a benchmark slide — the pricing page confirms it. For real-time conversational apps or coding assistants where perceived speed is the product, that gap versus GPU inference is meaningful.

Two things give me pause. One: Groq's model catalog is limited to open-weight models — Llama, Mistral, and a handful of others. If your roadmap requires GPT-4-class capabilities, you're still paying OpenAI. Two: no public funding data is available, and custom silicon companies burn cash fast. I'd want to know their runway before standardizing on this.

The OpenAI-compatible API is the smartest thing they've done. Two lines of code to switch base URL and API key means you can pilot without rewriting anything. The Batch API at 50% lower cost with no rate limit impact also signals they're building for production workloads, not just demos.

The board question I'd anticipate in 18 months: what happens if Groq raises prices or narrows the model catalog? The switching cost stays low, which is your answer. Pilot it for latency-sensitive workloads. Don't retire your fallback inference provider.

840 tokens per second on Llama 3.1 8B at $0.05 per million input tokens. Those two numbers together are what makes Groq architecturally interesting rather than just competitively cheap. The LPU isn't a GPU with better drivers — it's purpose-built for sequential, memory-bandwidth-bound inference workloads, which means the speed advantage is structural, not a tuning artifact. If you're building real-time conversational infrastructure or a latency-sensitive coding assistant, that gap versus standard GPU-backed providers like Together AI or Fireworks AI is meaningful at scale.

The two-line migration path — swap base_url, swap API key — is the right integration story. It means your engineering team isn't committing to a proprietary SDK or a novel abstraction layer; the lock-in lives in the routing decision, not the codebase. The Batch API at 50% cost reduction with a 24-hour to 7-day window adds an async workload layer that makes this viable beyond just interactive use cases.

The strategic risk I'd flag to any team considering a 3-year dependency: Groq's model catalog is third-party open-weight models — Llama, Mistral, Kimi K2. If Meta changes its licensing posture or Groq loses a hosting agreement, your model selection narrows without warning. The enterprise tier mentions fine-tuned model support and on-premises deployment, but no public pricing and no documented SLA surfaces from the evidence available.

If we adopt this at scale, in 3 years we have either a durable cost and latency advantage over GPU-incumbent providers, or we've built latency assumptions into our product contracts that a catalog change can break. That's not a reason to avoid Groq — it's a reason to abstract the model routing layer in your own infrastructure before you depend on it.

Token rates are public and specific. Llama 3.1 8B Instant: $0.05 input / $0.08 output per million tokens. Llama 3.3 70B: $0.59 / $0.79. Kimi K2 output hits $3.00/M at the high end. Batch API cuts that 50% for async workloads. Compare to Together AI or Fireworks AI — Groq's rates are competitive, and the OpenAI-compatible endpoint means migration cost approaches zero.

A team running 500M output tokens monthly on Llama 3.3 70B pays roughly $395/month — $4,740/year. Scale to 2B tokens by year 3 with typical growth, and you're at $1,580/month. No seat multiplier. No SSO surcharge. The math stays clean as long as you stay pay-as-you-go.

The enterprise tier is where visibility collapses. On-prem deployments, fine-tuned models, dedicated support — all require contacting sales. No published rates, no contract terms, no auto-renewal window disclosed. For procurement teams, that's a negotiation blind spot. Also: no published overage policy for rate-limit breaches on the free tier. That's a forecasting gap, not a dealbreaker, but budget owners should note it.

Two-line migration to swap from OpenAI is the right call. Change base_url, swap the API key, done. No new SDK to learn, no new request schema, no adapter layer. That's the kind of decision that signals someone on the team actually debugs integration tickets. The JavaScript and Python SDK paths both leverage the existing OpenAI client library rather than shipping a proprietary wrapper — good sign for maintainability.

The speed numbers are where Groq separates from Together AI and Fireworks AI on paper. 394 tokens/sec for Llama 3.3 70B versus 840 for the 8B variant. For a real-time conversational agent or a streaming coding assistant, that gap changes UX behavior, not just benchmark charts. The Batch API at 50% cost reduction with a separate processing queue also tells me someone thought about the pipeline use case, not just the chat demo.

The friction that accumulates is model catalog instability. The docs indicate the catalog is subject to change, and for any production service pinned to a specific model version, that's a rotation risk. Rate limits on the free tier aren't publicly specified in the evidence, which means you're discovering them at runtime. No support email listed publicly either — enterprise inquiries go through a separate access page, which is fine until it isn't.

Prompt caching exists, no extra fee for the feature itself, only charges on cache hits. At $0.50/M cached input tokens for Kimi K2, the math still works for high-repetition prompts. The 24-to-7-day batch processing window is wide — plan your pipeline scheduling accordingly or jobs sit longer than expected.

The number that keeps jumping out is 840 tokens per second for Llama 3.1 8B. That's not marketing rounding. That's a meaningful gap over GPU-based providers like Together AI or Fireworks AI running the same model. For anything real-time — a voice agent, a coding assistant, a chat interface where someone's watching a cursor blink — that gap is felt by the person on the other end. That's a real thing.

The OpenAI-compatible API is genuinely low-friction. Two lines of code to switch your base URL and key. Developers already burned by migration tax from one provider to another will appreciate that Groq isn't asking you to relearn anything. The pricing page shows Llama 3.1 8B at $0.05 per million input tokens, which is cheap enough that you almost don't have to think about it at small scale. The batch API at 50% off is a smart feature for anyone running async pipelines.

Here's the quieter concern: the scored dimensions are mostly API-side gaps dressed as UX gaps. GroqCloud is the console, but there's no support email listed publicly, the website evidence shows a Next.js frontend with Google Analytics, and the mobile story appears to be basically nothing. For a developer tool that's web-only, that's not scandalous — but it's also not a product team that's sweating the daily-use details hard.

Three months in, you're either locked into this for latency reasons or you've diversified. The model catalog changes, the docs indicate no fine-tuning on the standard tier, and enterprise customization requires a separate conversation entirely. Speed is the product. Everything else is still catching up.

Three things I notice before reading the docs. One: 'doesn't flake when things get real' is on the H1 — punchy, but vague. Two: no support email listed anywhere. Three: enterprise tier shows a $0 price, which usually means 'call us.' That's normal for enterprise. Still a flag.

The speed numbers are specific enough to take seriously. 840 tokens/second on Llama 3.1 8B at $0.05/M input tokens. That's not marketing copy — that's a number I can benchmark against. The OpenAI-compatible API is the right move: two-line migration lowers switching cost, which cuts both ways. Easy in, easy out. Exit portability here is genuinely good — if Groq disappears, you're back on Together AI or Fireworks AI with a base_url change.

Long-term is where I hedge hard. Custom silicon is a generational bet. It worked for Google's TPUs behind a massive moat. Groq is building that moat in public, against AWS, against Nvidia, against Cerebras who's running the same play. No public funding data visible on the site. Changelog exists — that's something. But I'd want to see 18 months of model catalog stability before calling this a safe infrastructure dependency.