See Groq CEO Jonathan Ross demo Vapi (running on Groq chips) on CNN International
Now that we're 600+ strong in this community (in less than 24 hrs 🤯), I think it's a great time to say hello to you wonderful humans (we're all humans here… right?!) and offer up a formal-ish introduction.
With that, HELLO 
and welcome to the Groq community! We are a group of passionate engineers on a mission to set a new standard for AI inference speed to help accelerate your apps in spaces from healthcare to marketing and beyond.
At Groq, we believe the current state of inference tech just isn't cutting it in today's data-driven world, which is precisely why we developed a revolutionary new architecture that enables the WOW-inducing speed that you've been seeing. You've probably heard of GPUs and CPUs - and today we're excited to teach you all about the new kid on the block called the Language Processing Unit (LPU). Groq is the creator of the LPU Inference Engine, the fastest language processing accelerator on the market.
We are currently offering access to our LPU Inference Engine via the GroqCloud and you can apply for early access here. We usually try to respond to requests within 48 hours, but please be patient with us as we work through the influx of requests we've received in the past day! For now, we're looking for 1) folks building apps that take advantage of speed and 2) folks who do not require a fine-tuned implementation of Llama 2 70B or Mixtral 8x7B. Please also note that we're currently working on transitioning our API access process to a self-serve model that will be available soon!
Stay tuned for exciting news for this community that we'll let you know about soon...
We hope this community helps us educate, build, and simply just work together (and have fun) with like-minded AI enthusiasts to make a difference in the world. We're super excited about all the curiosity around Groq and appreciate all of you asking questions, inquiring, and offering up helpful feedback. We are so thrilled to have you join us on this journey and can’t wait to see all the incredible things you’ll accomplish with Groq. Let’s redefine fast together!
We are excited to announce our brand new LangChain integration!
Package: https://www.npmjs.com/package/@langchain/groq
Docs: https://js.langchain.com/docs/integrations/chat/groq
Huge thanks to jacoblee93 and especially brace for writing the integration and making this a reality!
Hello, friends (ALL 1000+ OF YOU)! We've got more exciting news!
Python LangChain integration now available!
You've seen the JS integration and now it's time to make way for the Python LangChain integration! Huge thanks to Erick and Graden! LangChain is a framework for developing apps powered by LLMs and now Groq is part of that party with our very own JS and Python integrations so that you can build cool stuff. And make sure to post your projects in #channel for us to check out!
Package: https://pypi.org/project/langchain-groq/
Docs: https://python.langchain.com/docs/integrations/providers/groq
On another (sappy) note, it’s only been a few days since we’ve all met each other, but we want to thank you for being such an awesome and collaborative community. We absolutely love hearing all your feedback and working together to make our developer experience a great one!
Hey everyone! We have just deployed a change to our website to force login. In coordination with this, we have revoked the anon_token api which was causing a large amount of unwanted api traffic for us. We believe that this should alleviate capacity issues we have been seeing and should translate to better latencies. We're continuing to work hard on scaling our systems.
As always, please keep the feedback up!
Introducing LlamaIndex Groq integration for Python and Javascript!
LlamaIndex is a data framework for LLM-based applications that benefit from context augmentation, such as Retrieval-Augmented Generation (RAG) systems.
Thanks to Graden and the LlamaIndex team!
Python: https://docs.llamaindex.ai/en/stable/examples/llm/groq.html
Javascript: https://ts.llamaindex.ai/modules/llms/available_llms/groq
Hi, friends (ALL 1800+ OF YOU)! As our very own andreVSworld would say, happy timezone - wherever you're reading this from! Two pieces of exciting news today...
1) Powered by Groq Badges for Your Apps!
So many of you have been integrating Groq into your projects and building the coolest demos we've had the pleasure to try out. Our team was so excited about this that they thought, "hey, why don't we have badges?!", and that's how our "Powered by Groq" badges were born. Maybe that's how all badges are born...
If you're actively building with Groq API and integrating Groq into your work, we encourage you to use our brand new and shiny "Powered by Groq" badges to let everyone know that your creations are powered by Groq, the world's fastest inference engine for LLMs. Here are all the different badges we've made just for you. As for language, we'd greatly appreciate incorporating terms like "... powered by Groq", "... built on Groq", and "... developed on Groq" into your descriptions.
2) Gemma Teaser!
Ask and you shall receive, my friends! Quite a few of you have been asking about Gemma. We've already been telling you that there are things in the works, but that's definitely not as exciting as seeing it with our own eyes. In fact, I was super excited to see this teaser demo that was randomly dropped in one of our internal engineering channels just 15 minutes ago!
HOT OFF THE PRESS (Slack) and straight to your door (screen), we present to you a preview of Gemma powered by Groq in all its glory! How many of you would like to actually try it out?!
Our beloved Gavin, in his short and sweet style, posted the news in the above link... but I want to share the news with a few more words with all 2,600+ of you! It's official, folks - Groq is super excited to announce that we have acquired Definitive Intelligence and will be cooking up many more exciting things in the near future!
You can read more about the acquisition here. All aboard the Groq-etship!
Hi, friends (4,396 friends to be exact)! We hope your weekend is off to a great start. Fridays are usually when most people wind down for the weekend, but Groq is about to turn things UP! The Groqetship is fully fueled on all the excitement and love we've been receiving from all of you, so we've been hard at work...
As you know, we had recently dropped a teaser for Gemma...
Well, my friends, it's no longer just a teaser! Gemma-7b-it is officially available on Groq! It's the real deal and you can head over to https://groq.com or https://console.groq.com to see for yourself!
If you're like us and enjoy listening to podcasts or music while you tinker around with APIs to build cool stuff, we also want to share the latest episode of The All-in Podcast where Groq has been the talk for the third week in a row! Our very own Sunny Madra, Head of GroqCloud, gives an update on us starting at 44:46!
Happy exploring and peace out, or as Chamath would say - Groq, Groq!
JSON mode (beta) for our API service!
With JSON mode, you can now ensure that all chat completions are in valid JSON format. Simply set "response_format": {"type": "json_object"} in your chat completion request then use the system prompt to describe the schema of your JSON output. For more info and examples, check out the JSON mode docs.
Please note: JSON mode is in beta and we do not currently support an output schema or streaming responses.
Technical details: we have implemented JSON mode in the following manner:
- We inject a small system prompt directing the LLM to output JSON format
- We correct schema errors often made by the LLM, such as eliminating unnecessary escapes like those before underscores (e.g.,
"bad\_escape") - If the response is still invalid JSON, we go back to the LLM, providing any parse errors encountered, and request it to correct the JSON
This is our first release of this feature, and as our technology matures, we plan to improve the implementation with better techniques such as constrained decoding.
Please tell us about your experiences with JSON mode so we can continue to improve it.
Hi, friends (ALL 7,070 OF YOU)! I'm excited to let you all know that our Groq API Cookbook is now fully cooked and available for you to garnish with your contributions!
So many of you have been sharing the greatest tips, building the coolest apps, and recording the most informative demos. Instead of letting that wisdom get lost in various channels and platforms, we figured it would be good for it all to live on in a repo that our open-source community can use as a resource to both learn from and build with.
We’re looking for contributions in the form of tutorials, Jupyter Notebooks, and guides - can’t wait to see what you AI chefs cook up to contribute to the cookbook!
P.S. Special shout out to our very own amazing engineer, dloman1108, for the first contribution of a tutorial for extracting social determinants of health from clinical notes with JSON mode! Check it out here.
Things are moving fast... 2 announcement posts in one day for a last minute, hot-off-the-press update... drumroll…
Tool Use/Function Calling (beta) for Groq API is now available!
The Tool Use/Function Calling feature allows models available on GroqCloud to take user-defined functions as inputs and generate structured output to invoke them from external tools or codebases. The ability to generate structured JSON through functions enables consistent and predictable outputs for all the cool applications you’re building with Groq API.
So excited about all the new possibilities for your applications powered by Groq and cannot wait to hear your feedback in the community!
See the docs here: https://console.groq.com/docs/tool-use
Hi All -- As you know, we have been absolutely awed and overwhelmed by the massive interest and excitement around Groq. It’s been exciting to see so many of you wanting to build your apps powered by Groq and we are committed to enabling that. Due to the unexpected response and excitement, we are extending our free beta of GroqCloud as we continue to rapidly scale out to build more capacity and meet everyone’s needs. This means that it will take a little longer than originally expected for us to launch paid access to higher rate limits.
During our extended free beta, we will continue to listen and learn from your invaluable feedback, ship new models and features you’re interested in, and not only be the fastest inference platform, but the platform built with devs, by devs, and for devs.
Once again, we appreciate the tremendous support and are excited to serve your need for speed (among other things)! For any questions, please feel free to continue interacting with Groqsters within the community and stay tuned for fun updates in the works that this community will be the first to know about.
Some end of week updates:
- Upgraded Gemma to Gemma v1.1
- Added Errors page to documentation
- Tool Use: reduced tool use to single tool at a time, enforced
tool_choice - Testing
Whisper-Large-v3with small amount of users
Hi, friends (ALL 7,945+ OF YOU)! Once again, although Fridays are when most people wind down for the weekend, Groq is about to turn things UP! We have extremely exciting news to share with you today that I personally can't wait to see your reaction for...
1) LLAMA3 70B & LLAMA3 8B NOW LIVE ON GROQ!
So many of you have been asking and asking while we've been listening and listening! Llama 3 has been all the rage within this community to the point where there were requests even weeks ago to make it available once it was released. Well, the time is now, folks! Llama 3 powered by Groq and ready for you to explore!
We absolutely cannot WAIT to see what you cook up and... speaking of cooking up... if you create any tutorials or guides using Llama 3 or any of the models powered by Groq, we would love for you to contribute to our Groq API Cookbook.
2) New Groq homepage just dropped!
Don't you love the look and feel of a sleek, minimalist website? Make sure to also visit our homepage to play around with Llama 3 and to check out the awesome work of our engineers. We'd love to hear your feedback!
Sometimes new processes prove not to be the best! 
Re-posting this announcement from earlier in the month as we are getting a lot of interest post Llama-3 launch.
Hi All -- As you know, we have been absolutely awed and overwhelmed by the massive interest and excitement around Groq. It’s been exciting to see so many of you wanting to build your apps powered by Groq and we are committed to enabling that. Due to the unexpected response and excitement, we are extending our free beta of GroqCloud as we continue to rapidly scale out to build more capacity and meet everyone’s needs. This means that it will take a little longer than originally expected for us to launch paid access to higher rate limits.
During our extended free beta, we will continue to listen and learn from your invaluable feedback, ship new models and features you’re interested in, and not only be the fastest inference platform, but the platform built with devs, by devs, and for devs.
Once again, we appreciate the tremendous support and are excited to serve your need for speed (among other things)! For any questions, please feel free to continue interacting with Groqsters within the community and stay tuned for fun updates in the works that this community will be the first to know about.
PSA: Our devs have deployed a software update that increases our capacity and also increases per user throughput for Llama 3 70B. Look out for 2% faster tokens per second!
Update: We're phasing out our Llama2 models in the next 48 hours to make way for increased capacity with the more advanced Llama3 models, featuring 70B and 8B architectures. If you're currently using our older models, we recommend migrating to Llama3 for improved performance and capabilities.
Llama2 has been taken down.
Excuse us while we’re under construction - in an attempt to make our server sleeker and more organized, we are polishing up by removing channels that don’t get a lot of activity or views. We will be archiving the education, media, and research channels by end of week.
Please see an update to our Terms & Conditions. Especially of note is Section 5 regarding Generative AI which includes the below:
Groq does not retain Prompts, Output or your Training Data on its servers.
Vercel has added support for Groq as a provider to their AI SDK!
To learn how to create a custom provider instance powered by Groq, check out their docs: [https://sdk.vercel.ai/providers/ai-sdk-providers/groq](https://sdk.vercel.ai/providers/ai-sdk-providers/groq)
If you build anything cool, share it in the community or in the Groq API Cookbook so others can see your work!
We're excited to announce the launch of our GroqChamps Ambassador Program!
The GroqChamps Ambassador Program recognizes and rewards exceptional Groq community members who help make the Groq community better by contributing their knowledge and expertise to help other members. This program is designed to create a positive, helpful, and supportive community and to improve members' experience with Groq.
If you have a passion for helping others, knowledge about Groq and GroqCloud, technical expertise, and regularly participate in our community, please reach out to us at groqchamps@groq.com with your resume and username!
To learn more about GroqChamps, see the PDF attached below. We can't wait to collaborate with you!
Groq API now supports combining streaming with tool use!
This new feature enables you to use streaming responses together with tool calling for even more powerful applications. Have fun building!
Reply
Login to the community
No account yet? Create an account
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.