We’ve been playing around with the Batch API, and have some suggestions for the Groq Console:
- “In progress” batches are not currently displayed, so unable to monitor from console
- Logs for completed batches, or entries within the batches are not made available in Logs
- Cost/Token usage for completed batches is not made available in Usage/Logs
- Any batch completions are not shown in Usage or Activity
In addition, Batch Input tokens are specified on the Invoice, but the Invoice is not split by Project, nor reference a project, so it’s hard even to track costs there.
Whilst we will look to continue using Batch, as it’s so much cheaper for large token projects, the above improvements would allow us to monitor usage more effectively and provide me (as CTO) the confidence the team can monitor usage themselves.