I am seeing huge latency in the requests for llama 8b instant model in production. Anyone else facing this? The status page is not showing any issues.
Thanks for reporting, we’re looking into this
Could you please paste your request IDs below so we could trace the errors?
The issue solved after few hours but here are few request ids:
req_01k1cd3te2ehttgeshnwrcy5js
req_01k1cdg0sxem2tbs7f35ccbza4
req_01k1ccspt1f16trf6fajp3wsa0 req_01k1cct27ceg3r8bj20bkz8ek4
I have been having alot of problems with this model in batch mode also. Only just recently it started to behave better but still seems slow, before a batch of 40k prompts was done in 15-25 mins. In the last 2 days it took over 10+ hours for 1 batch.
screenshot of it working abit better now:
req_01k1hsrhjwfsp98s0rqgpnvga1
req_01k1hsrh27e6xsgdx3v8m1pv1r
req_01k1hsrh04em0br1kb5bkcjbq7
req_01k1hsrh01fsk8h9bkxqjpekxn
Thank you for the report, I’ve added your request IDs to the issue. Hopefully their fix will tackle all of these errors
It seems the batches are going through faster now but overnight a few of them stalled and are almost complete like 98% done see below the batch ids.
batch_01k1fc8z97fmhvp4x4n6nnfzp
batch_01k1hj9z66e3y87y2ggpv78qkk
batch_01k1hja89wfne9jbcg32wws47y batch_01k1hjahx0fnftwqtephkvt6nt batch_01k1hjbqnqfr4rrab1azhex0s0 batch_01k1hjbdk1fhzbq626np83rnq6
batch_01k1hjavqge47vw72zne260btc
Thank you for reporting, added these to the issue tracker as well