The cost of AI is coming down rapidly. In many ways, this is a great benefit, but it’s also a huge problem because you can’t plan your budget. Oh, believe me, I’m not complaining¹, but it’s not all upside.

Earlier today, I wrote a post on Graph RAG. One of my main complaints was the cost, with the average cost per query running about $1. No sooner had I posted it than I read about OpenAI’s newest model, GPT-4o-mini. GPT-4o-mini is fairly similar to GPT-4o but costs 1/33rd per token! Now the average query cost is $.03.

That’s a 97% discount!!! Instead of costing $60 to prepare 1,000 documents, it’s closer to $1.80, and instead of $60,000 to prepare the data on my million-record dataset, the cost is now closer to $1,800.

Change is never easy.

Because my RAG AI platform is built modularly and parameter-driven, I should have been able to simply change the model name from “gpt-4o” to “gpt-4o-mini” and save 97%! That worked fine for my core platform (after a quick “pip install openai –upgrade” to make sure my OpenAI APIs had the new model). I ran my automatic evaluations and was very pleased to find the evaluations not notably different from GPT-4o. I often have to tweak prompts after model changes, so this was a pleasant surprise.

But for my experimental Graph RAG, it stopped returning data. A little investigation showed that one of the prompts was behaving differently, and I was not getting the expected outputs back. A quick prompt change and some testing, and it looks great.

Conclusion

This was a big reminder that when reading about AI, don’t trust anything more than a month or two old. Things are simply changing too fast.

It’s also a reminder of how important automatic evaluations are. Being able to test a new model in minutes allows me to quickly take advantage of anything new that comes along.

Ok… enough blogging… I’m off to experiment more with my new, very fast and very inexpensive model!

1. Actually, it’s 97% upside and I’m thrilled! I framed it negatively because I’m trying a little clickbait today just to see how it affects click through rates…