When dealing with fast-moving technology, there’s a conundrum: things that are difficult to do today are likely to be far easier if you just wait a bit for the technology to evolve. For instance, I am building a Virtual College Advisor that gives quality real-time advice to potential college students. This is exceptionally difficult with today’s technology but might be trivial in just a few years with newer technology.

This problem has been true of technology forever but is doubly true with Artificial Intelligence (AI) as major advances are announced daily, this conundrum is called the “Wait Calculation.” The answer to the conundrum, mathematically speaking, is that the shorter the distance and the fewer obstacles you would have to work around, the more it pays to move ahead now.

Technology on the March

The history of computers has always been about coding around the limitations of the hardware. The Y2K ‘crisis’, the fear that all the world’s computer systems would come to a halt at midnight on January 1^st, 2000, due to computer systems storing year data as two digits, was brought about because in the early days of computing, space, both RAM used for internal calculations and hard drive space, were extremely expensive. Saving two digits by storing 1995, as 95, was not an example of poor coding by programmers, in fact, it was a wise move because saving those two digits mattered. Those two digits saved could be used for more important work.

These days, as both RAM and hard drive space are plentiful and inexpensive, it seems crazy, or lazy and sloppy, to not use a four-digit year, but 30 years ago this was a solid business decision. Back in 1990 I was thrilled to purchase a HUGE 70MB hard drive (yes, Megabyte) for a mere $780. That works out to $11,400 per Gigabyte. Today, I see a 2 Terabyte hard drive on Amazon for $64.99 or $.03 per Gigabyte. Not even factoring in inflation, storage is at least 380,000 times cheaper.

With space at such a premium, the poor business decision was using four-digit years when every extra digit meant finding another digit to save elsewhere. The point is, often if you wait long enough the complexities you have to build around go away. Therefore, there is always a tension between whether you should do things now, the hard way, or wait until technology evolves and it will be easier.

How this Impacts AI

I see this clearly as I work building Artificial Intelligence (AI) applications, in the form of token limits. Tokens are how AI represents words. For a more technical look at tokens, Tim Lee has a great primer here. Tokens aren’t exactly words, but the difference is immaterial for this discussion. From here on out I’ll refer to them as ‘words’ even though they aren’t exactly isomorphic.

The token limit is how large each individual response can be. You may have heard about the Context Window, which is rapidly growing with AI models. Token limits differ from the Context Window. The Context Window is essentially how far back ChatGPT can remember your previous conversation whereas the token limit is a limit on each individual prompt and response. Context windows are increasing rapidly, but token limits have been increasing more slowly.

Right now, the limit on tokens in GPT-4 is 4,096 tokens. In essence, this means that the combined total tokens of the input and the output of a query can be no more than 4,096 tokens. This means the larger the input, the smaller the possible output. While 4,096 words is more than enough if you are just chatting with ChatGPT, it’s restrictive when using Retrieval-Augmented Generation AI and feeding GPT-4 the additional information, outside of what it already knows, to answer queries.

In my case, I’m developing a Virtual College Advisor. Let’s say you ask it about good Economics programs. To answer this I must look at many different resources, including the websites of over 6,000 secondary institutions, find the most relevant data, package it up and send it to GPT-4 to get a comprehensive answer. Even after I compress things and remove extraneous data it is often far more than 4,096 tokens worth of information.

Because of the token limit, to get this data to GPT-4 I have to ‘chunk’ it. Divide it into pieces and pass it through. Then I have to take the results, recombine them and give them to the user. This is a LOT of processing and possibilities for error. It’s a less than ideal implementation, but it is what has to be done to get around the token limit.

Another strategy would be to wait. The token limits are increasing and any day a change might come that allows me to pass all of that data in without having to resort to the complicated expediency of chunking it. At the rate AI is evolving, this could happen any day.

The Costs of Moving Ahead and of Waiting

In his thought provoking post, “The Lazy Tyranny of the Wait Calculation,” Ethan Mollick discusses this very issue. He points out that there are some benefits to moving ahead even IF technological advances make your efforts irrelevant.

Chief amongst these is that by doing it now you learn the intricate details of how to do it. For instance, imagine your computer mouse has stopped working and you don’t know how to fix it. You do have a backup, but you’ve never replaced a mouse before and there are no instructions with it. A friend is coming over later to install the replacement. If you attempt to install the replacement yourself a few things might happen:

1. You might figure it out. You will then have your mouse replaced sooner than waiting and you now know how to replace a mouse.
2. You might not figure it out. You might or might not have learned anything and you will have wasted some time, but when your friend gets there, he will replace it for you.
3. You might break something while attempting to fix it yourself, causing a longer delay in your mouse being replaced.

Mice are pretty simple and #3 isn’t very likely, although it is possible.

Broken Gets Fixed, Shitty is Forever

With code, there is a bigger problem. It’s that once built, working code often doesn’t get ‘fixed’. So even though a better way comes about the old code sticks around because of the time and the risk of fixing it. This is what my friend Andrew Clay Shafer calls, Broken gets fixed. Shitty is forever. The simple fact is that once I’ve built a Virtual College Advisor by working around the current limitations, I’m not likely to take the time to rewrite it, even if it’s pretty easy to do so. I’ll put my time towards more useful things. This is the norm, not the exception. Just today I saw a job at a well-known bank advertising for a COBOL programmer. COBOL is a computer language that was antiquated when I went to school decades ago. Banks still use it because it works and they have better things to do than to redo code that’s working.

Conclusion

One of the major goals of my Virtual College Advisor has always been for me to better learn RAG AI at a deep hands-on level. Therefore, I am moving ahead with my project, even though it might be made moot quickly. Nothing is as telling about a product, or a person, as their limitations. Once you know those you know a lot about them. I have found that deep knowledge is often extraordinarily helpful later.

If your personal bias isn’t towards learning in the way mine is, the calculation shifts. In that case, for an AI project, I think I’d wait. With things moving so fast, AI is likely to catch up quickly and make whatever you are looking to do easier. You can use that time saved productively elsewhere.