Further exploring Llama Index’s implementation of Graph RAG, I find that it sometimes produces significantly more accurate answers but with a price to pay.
I spoke in a previous post about my disappointing experiences with Graph Retrieval Augmented Generation (Graph RAG). Graph RAG searches your data by relationships instead of by semantics and has great potential to yield more accurate and complete results. However, data preparation and simple queries require large numbers of calls to LLMs, leading to significant costs in both time and money. These calls are complex enough that only the latest LLMs can handle them, making it challenging to find open-source alternatives.
Initially, I planned not to explore Graph RAG further due to the high costs involved, including an estimated $1,800 for data preprocessing and over $1 per query. However, just as I was giving up, gpt-4o-mini was launched with prices 97% lower than gpt-4o, reducing my costs to under $100 for setup and $0.03 per query. At these prices, it was worth exploring further. My Graph RAG implementation is loosely based on Microsoft’s publicly available version.
Half baked?
In searching for a test bed where Graph RAG might shine, I thought about baking. I have a baking RAG AI that provides comprehensive answers to questions about baking, focusing on cookies and cakes. It’s adequate but doesn’t match the quality of results from other RAG AI implementations, like my makeup RAG. This seemed like an area where Graph RAG could make a difference, especially since techniques for making cookies might offer insights into cakes, and vice versa. That was the hypothesis, anyway.
My baking RAG AI encompasses over 8,000 documents focused on how-tos rather than recipes, though recipes are included.
Types of Graph RAG
If you want AI to provide answers from your private data and need those answers to be as accurate as possible, Graph RAG AI may work for you. There are two types of Graph RAG:
- Global: Allows you to ask questions of your entire dataset, such as “Which themes get the most likes for my baking videos?”
- Local: Provides better answers to specific questions by exploring relationships, like “I’m out of lemon peel for my cake. What can I use instead?”
For this post, I focused on the global style, as I believe the ability to get general answers from a large dataset is potentially more impactful than more precise answers.
How Global Graph RAG works
To look across all documents without examining the entirety of each document every time, Graph RAG preprocesses the data:
- Each document is fed into an LLM with instructions to extract any entities and the relationships between them. For example, “Ricotta” > “Cheesecake” indicates that Ricotta is a type of cheesecake.
- These entities create a tree structure called a Graph.
- Nodes on the graphs are clustered into “communities” of related information.
- Each node is summarized to indicate what the community is about. For example, a summary might note: “The relationships between sour cream and plain yogurt highlight their interchangeability as dairy substitutes. Both entities can be used in recipes to achieve a similar texture and flavor, making plain yogurt a viable alternative to sour cream.”
At query time, Graph RAG passes the query along with each community summary to the LLM. The LLM then consolidates responses from all summaries to generate the final answer.
Question #1
I asked both my RAG AI and Graph RAG five baking questions and compared the results. Although I strongly advocate for automating evaluations, these results were compared manually (meaning, I reviewed them) because automating evaluations involves creating rubrics, which is tough without seeing initial results. If I continue with Graph RAG, I will certainly create automated evaluations.
Given that baking suggestions are ubiquitous online, I also asked gpt-4o these questions as a baseline. The first query, “What are the main ingredients commonly used in baking recipes?” produced similar answers across gpt-4o, RAG AI, and Graph AI. The main difference was in timing: gpt-4o and RAG AI responded almost instantly, whereas Graph AI took much longer.
Query: What are the main ingredients commonly used in baking recipes?
ChatGPT Response
The main ingredients commonly used in baking recipes include:
- Flour: Essential for providing structure to baked goods.
- Sugar (granulated and powdered): Adds sweetness and can affect texture.
- Eggs: Provide moisture, structure, and leavening.
- Butter: Adds richness and moisture.
- Baking Powder: A leavening agent that helps baked goods rise.
- Salt: Enhances flavor and balances sweetness.
Flavoring agents such as vanilla, almond extract, and brown sugar are also commonly used to enhance taste. Some unique recipes might include additional ingredients like tahini or coconut for distinct flavors. Accurate measurement of these ingredients is crucial for successful baking results.
RAG Response
When it comes to baking, there are a few essential ingredients that you’ll find yourself using quite frequently. According to Good Food, here are some of the staples you should always have in your pantry:
- Baking Powder: This is a raising agent made up of bicarbonate of soda, which helps your baked goods rise.
- Flour: Usually, all-purpose flour is a go-to for many recipes, but sometimes you might need specialized flours like cake flour or bread flour.
- Sugar: Granulated sugar is commonly used, but brown sugar, powdered sugar, and other types might be called for depending on the recipe.
- Butter: Unsalted butter is typically preferred so you can control the salt content in your baked goods.
- Eggs: These provide structure, leavening, and moisture.
- Milk: It adds moisture and can help with the texture of your baked items.
- Salt: Even in sweet recipes, a pinch of salt can enhance flavors.
- Vanilla Extract: This is a common flavoring agent that makes your baked goods taste delicious.
For more details, you can check out Joy of Baking, which also lists these and other essential baking ingredients in their comprehensive guide. Happy baking!
Graph RAG Response
In baking, key ingredients include:
- Flour: Essential for a variety of baked goods like cakes, cookies, and bread.
- Sugar (granulated and powdered)
- Eggs
- Butter
- Baking Powder
- Salt
These ingredients are crucial for a variety of baked goods like cakes, cookies, and bread. Additional flavoring agents such as:
- Vanilla
- Almond Extract
- Brown Sugar
These enhance the taste. Unique recipes may feature additional ingredients like tahini or coconut for distinct flavors. Accurate measurement of these ingredients is crucial for successful results. For more baking insights or recipes, resources like “Once Upon a Chef,” “The Kitchn,” and the Good Food Magazine app are recommended.
Answer questions on a custom dataset
Of course, RAG AI works best on data that the LLM (like gpt-4o-mini) has never seen, offering a way to extract answers without needing to train or fine-tune a model, which can be extremely costly.
The next question, “What are the most popular types of baked goods mentioned in the documents?” aimed to leverage Global Graph RAG’s strengths. As expected, ChatGPT’s response was general and unrelated to the specific documents. RAG AI found three documents and extrapolated an answer, but it was generic—mentioning ‘cakes’ and ‘cookies’. Graph RAG, however, provided a richer set of specific answers, including items like Buttermilk Coconut Waffles and Unicorn Cake.
ChatGPT-4o Response
The most popular types of baked goods mentioned in the documents are cakes, cookies, and bread. These are commonly highlighted as examples of items that use the key baking ingredients like flour, sugar, eggs, butter, baking powder, and salt.
RAG AI Response
From what I can see in the provided documents, it looks like the most popular types of baked goods include a variety of items:
- Cakes – Nothing’s cozier than a kitchen filled with baking smells, and homemade cakes are among the most mentioned delights. (source)
- Cookies – Mentioned various times, including meringue cookies and shortbread cookies. (source)
- Biscuits – Specifically, Southern Cathead Biscuits are highlighted. (source)
- Brownies – The site also talks about a delicious-sounding skillet brownie. (source)
- Bread – Various types like brioche and cornbread are noted as popular. (source)
- Flatbreads and Danishes – Including lahmajun (Armenian flatbread) and cheese danishes. (source)
On King Arthur Baking, there’s a strong emphasis on bread as well, noted as a favorite among their audience. (source)
From cakes to cookies to a variety of breads, these baked goods are very popular among baking enthusiasts!
Graph RAG Response
The baking community is characterized by a dynamic blend of traditional and innovative baked goods, featuring favorites like:
- Carrot Cake
- Various Cheesecakes
- Lemon Blueberry Bread – which uses Baking Powder.
Recent trends highlight unique items such as:
- Apple Pear Gruyere Pie
- Lemon Cranberry Pound Cake
- Tahini Cookies
- Vegan Frosting
- Edible Cookie Dough – with an emphasis on high-quality and gluten-free ingredients.
Popular treats include:
- Coffee Cake Muffins with streusel topping
- Kumquat Desserts
- Unicorn Cake
While some discussions have ventured into unrelated areas like Persian cuisine and emotional themes, the overall landscape reflects a rich diversity in flavors and styles, with notable mentions including:
- Meringue
- Buttermilk Coconut Waffles
- Peanut Butter Sandwich Cookies
- Cornmeal Cherry Jam Sandwich Cookies – catering to a wide range of tastes and dietary preferences.
What level of Entity?
I assumed the next question would also favor Graph RAG: “What are some common baking techniques or methods discussed?” Given that gpt-4o wouldn’t know specifics from the documents and RAG AI would sample only a few, I expected Graph RAG to excel. However, it did not, effectively giving up on the question.
After several hours of investigation, I discovered the issue lay in entity creation. Graph RAG interpreted entities as nouns—things that have relationships. Therefore, verbs like “mix,” “fold,” and “bake” were rarely recognized. For instance, “Mixer” appeared 29 times, but “mix,” essential to baking, appeared only once. I’ll need to refine my script to ensure verbs are included, at least the important ones.
ChatGPT-4o Response
Here are some common baking techniques or methods that are often discussed:
- Creaming Method: This involves beating butter and sugar together until light and fluffy. This method is commonly used in cake and cookie recipes to incorporate air into the batter, which helps the baked goods rise.
- Folding: A gentle mixing technique used to combine a light, airy mixture (like whipped cream or egg whites) with a heavier mixture (like batter) without deflating the air bubbles. It’s often used in recipes like soufflés and mousses.
- Blind Baking: Pre-baking a pie crust or tart shell before adding the filling. This technique helps to prevent a soggy bottom by ensuring the crust is fully cooked.
- Sifting: Passing dry ingredients like flour, cocoa powder, or powdered sugar through a sieve to break up clumps and aerate them, ensuring a smoother texture in the final product.
- Kneading: A method used in bread making to develop gluten in the dough, which gives the bread structure and chewiness. It involves folding and pressing the dough with the heels of your hands.
- Proofing: The process of allowing dough to rise before baking. This is crucial in yeast bread recipes to develop flavor and texture.
- Whipping: Beating ingredients like cream or egg whites to incorporate air and increase volume. This is essential for making whipped cream, meringues, and some types of cake batters.
- Tempering: A technique used when working with chocolate or custards. In chocolate, it involves melting and cooling chocolate to a specific temperature range to achieve a glossy finish and snap. In custards, it involves slowly mixing hot liquid into eggs to prevent curdling.
- Scoring: Making shallow cuts on the surface of bread dough before baking. This allows the bread to expand properly and adds decorative effects.
- Bain-Marie (Water Bath): A method used to gently cook delicate dishes like custards or cheesecakes by placing the baking dish in a larger pan filled with hot water. This helps to regulate the temperature and prevent cracking.
- Fermenting: A process where dough is allowed to rest and rise, often overnight, to develop flavor. It’s commonly used in sourdough and artisan bread baking.
- Docking: Pricking the surface of pastry dough with a fork before baking to allow steam to escape, preventing the dough from puffing up.
These techniques are foundational in baking and are used to achieve specific textures, flavors, and appearances in baked goods. Understanding and mastering these methods can significantly improve baking outcomes.
RAG AI Response
Ah, you’re interested in common baking techniques and methods! Let me share a few insights from different resources:
- Baking Bread: According to Max Bernstein on Serious Eats, understanding the science behind baking bread is crucial. This involves techniques such as proper kneading to develop gluten, allowing the dough to ferment properly, and knowing the ideal baking temperatures. Each step, from mixing to proofing and baking, can significantly affect the final product. For more detailed guidance, you can check out the full article on breadmaking: The Science of Baking Bread (and How to Do It Right).
- Using Different Types of Flour: Different flours have different protein contents, which can affect the texture of your baked goods. For instance, bread flour has a higher protein content compared to cake flour, which makes it more suitable for chewy bread rather than tender cakes. Serious Eats has a thorough guide on this: Types of Flour: A Guide.
- Silicone Baking Mats: While convenient, silicone baking mats can sometimes lead to cookies that spread too much and become too thin. Adjusting your baking times and temperatures might be necessary when using these mats. You can dive deeper into this on the same guide mentioned earlier: How Silicone Baking Mats Are Ruining Your Cookies.
- Fermenting Sourdough: Proper fermentation is key for a well-developed sourdough. Managing the time and temperature during the fermentation process can influence the flavor and texture of the bread. This is also covered in the baking guides on Serious Eats: How to Ferment Sourdough Properly.
These are just some of the techniques discussed in the baking community. If you want to dive deeper, the Serious Eats guides are a great place to start!
Graph RAG Response
The aggregated information indicates that community summaries on baking often overlook critical techniques such as kneading, proofing, and mixing, instead prioritizing themes like customer service and emotional connections. Specific methods are infrequently addressed, with some discussions straying into unrelated topics. However, key baking practices can be inferred, emphasizing the significance of techniques like whipping for Meringue, flavor enhancement through extracts, and proper hydration of flour for dough consistency.
Highlighted key baking techniques include:
- Mixing: Crucial for achieving the right balance in doughs and batters.
- Creaming: Combines sugar and butter for a light texture, essential in cookies and cakes.
- Whipping: Incorporates air into egg whites for Meringue.
- Flavoring: Enhances baked goods with various flavor agents.
- Baking: Affects the final texture and consistency of baked items.
For a thorough understanding of these techniques, it is advisable to consult dedicated baking resources like Allrecipes or the Food Network. Additional context can help address specific baking method inquiries.
Lack of detail
The next question was: Are there any specific baking tools or equipment frequently mentioned? Here again, Graph RAG underperformed. GPT-4o, lacking access to specific documents, gave generic answers. RAG AI provided detailed answers from three specific documents.
The problem with Graph RAG’s response was the generality of the summaries. They lacked the detail needed for specific answers. I suspect this query would perform better with the Local version of Graph RAG. I could tweak the prompts to include more details, but there’s a risk of overloading the summaries.
ChatGPT Response
- Mixers:
- Stand Mixer: A versatile tool for mixing, kneading, and whipping. It often comes with attachments like dough hooks, paddles, and whisks.
- Hand Mixer: A portable alternative to stand mixers, useful for mixing batters and whipping cream.
- Baking Pans and Sheets:
- Cake Pans: Available in various shapes and sizes, including round, square, and rectangular. They are essential for baking cakes and brownies.
- Baking Sheets: Flat metal trays used for baking cookies, roasting vegetables, and more.
- Muffin Tins: Used for making muffins, cupcakes, and other small baked goods.
- Measuring Tools:
- Measuring Cups and Spoons: Essential for accurately measuring ingredients like flour, sugar, and liquids.
- Kitchen Scale: Provides precise measurements, especially useful for baking bread and pastries where accuracy is crucial.
- Rolling Pins: Used for rolling out dough for pastries, cookies, and pie crusts.
- Silicone Baking Mats: Non-stick mats that are placed on baking sheets to prevent sticking and make cleanup easier.
- Pastry Brushes: Used for applying glazes, egg washes, or butter to baked goods.
- Oven Thermometer: Ensures the oven is at the correct temperature, which is vital for consistent baking results.
- Cooling Racks: Metal racks that allow air to circulate around baked goods, helping them cool evenly.
- Parchment Paper: Non-stick paper used to line baking sheets and pans, making it easier to remove baked goods and reduce cleanup.
- Spatulas and Scrapers: Used for mixing, folding, and scraping batter or dough from bowls.
- Sifters: Used to aerate flour and other dry ingredients, ensuring a smooth and even texture in baked goods.
- Pie Weights: Used to prevent pie crusts from puffing up or shrinking during blind baking.
- Baking Stones: Used for baking pizzas, breads, and pastries to achieve a crispy crust.
- Pastry Bags and Tips: Used for decorating cakes and cookies, filling pastries, and piping dough.
- Dough Scrapers: Handy for dividing and shaping dough, and for cleaning work surfaces.
These tools and equipment are staples in the baking world, helping bakers achieve consistent and professional-quality results. Whether you’re a beginner or an experienced baker, having the right tools can make a significant difference in your baking outcomes.
RAG Response
If you’re looking to get started with baking and need some recommendations for tools and equipment, you’re in the right place! Here’s a rundown of some useful resources and what you can find there:
- MikeBakesNYC:
Mike has compiled a comprehensive list of his favorite bakeware, ingredient brands, and various baking tools. Whether you need a new mixing bowl or are on the hunt for the perfect cookie sheet, his recommendations are tried and tested. Plus, many of the links are affiliate links, so purchasing through them can help support his work. You can find his complete list here: MikeBakesNYC Products.
- Baker’s Secret:
For those just starting out, Baker’s Secret offers a beginner’s guide to baking from scratch. This guide can help you understand the essential tools and bakeware you’ll need and offers some great tips for getting started. Check out their guide here: The Beginner’s Guide to Baking from Scratch.
- The Kitchn:
If you’re looking for a broader range of options, The Kitchn has a fantastic article detailing the best places to shop for bakeware. They cover everything from budget-friendly options to high-end bakeware, ensuring there’s something for everyone. You can read their recommendations here: The Best Places to Shop for Bakeware.
These resources should help you get set up with the right tools to embark on your baking journey! Happy baking!
Graph RAG Response
The aggregated insights from community summaries indicate a predominant focus on ingredient interactions, flavor profiles, and emotional themes in baking discussions, rather than specific baking tools or equipment. While notable culinary figures are referenced, the emphasis is on their expertise rather than the tools they use. A few mentions of essential items, like a Digital Scale for measuring brown sugar, highlight the importance of precision, but overall, the discussions prioritize creativity and culinary practices. For comprehensive information on baking tools, specialized resources or cookbooks are recommended, as the current summaries lack detailed references to equipment.
Key Baking Tips
The fifth question was: “What are some tips or tricks for successful baking that are shared in the documents?” Here, ChatGPT provided a solid list of tips. RAG’s response was weak, merely listing websites with more details. If a document had been labeled “tips for successful baking,” RAG likely would have performed better.
Graph RAG’s answer was the best, even better than ChatGPT’s, including valuable tips like “Quality ingredients” and “practice regularly.”
ChatGPT Response
Successful baking often hinges on attention to detail and a few key principles. Here are some common tips and tricks shared in baking guides and documents:
- Precision in Measurements: Baking is a science, so precise measurements are crucial. Use measuring cups and spoons for dry ingredients and liquid measuring cups for liquids. Weighing ingredients with a kitchen scale can provide even more accuracy.
- Room Temperature Ingredients: Many recipes call for room temperature ingredients, such as butter, eggs, and milk. This helps in better mixing and creates a smoother batter or dough, leading to more consistent textures in the final product.
- Proper Mixing Techniques: Overmixing or undermixing can affect the texture of baked goods. Follow recipe instructions carefully, whether it’s mixing until just combined or creaming butter and sugar for a specific amount of time.
- Preheating the Oven: Always preheat the oven to the specified temperature before baking. This ensures that your baked goods cook evenly and rise properly.
- Understanding Your Oven: Every oven is different. Use an oven thermometer to check the accuracy of your oven’s temperature, and be aware of hot spots by rotating pans if necessary.
- Baking Pans and Linings: Use the correct size and type of pan as specified in the recipe. Greasing and lining pans with parchment paper can prevent sticking and make for easier removal of baked goods.
- Measuring Flour Correctly: Spoon flour into your measuring cup and level it off with a knife. Scooping directly from the bag can pack the flour, leading to excess flour and dense baked goods.
- Chilling Dough: For cookies and pie crusts, chilling the dough can improve texture and prevent spreading during baking. It also helps in shaping and handling the dough.
- Testing for Doneness: Use a toothpick or cake tester to check for doneness. Insert it into the center of the baked good; it should come out clean or with a few moist crumbs, depending on the recipe.
- Cooling Properly: Let baked goods cool in the pan for a few minutes, then transfer them to a wire rack to cool completely. This prevents overbaking and helps maintain the desired texture.
These tips are often found in recipes and baking guides to help achieve the best results.
RAG Response
For successful baking tips, here are some fantastic resources I found:
- Serious Eats Baking Guides: This website has a comprehensive collection of baking guides that cover everything from making great pizza in an outdoor oven to understanding the science behind vegan meringues. You can explore tips on how to properly ferment sourdough, the types of flour you should use, and even the pros and cons of silicone baking mats. The guides are designed to help you perfect both sweet and savory dishes. Check it out here: Serious Eats Baking Guides.
- Baker’s Secret – Grandma’s Tips to Successful Baking: This blog offers time-tested tips and tricks from grandmothers who have mastered the art of baking. While the full details are not listed in the transcript, the blog likely includes practical advice that has been passed down through generations, ensuring your baked goods turn out just right. You can explore it here: Baker’s Secret – Grandma’s Tips to Successful Baking.
- Tasty Baking Recipes: Tasty’s website offers a plethora of baking recipes that come with easy-to-follow instructions and helpful tips. From cookies to cakes, this site covers a wide range of baking delights. The recipes are often accompanied by videos, which can be incredibly helpful for visual learners. Visit Tasty’s baking section here: Tasty Baking Recipes.
Each of these sources provides valuable insights and practical advice that can help elevate your baking skills. Happy baking!
Graph RAG Response
To improve your baking skills and achieve delightful results, consider these key tips:
- Quality Ingredients: Use fresh, high-quality ingredients, including flavorings like vanilla and almond extracts.
- Accurate Measurements: Invest in a digital scale for precise ingredient measurements, especially for dense items like brown sugar.
- Room Temperature Ingredients: Bring butter, eggs, and milk to room temperature for better mixing and texture.
- Preheat Oven: Always preheat your oven for even baking and proper leavening.
- Avoid Overmixing: Mix wet and dry ingredients just until combined to prevent toughness.
- Monitor Baking Times: Use the toothpick test to check for doneness, as baking times can vary.
- Experiment: Try different ingredient combinations and baking methods for unique flavors.
- Adjust Liquids: Modify liquid ingredients when using almond flour to avoid dryness.
- Chill Dough: Chill cookie dough to maintain shape and improve texture.
- Engage with Community: Share your baking experiences and learn from others.
- Practice Regularly: Consistent practice enhances skills and baking results.
By applying these strategies and exploring new techniques, you can elevate your baking and enjoy creating delicious treats.
At what cost?
While I mentioned cost and performance issues earlier, I avoided delving into them until exploring accuracy. The performance is poor—about 30 minutes per query, excluding preprocessing, which took over a dozen hours overnight. This is based on a large dataset of 20,000 baking documents, aggregated into 1,000 community summaries. That’s 1,000 calls, each incurring a cost. Fortunately, gpt-4o-mini significantly reduced the price, making these 1,000 queries cost less than ten cents.
I could improve speed by making the process asynchronous, allowing multiple LLM calls simultaneously. Caching might also help. Global Graph RAG must examine all nodes, but I could prune the data tree to remove unrelated nodes. An algorithm could select the most relevant node and the five most tightly connected nodes, processing only those. Even with these improvements, global RAG is likely to remain slow, measured in minutes rather than seconds.
That said, the potential to extract general data from the entire dataset is revolutionary. It could help us understand trends and connections we don’t currently grasp.
Conclusion: Already interesting for some tasks. It will get even better
I tried five queries. For two of them, the Global Graph RAG was clearly superior. For one, it was about the same, and for two, it did not perform well. Because the Global Graph RAG is so slow, it’s only worth using on queries where superior results are a must, that are general and global enough to best use a Global Graph Rag and where time is not a factor.
I suspect that this post won’t age well and that within months significant optimizations will have been figured out to reduce the time taken for a Global Graph RAG on larger dataset. Certainly we should see an optimizer that automatically chooses to use Global Graph RAG only when it is the best option. I’m looking forward to seeing changes that make it even more useful.
Global Graph RAG will unlock a world of possibilities as it unlocks the ability to get overall data from our dataset, instead of normal searches or RAG that allow us only to look at a portion of it. Global Graph RAG is in its infancy, but those who master it early are likely to reap outsized rewards.
Leave a Reply