HAVE A CONVERSATION WITH AN EXPERT

To start, Conversational AI works well only if the data and the information underneath have good quality. You know, RAG means you’re not using the large language model itself as a source of knowledge and truth, but still your own knowledge base. So we still have the need to integrate data, typically from various sources, and have a kind of a database of linked facts in place, which then can be used to generate really good results and answers if you have a question answering engine, for instance.

To explain why knowledge graphs are so critical for all this, we can break graphs down into three parts.

First, a knowledge graph is like this big consolidated view of all the data and content you have. It’s there to link all your data and business objects together and act as the “brain” of the system.

Secondly, enterprise knowledge graphs typically have a specific knowledge model at their core. And the knowledge model is an element of the entire graph database that provides rich information about your domain. So it’s a domain specific knowledge model which provides additional context to your application and, for instance, can be used to fine tune your prompts. 

The third element of the graph is that it can also be used for user intent classification. It’s very important to understand the user intent when you look at the workflow of a typical RAG architecture. A good RAG application tries to fine tune the output dependent on the user persona or behavior of the user. Even if you’re not logged in with a specific account and are using the platform anonymously, the knowledge model plays a very important role in matching the text that’s entered in the search field – this is the intent aspect – to the concepts of the knowledge model, which is the classification part. 

This is all part of the knowledge graph and the reason why they’re becoming so important in the meantime.

Yes, so ChatGPT processes public content, but at the workplace you’re not asking questions about public content, you ask questions about content stored in your enterprise systems. Is it your CRM, your technical documentation, your knowledge bases across different departments? Content comes in different structures and formats; it’s unstructured, semi-structured, componentized. And then you have a lot of structured data. 

And as an enterprise user, you don’t have enough time to go through all these non-personalized answers; you have the expectation that the system should understand your role and your responsibilities. You ask a question to get some valuable insights into your enterprise data sources and then the system really needs to be spot on. Anything else would not be accepted and the systems won’t get used. So the people would lose trust after a couple of attempts of not finding anything better. 

I think it’s really mandatory to think about personalization strategies when talking about enterprise RAG systems. And here, Graph RAG plays an important role starting also with the query formulation itself. So when you enter the company as a new employee and you get onboarded and you ask a question about the products that your company develops or ships, you likely will ask different questions than the experts who have been with the enterprise for 20+ years. In this case, something like a query assistant which helps you formulate the question can go a long way. Knowledge models and knowledge graphs play an important role because they can, in real time, help you create meaningful queries, even if you’re not fully equipped with the deep knowledge that your senior colleagues already have. You have learning by doing, even when asking questions. 

And then also the follow up questions typically produced by such conversationalized systems may give you insights in how you could go one level deeper in terms of asking more specific questions. All that also is enriched by context and knowledge from the knowledge graph running in the background. So now it’s really kind of a digital helper assistant doing the trick and the magic at each and any step during such a dialogue. Starting from query formulation over query interpretation, retrieving the information from different data sources, enriching the question with background information running in the background, doing prompt fine tuning, and then also finding good ways to present the information in a more personalized way, depending on the role and experience that people have in different companies.

There are some mega trends out there which force companies to finally invest more money into such applications, which is for instance, a shortage of skilled workers. There’s high turnover and it takes a while to onboard new employees to the skillset that’s necessary to complete a task. Conversational AI can really help there in the case of assisted prompt engineering. New employees often don’t even know which questions to ask but the AI assistant can help them formulate the questions with auto suggestions. And then if it’s  a sophisticated chatbot, it will also be able to recommend articles for additional reading and summarized conclusions. 

And then a more general observation is that every company is looking for precision, trustworthiness, and less hallucination. And at the end of the day it’s all about costs. Using LLMs could mean a huge cost explosion, but that’s not necessary if you still use LLMs in a smart way, combined with other technologies like knowledge graphs, so you can pre filter only those pieces of your knowledge repositories which are important to give an answer. That can be done at a very low cost using semantic classification, tagging, traditional machine learning, etc. This filter goes to the LLM, and by that you reduce the number of tokens tremendously, and therefore the associated costs. 

Last but not least, customers are looking for explainability and no black box approaches. They’re looking for governance models for the AI systems. AI governance is extremely important because it helps to mitigate AI risks and consequently reputational risks, litigation risks and other quite big risks attached to the usage of AI without a strategic planning and a governance model around it.

So when an LLM produces incoherent or factually incorrect or “half-baked” answers, this is a hallunication. The thing is that probabilistic models from time to time just come up with crazy facts and answers. So there’s no way to fully stop hallucination really. 

With LLMs, either you have low precision and less hallucination, or you have higher precision and higher hallucination. In sharp contrast to that, knowledge graphs provide an explicitly available set of facts and information that you can really look at. You can just retrieve from sources wherever a certain governance process and quality management process has been attached to. So you may be able to assume all that we have written in the knowledge graph is at least authorized and somebody is at the end of the day accountable if the information is wrong. So we call that a trust system. 

Now with the LLMs in the mix, we can no longer assume it’s true. We don’t know who we should make accountable for the information it’s producing, so it’s a bit of a dilemma. On the other hand, we would like to have these types of digital helpers in the workflows to accelerate the speed of discovery and then the speed of producing new content, etc. So we have to be very careful with integrating LLMs into these systems and the only way we can do that is to set up guardrails. 

Those guardrails can be knowledge graphs, because you can set up a set of facts and information that you don’t necessarily need to include in any answer, but the background information can ensure that the answer is factually correct even if it’s not explicitly mentioned. You can “inject” the right information and force the LLM into the right direction so that you can guarantee a minimum of quality. Relevancy is really a keyword here: LLMs are not very good when it comes to filtering out the most important pieces of all the data you have in your enterprise. So filtering out can be nicely done with a knowledge graph. And by that you already again reduce hallucination.

I think Graph RAG has definitely a huge potential to become the de facto standard to deliver guardrails for AI applications, fulfilling all these criteria like trustworthiness, etc. LLM + Graphs is a big trend I’m observing and we have proof with this based on a LinkedIn group that we own. It’s been up and running for around 4 years now, I think, and it just steadily grew over the last few years, but then recently when Graph RAG came, the member count skyrocketed really like in a way I’ve never seen before. 

I think this shows that knowledge graph technologies are definitely no longer a niche, it’s really taking up now. It’s still not completely sorted out which “flavor” of Graph RAG should be used under which circumstances –  which data is needed and which graph structure and knowledge models and use cases –  so at the moment, there’s lots of experiments taking  place where people are trying to bring all the elements together. I think there will be some methodologies emerging in the next few years for specific use cases where it will become more and more clear on how to build such applications for specific use cases in particular sectors and industries. And only then, I would say the next wave of specialised companies and integrators and consultancy firms will come out. 

By expanding on already used knowledge models and knowledge graphs, you can rather quickly set up a new application successfully. I think graphs will be a tool to accelerate this way to produce AI applications serving real and specific business needs. So it’s a beautiful combination now, as actually we already proposed many years ago. At that time we called it Semantic AI: the merging of machine learning and knowledge graphs. In the meantime, we have Graph RAG  using LLMs and graphs together, maybe still with some other machine learning methodologies in place. So finally, the symbolic and the statistical AI communities are coming together, right?

Still, a lot of human expertise will be needed. The human in the loop. The subject matter experts, the domain experts are still the main drivers of such applications. They need to make sure the quality is good and work on iterations of the model. At the end of the day, there needs to be an organisation or person who feels responsible and accountable for the application. I think here we will see a lot of regulation and some lawsuits soon. If organizations are not following these principles, they will drop out; people will lose trust in their services. And as I’ve said before, trust is the most important currency in the 21st century.

When thinking of Graph RAG, it’s like two gears or wheels kind of coming together.

So the one is where the graph is accelerating, at least enhancing the quality and providing better context etc. to the LLMs. In other words, it really drives the usefulness of an LLM. So this is the one wheel, and then the other one is where the LLM is driving the creation of knowledge models in a way that it can lower costs and increase quality and scope and stuff like that. And these two gears have to work together and direct each other to keep the machine going.

In both aspects, users play an important role. In one wheel [the LLM], the end users, by using the system, will vote on the usefulness of results that are delivered so that the systems will learn. The other wheel [knowledge graph] has the subject matter expert who defines the model and states what is correct in the first place. These are the drivers, and the human should have the most influence overall. You can’t have a good Conversational AI application – whose whole purpose is to serve the human using it, by the way – without this human element.

We want to create knowledge models from the subject matter experts who understand the needs of the users and what they can do better with the delivered information. Using this same wheel or gear metaphor, I would also like to point to the Taxonomy Advisor, which is driving the development of the knowledge models on the “LLM gear,” where subject matter experts working on the domain models get some help from an LLM to create their taxonomies. They get suggestions of additional concepts that should be included in the model, how they should be included, how they could be named, and which synonyms and alternative labels to add. These suggestions are crucial to enhance the precision of the entire system, providing additional context to the information and therefore output of the Conversational AI application – this is the knowledge graph gear.

There’s a lot of ways to use LLMs to improve and accelerate the development of knowledge models. The Taxonomy Advisor is definitely one of the leaders in the field; what would have probably taken you a couple of hours to build, you can now build as a subject matter expert in less than 15 minutes. It’s up to ten times faster with the same level of quality. And this is really a key metric, I would say, that will raise the appetite of those who are not familiar with knowledge models and knowledge graphs to start using them. Because there was always this counter argument, “Oh yeah, if we only had a knowledge graph, then probably we would start using it, but we don’t know how to build it. It’s too expensive, too complicated, sophisticated, it’s an academic thing.” Now with the Taxonomy Advisor, this is not a big business argument. You can just accelerate the potential, lower the cost, and still have good quality in place to make use of it in your Graph RAG system later on. 

I think it’s important to make clear that there are many companies currently working on RAG. Majority have not initially come from the graph side; they try their best to produce good RAG quality with vector databases etc. Of course, given all we’ve talked about here I think it’s extremely important to make use of the graph component. So if somebody wants to do the next steps, they should really look into our offerings. 

They can get started with the GenAI Starter Kit to quickly run proof of concepts – which we have managed to do in recent months with large companies very successfully. We also have the GenAI Product Bundle which has all the elements you need to kick off a PoolParty project with us. And if people want to see how a Graph RAG approach culminates in a proper Conversational AI application and actually try it out, they can find that in knowledge-hub.eco as an example or check out our eBook on the knowledge-hub.eco to see it’s special features.

In any case, I think it’s good to end this interview on the note that now is the time to work on Conversational AI applications. Companies likely already have enough capacities to create them either for internal or external purposes, so it’s just about linking all these things together in an architecture that suits their need. It’s just important that they use something they trust and that the subject matter experts are included; that knowledge cannot be outsourced.