Knowledge Graph (KG) and its application in Natural Language Processing (NLP)

What is a Knowledge Graph?

A Knowledge Graph (KG) is a structured representation of knowledge that represents things in the real world and the relationships between things in a graph structure. Each node represents an entity (e.g., a person, place, object, or concept) and the edges represent the relationships between these entities. The main advantage of knowledge graphs is their ability to enable machines to understand complex, multi-relational data and to provide clear information in a structured way.

Most studies implicitly adopt a broad definition of KGs, where they are understood as “a graph of data intended to accumulate and convey knowledge of the real world, whose nodes represent entities of interest and whose edges represent relations between these entities” (Hogan et al., 2022).

KG in NLP

KG application taxonomy in NLP

In the past decade, the application of knowledge graphs in the field of Natural Language Processing (NLP) has primarily focused on two directions:

  1. Knowledge acquisition: Knowledge acquisition contains NLP tasks to
    • construct KGs from unstructured text (knowledge graph construction)
      • knowledge extraction, which is used to populate KGs with entities, relations, or attributes
      • knowledge integration, which is used to update KGs
    • conduct reasoning over already constructed KGs (knowledge graph reasoning)
  2. Knowledge application：common NLP tasks, which are enhanced through structured knowledge from KGs.

Knowledge Graph Construction

To build knowledge graphs, several tasks are involved. The first task is entity extraction, which involves identifying and extracting real-world entities from unstructured text. This helps to identify the key entities mentioned in the text (such as people, places, or objects) (Al-Moslmi et al., 2020).

Once the entities are extracted, the next task is relation extraction. This involves identifying and understanding the relationships and interactions between the extracted entities. By analyzing the text, the connections between entities can be identified, such as “person A works for company B” or “product X is part of category Y” (Zhang et al., 2019a).

In the construction of knowledge graphs, entity linking is another important task. Entity linking involves connecting the entities mentioned in the text to the existing entities in the knowledge graphs. This helps to establish the relationships between entities across different texts or languages (Moon et al., 2018; Wu et al., 2020).

To avoid redundancy and repetition in future tasks, entity alignment can be performed. This involves aligning entities that are synonymous or similar but exist in different knowledge graphs or different languages. By aligning these entities, we can ensure consistency and eliminate duplication (Gangemi et al., 2016; Chen et al., 2018).

Lastly, the structure and format of the knowledge presented in the knowledge graphs are defined through ontology construction. This involves defining the rules and schemes of the knowledge graph, which determine how the information is organized and represented (Haussmann et al., 2019).

In summary, knowledge graph construction involves tasks such as entity extraction, relation extraction, entity linking, entity alignment, and ontology construction. These tasks collectively contribute to the creation of comprehensive and structured knowledge graphs from unstructured text and existing knowledge sources.

Example of ontology construction

An example of ontology construction can illustrate how it involves defining and designing the structure and format of a knowledge graph.

Let’s consider the scenario of constructing a knowledge graph about animals. During ontology construction, we need to determine how to organize and represent knowledge related to animals.

Firstly, we define a top-level concept, such as “Animal,” as the root node of the entire knowledge graph. Then, we can define subconcepts related to animals, such as “Mammal,” “Bird,” “Reptile,” and so on.

Next, we can define attributes and relationships for each subconcept. For example, for the “Mammal” subconcept, we can define attributes like “Weight,” “Diet,” “Mammal Reproduction,” and so on. We can also define relationships between concepts, such as the relationship between “Mammal” and “Fur” or between “Bird” and “Flight.”

Furthermore, we can establish constraints and rules to ensure the consistency and integrity of the knowledge graph. For instance, we can specify that an animal cannot belong to both the “Mammal” and “Bird” subconcepts simultaneously because they possess mutually exclusive attributes like mammalian reproduction and flight capability.

Through ontology construction, we create a hierarchical and interconnected knowledge graph that describes the characteristics, attributes, and relationships of animals. This enables computer systems to better understand and reason about animal-related knowledge, supporting various applications such as animal classification, animal behavior analysis, ecosystem simulation, and more.

In conclusion, ontology construction involves defining concepts, attributes, relationships, and rules to provide structure and format to a knowledge graph, facilitating the organization and representation of knowledge within a specific domain.

Knowledge Graph Reasoning

Once a knowledge graph (KG) is constructed and contains structured world knowledge, it can be used to derive new knowledge through reasoning. In this context, there are two main tasks: entity classification and link prediction.

Entity classification involves classifying entities within the knowledge graph into different categories or classes. It helps assign labels or categories to the entities based on their characteristics or attributes.

On the other hand, link prediction aims to infer missing links or relationships between entities in the existing knowledge graph. It involves predicting potential connections or links between entities based on the available information in the graph. This is often done by ranking entities as potential answers to queries.

To support these reasoning tasks, knowledge graph embedding techniques are used. These techniques create dense vector representations of the knowledge graph, enabling it to be used for downstream machine learning tasks. In some cases, the embeddings are learned jointly with text data, combining textual information with the graph structure to enhance the representation.

Overall, knowledge graph reasoning involves utilizing the structured knowledge in the graph to perform tasks such as entity classification, link prediction, and knowledge graph embedding. These techniques help uncover hidden relationships, make predictions, and enable the application of machine learning algorithms to leverage the knowledge graph for various purposes.

Knowledge Embedding Technique Example: TransE

One common technique for Knowledge Graph Embedding is TransE. TransE is a method used to represent entities and relationships in a knowledge graph as low-dimensional vectors.

To illustrate how TransE works, let’s consider a simple example. Suppose we have a knowledge graph that includes entities like “cat” and “dog” and a relationship “belongs to”. We want to represent these entities and the relationship as vectors, capturing the semantic relationships between them in a vector space.

In TransE, each entity and relationship is mapped to a fixed-dimensional vector. For our example, we can represent “cat” as a vector, v_cat, “dog” as v_dog, and the “belongs to” relationship as v_belongs_to.

The key idea of TransE is to capture relationships through vector operations. For a correct triple (“cat”, “belongs to”, “animal”), we expect that the result of v_cat + v_belongs_to in the vector space is close to v_animal, where v_animal represents the vector for “animal”.

During the training process, TransE optimizes the vector representations so that correct triples are closer to each other in the vector space, while incorrect triples are farther apart. This allows us to predict missing entities or relationships in an unknown triple using vector operations.

TransE is just one of many Knowledge Graph Embedding techniques. Other commonly used methods include TransH, TransR, DistMult, and more. These techniques aim to learn vector representations that capture the semantic relationships within the knowledge graph, supporting various tasks such as relation prediction, entity classification, question answering, and more.

In summary, Knowledge Graph Embedding techniques, such as TransE, enable the representation of entities and relationships in a knowledge graph as low-dimensional vectors, capturing their semantic relationships in a vector space. This provides a powerful foundation for applications and analysis of knowledge graphs, helping us understand and utilize the knowledge within them.

Reference: Bordes, A., Usunier, N., Garcia-Duran, A., Weston, J., & Yakhnenko, O. (2013). Translating embeddings for modeling multi-relational data. In Advances in Neural Information Processing Systems (pp. 2787-2795).

Knowledge Application

Knowledge graphs (KGs) have various applications in popular Natural Language Processing (NLP) tasks. Let’s explore some of the most common ones.

Question answering (QA) is a widely used NLP task that leverages KGs. It can be divided into two types: textual QA and question answering over knowledge bases (KBQA). Textual QA extracts answers from unstructured documents, while KBQA retrieves answers from predefined knowledge bases. KGs are naturally connected to KBQA, but they can also be used as a source of common-sense knowledge for answering questions in textual QA tasks. This approach not only helps generate answers but also makes them more understandable.

Semantic search goes beyond literal matches and focuses on understanding search intent and query context. KGs are utilized in this area to enhance search, recommendations, and analytics. For example, ConceptNet is a large semantic network that captures everyday concepts, while the Microsoft Academic Graph provides a KG of scholarly communications and relationships.

Conversational interfaces benefit from the world knowledge stored in KGs. Knowledge from KGs can be employed to generate informative and contextually appropriate responses for conversational agents. Researchers have explored knowledge-aware dialogue generation to improve the quality of conversational interactions.

Natural language generation (NLG) is a subfield of NLP that involves generating natural language output. KGs are used in NLG to generate natural language text from KGs, create question-answer pairs, perform image captioning, and assist in data augmentation in low-resource settings.

Text analysis encompasses various NLP techniques applied to process and understand textual data. It includes tasks like sentiment detection, topic modeling, and word sense disambiguation, where KGs can contribute to improving the accuracy and quality of analysis.

Augmented language models combine large pretrained language models (PLMs) such as BERT and GPT with the knowledge stored in KGs. By integrating KGs into PLMs, structured knowledge can be utilized during language model training and inference, enhancing their understanding and generation capabilities. Examples of such models include ERNIE, COMET, K-BERT, and KEPLER.

In summary, KGs find applications in question answering, semantic search, conversational interfaces, natural language generation, text analysis, and augmented language models. They enrich NLP tasks by incorporating structured knowledge and improving the performance and interpretability of NLP systems.

Reference

Schneider, P., Schopf, T., Vladika, J., Galkin, M., Simperl, E., & Matthes, F. (2022). A Decade of Knowledge Graphs in Natural Language Processing: A Survey.