What is Evidence?

Evidence is the collection of documents and data used by the models when generating answers to your natural language query. When a user asks a question in Axon, the knowledge graph populates answers to this query as individual nodes in the knowledge graph. Each node on your knowledge graph can be considered an answer to your query, and the evidence for a given answer is provided through an Evidence View.

The Evidence View displays all unstructured documents used by the models as evidence for a given answer. For additional context around an answer, any structured data about a given answer is also provided, so users can quickly reference additional properties for a given entity (see below for more information).

Evidence View Breakdown

Here is an example Evidence View for the answer “fever”, which is an answer generated for the query “What are symptoms of covid-19?”.

From top to bottom, the Evidence View displays:

1. The question asked in the Axon search bar (“What are symptoms of covid-19?”).
2. The total number of unique answers that Axon generated based on the data in your data fabric.
3. The answer generated by the model, which is be the same as the node found in your knowledge graph.
4. The answer’s supporting evidence, including the document it came from, as well as the paragraph where the answer was derived.**Note: In instances where an answer can have several pieces of evidence, the Evidence View shows one document, and all other supporting evidence is displayed in the “Show More” section underneath. In this scenario, you can see the answer “fever” has 73 additional supporting pieces of evidence.

Supporting Evidence: NER & Answer Highlighting

For each document defined as supporting evidence, you will see several additional layers of metadata and deep learning applied. Here’s a breakdown of how to interpret highlighting and additional metadata.

Here is an example evidence section for “hemicentin-1”, an answer to the question “What genes play a role in macular degeneration?”.

From top to bottom, here’s a breakdown of what information is available for an answer’s supporting evidence.

1. The answer (“hemicentin-1”) is displayed at the top, and shown in the knowledge graph as the node.
2. The probability scores (Overall Score and Individual Scores) for that answer are displayed.
3. The Layar instance where that document came from is displayed (e.g. “User Data”, grey)
4. The official title of the first supporting document, with a hyperlink to the original document.
5. The paragraph or text snippet where the deep learning algorithms found supporting evidence for an answer. The answer is highlighted in yellow, and any autodetected NER tags are highlighted throughout the text snippet.
6. If there are additional documents that support the same answer, you will see a “Show More” section, where you can view all remaining evidence for an answer. In this example, there was one more document that supported the answer).

Graph Properties & Structured Database Results

If an answer in your knowledge graph also matches an entry in your connected relational graphs or structured databases, they will be visible in the top right of the Evidence Section.

For example, the answer “ropinirole” matches a node in my Neo4J connector. I can also add any related nodes from that graph database directly into my knowledge graph by clicking the “Add to Graph” button in that section.

For additional inquiries about the Evidence View, please reach out to us at [email protected] for further assistance.

The Evidence View provides all of the data and used in Vyasa’s deep learning algorithms to answer your natural language query. Here’s how you can view what supporting documents are the evidence for a question or specific answer.

Viewing the Evidence Tab

1. Click on the node of interest. This node can be either a question node (e.g. “What are the symptoms of COVID-19?”) or an answer node (e.g. “fever”).
2. The Evidence View for that node will be on the right. Question nodes will display all answers, and each answer’s supporting documents. Answer nodes will only display the data supporting that specific answer.

After selecting the filters you are interested in, you may wish to clear all of the filters and review the original knowledge graph. Here’s how:

1. In the top right of your knowledge graph, there is a box with all of the current “Active Filters“.
2. Click the “Clear Filters” button.
3. All filters will be removed from the graph, and no history of previous filter settings is saved. If you wish to re-add filters, go back into your Filter tab and add them again.

If you have asked a series of questions in Axon, it’s likely that your knowledge graph has become cluttered with answers. Search for answers that exist in a specific subset of data providers from your data fabric (PubMed, Clinical Trials, Patents, etc.).

Show Answers That Are Common Between One or More Questions

1. Ask multiple questions in the search bar.
2. Once the answer nodes have populated the knowledge graph, click the Filter tab.
3. Click the box underneath Data Provider.
4. Select the data providers you are interested in. Only nodes where the Evidence for the answers comes from these providers will be displayed.

If you have asked a series of questions in Axon, it’s likely that your knowledge graph has become cluttered with answers. If you would like to narrow down your results to only those defined in an ontology, follow these steps.

1. Ask multiple questions in the search bar.
2. Once the answer nodes have populated the knowledge graph, click the Filter tab.
3. Click the box underneath Ontologies
4. Select the ontology you wish to work with. Only terms that are mentioned in this ontology will appear as answers in the knowledge graph.

If you have asked a series of questions in Axon, it’s likely that your knowledge graph has become cluttered with answers. Search for a specific term of interest with the following steps.

Show Answers That Are Common Between One or More Questions

1. Ask multiple questions in the search bar.
2. Once the answer nodes have populated the knowledge graph, click the Filter tab.
3. Click the box underneath Search Terms.
4. Type in the term(s) you are interested in. Only nodes with those keywords as answers will be displayed.

For starters, let’s get you familiar with some of the terms we are using. NER is an acronym for Named Entity Recognition, which is a deep learning text analytics subdomain where terms and phrases are identified within text and classified into categorical (aka entity) types (proteins, cell lines, diseases, etc.). Read more about the benefits of NER tagging in What Are NER Concepts?

Let’s say you have asked a series of questions about COVID-19, and would like to use the Vyasa NER annotations to see which concepts are most frequently mentioned regularly across multiple answers and their Evidence.

1. Go to “Filters”
2. Under “Show Top Related Concepts”, select “Mentioned in multiple answers”.
3. NER concepts that are mentioned in the Evidence for more than one answer node will become new nodes in the Knowledge Graph.
4. If you want to know which NER concepts are found multiple times in the Evidence for each answer (an indication that it is a primary focus for the article), select “Mentioned multiple times per answer”.
   
   Pro Tip: If you are only interested in a specific subset of NER concepts, such as proteins and disease, you can whittle down your results by going to the Concept filter (see below) and select the few concept types you are interested in.

Why do these make a huge difference in the quality of results?

Although our elastic search tries to find appropriate evidence from query understanding, our transformer-based models are semantics driven, and fed the exact query as written. Tips 1 and 2 are central to getting good results consistently. The remaining tips can help users format questions that increase consistency in answers.

Tip #1. Ask a formal question.

1. Good Example: ‘What is a biotech company in Boston?’
2. Bad Example: ‘Biotech companies in Boston’ or ‘show me biotech companies in Boston’

Tip #2. Address the system as if you were querying a single document and ask for singular examples.

Questions are taken very literally. If you ask the question ‘What are analogs for GLP-1-R?’ you might get answers like ‘long lasting’. But if you ask ‘What is an analog for GLP-1-R?” you get ‘liraglutide’ from the same source paragraph.

‘What are drugs in clinical trials for the treatment of Covid-19?” yields worse results than “What is a drug in clinical trials for the treatment of Covid-19?”

Drugs in clinical trials as a group are novel, and repurposed, and are already being prescribed by doctors.

A drug in clinical trials is hydroxychloroquine.

So, play around with wording if you find yourself getting some snarky answers back.

Tip #3. Avoid Qualifiers or a request for an opinion.

1. Good Example: ‘What is a company that is researching drugs for Covid-19?’
2. Bad Example: ‘What companies are most successful so far in finding effective drugs for Covid-19?’

Tip #4. Ask a single, simple question instead of a compound question.

1. Good Example: ‘What company produces Albuterol?’
2. Bad Example: ‘What is Albuterol and who produces it?’

Tip #5. Avoid time based questions and use the date filter instead.

1. Good Example: ‘What is a company that has moved to Boston?’, filter data for search by date published.
2. Bad Example: ‘What companies have recently moved to Boston?’

Tip #6. Be as specific as possible.

1. Good Example: ‘What is a company that is researching drugs for the treatment of Covid-19?’
2. Bad Example: ‘What companies are working on Covid-19?’

Tip #7. To force the system to require a certain word or phrases to be present in their exact form during elastic search querying, use single or double quotes around the word or phrase.

1. Example: “What is a venture capital company in Boston?”
   Forced Behavior: “What is a ‘venture capital’ company in Boston?”

Users can ask questions in the search bar that will return real time answers to questions from sources within your data fabric. When a question is asked, it will create a node with the question, and answer nodes will appear and connect to it.

1. Type a starter question into the search bar at the top of Axon. It can be broad or specific. Ex: What is a symptom of COVID-19? What is a drug in clinical trials for Erythropoietic Protoporphyria (EPP)?
2. Select the data fabric you want to query. The Default Fabric is a combination of User Uploads and the Life Science Reference Data Fabric.
3. Select the data providers you want to query (for example “Patents”, “Clinical Trials”, and “User Uploads”). You can select more than one data provider for your search. Selecting no data provider means you are searching across the entire data fabric, and are looking at all of the data providers.
4. Ask multiple questions in the system to find overlaps or similarities between answers to different questions.

A knowledge graph application that enables derivation of dynamically generated knowledge graphs directly from integrated data and document sources integrated in a Layar Data Fabric.