Knowledge Engineering Teams, Tools and Phases of Work

The knowledge engineering process requires specific tools and phases of work.

• Team roles in the knowledge engineering process
• Knowledge engineering tools
• The work phases required to create a working expert system knowledge base

The Team Roles in the Knowledge Engineering Process

It takes a lot of humanity to make machines intelligent. At least, that’s true for expert systems.

An expert system with an empty knowledge base is actually quite dumb. It depends completely on people to collect its knowledge, put it in the right form, load it up and then look after its ongoing care and feeding.

This rough list describes some specialized team roles  who make it happen.

1. Subject matter expert:

The ‘SME’ shares his or her expert knowledge in the domain addressed by the expert system. You need to find a really good expert and upload their mind into the system.

2. Knowledge engineer:

This person does the things described here. Essentially, the knowledge engineer facilitates the acquisition of expert knowledge from the subject matter expert in a form that can be used by non-experts through the expert system.

3. Knowledge content specialist:

This team member collects expert knowledge as it comes spilling out of the SME. A good knowledge creation specialist won’t waste a single drop of it. The knowledge must be collected and packaged in the right format to eventually be used by the expert system.

4. Knowledge refinement specialist:

Before expert knowledge is loaded into the system, it has to be refined for everything from style, grammar or punctuation to the exact layout. It may need special tags, file names and other identifiers in hundreds or thousands of places.

5. Knowledge approver:

In domains like law, it’s critical for every bit of expert knowledge to be reviewed and approved for things like legal accuracy before it’s loaded into the system. Omissions can be just as important as errors if the expert system is built for non-expert users.

6. Knowledge loader: 

Once the knowledge is ready and approved, this team member has to put it into the system. We call this an ETL or extract – transform – load process. First, we extract all the useful data from the knowledge content. Next, transform or convert it into the format the expert system can use. Finally, it’s loaded into the system.

7. Quality assurance testing lead:

When the knowledge is first loaded into the knowledge base, it has to be tested before anything else happens. This team member coordinates a group of testers (usually project ‘insiders’) to do this first round of review. Any initial bugs or errors go back to the loader to be corrected.

8. User testing lead: 

Once early testing is complete, the circle of testers will gradually expand to include more and more testing with people who are less and less familiar with the project or the knowledge domain. The user testing lead collects and compiles feedback from this testing and presents it to the team who will identify necessary changes and improvements to the knowledge base.

9. Continuous improvement specialist:

You can’t just load the knowledge into the system and consider it “done”.  A well designed system will have ongoing sources of user feedback to support this process. Analytics and other data collected from use of the system must also be reviewed and analyzed.  The goal is to keep improving the system over time. The more it gets used, the smarter it gets.

10. Knowledge base curator:

Have you ever been sure you had an email with some important information in it but just couldn’t seem to find it when you need it? Or how about that thing where you’re editing a document with a couple of other people and completely lose track of the latest version with the best changes? Expert system knowledge bases can contain tens or hundreds of thousands of data objects. What they don’t need are version control problems and confusion. What we do need is a knowledge base curator to keep it all straight.

11. Subject matter expert networks:

Want to keep your knowledge base from going stale? Keep it fresh by creating networks of subject matter experts who periodically report on new developments or insights in the domain (e.g. new legislation and court decisions in the legal domain). These experts can also continually provide new knowledge that can be engineered and loaded into the database to make it deeper or wider.

Smart machines start with smart teams

I’ve never actually worked on a knowledge engineering project where each one of these roles was held by a different person. In many cases, people will look after multiple roles. But the tasks are all quite important, and the demands can be surprisingly high.

Want to create a smart machine? It helps to have a good team.

 

Knowledge Engineering Tools

This section introduces some of the tools a knowledge engineering team can use to create the logic for an expert system database.

Digital vs analog

Like many forms of facilitation, it’s possible to use post-it notes and large sheets of paper or walls, windows, etc. to create and capture the knowledge content.

These tools can be a great way to let people share the same brain space. But I’ve found they can be a bit slow. A skilled typist can enter a lot of information much faster than someone writing with a sharpie on a post-it note. The typist’s work will also be easier to read and edit.

Paper-based approaches also allow the team to create content that isn’t logically connected in the same way decision trees must be. This can actually be a bit of a drawback considering that an expert system will require logical connections between rules (fuzzy-ness has to actually be built-in, in the form of deductive logic). For these reasons, I’ve found decision tree software to be a good choice for capturing content.

Decision tree software

An expert system’s logic-based rules essentially take the shape of decision trees. Decision trees are capable of capturing and creating a large volume of rules that branch through the domain.

Many mind mapping software applications offer a decision tree structure. In my experience a left-to-right decision tree structure works well.

In live facilitation sessions, the team will have to work together on specific bits of logic. For reference, it can be helpful to refer to nodes in the decision tree as ‘objects’ or ‘an object’. Using an object as a reference point, the object that is connected to it to the right can be called a ‘child’ of that object. Another child of that object can be called a ‘sibling’ of the first child.

Word processors

Any longer text-based content can gradually be moved out of rough notes from decision tree software and into a common word processing file format.

Shared drives & cloud storage

As the decision trees grow, they become intricate and complicated. It’s very difficult to track version control. To avoid creation of conflict copies (e.g. two people were editing the decision tree independently, and caused version control saving errors) the team should use a shared drive or cloud storage solution with decent file conflict features.

I’ve found it helpful to remind the team to always save and upload any local copies of their work to the shared location as soon as they can. If it’s possible that more than one person needs to use the same decision tree at the same time, encourage them to agree on how to deal with version control issues manually.

Audio & desktop sharing 

Busy subject matter experts often have demanding jobs and tight schedules. To save travel time and to make every minute count, I’ve found it helpful to enable remotely conducted live facilitation sessions.

Remote sessions must be supported by audio conferencing software so everyone can talk to each other. They must also be supported by desktop sharing applications so that subject matter experts and the knowledge engineering team can all see the decision tree where the content is being recorded or revised.

The Work Phases Required to Create a Working Expert System Knowledge Base

This section covers the work phases required to create and deploy a working expert system knowledge base.

Pre-work

We’ve already covered a lot of the steps that happen before knowledge engineering actually starts. They include things like orientation, scoping and prototyping.

Raw content creation

This is the big part! Knowledge engineers work with subject matter experts to capture expert domain knowledge and structure it in the right form for the expert system’s knowledge base.

Subject matter expert validation

After the expert knowledge has been captured and structured in the right form, the knowledge engineering team reviews it with subject matter experts to validate it.

In reviewing the content, knowledge engineers will be asking questions like:

• Is this right?
• Does this structure make sense?
• Is there a better way to do this?
• Will users understand it?

Once the subject matter experts have validated the knowledge base content, it can go to the next stage.

Refinement

A specialist needs to refine all the knowledge base content and refine it for style, grammar, punctuation, layout and overall structure.

Review & approval

A subject matter expert – ideally someone other than the subject matter experts used in the content creation process – needs to review all the knowledge base content for errors, omissions or other inaccuracies.

If you’re knowledge engineering content for legal expert systems like I do, you’ll want this person to be a lawyer with strong expertise in the area.

ETL (extract, transform & load)

This stage is a little bit like programming the system. A specialist takes the content and extracts the rules the expert system needs, transforms it into a language and format the system can understand and loads it into the knowledge base.

Systems that include a “pathway builder” right in the platform may be able to skip this step.

QA (quality assurance) testing 

Knowledge content that has been loaded into the system must go through some initial quality assurance testing. Subject matter experts and other knowledge engineering team members can run through the knowledge to make sure it flows properly, displays correctly, and is free from obvious errors like typos and formatting.

User testing

Knowledge base content can pass through several stages of user testing. I recommend an “expanding circle” approach:

1. Start with a small, highly controlled group of users
2. Incorporate the first round of feedback into knowledge base improvements
3. Expand the circle of testers and test again
4. Incorporate this second round of feedback into knowledge base improvements
5. Repeat steps #3 and #4 as required
6. Closed beta testing: invite real users to test the system, but in a controlled environment. Rather than make your expert system freely available online, keep its location concealed, and only point the users to it via email or another direct means
7. Incorporate closed beta testing into knowledge base improvements
8. Open beta testing: make the expert system freely available online, without controlling direct access for users
9. Incorporate open beta feedback into knowledge base improvements

By the time knowledge base has passed through the successive rounds of user testing, it will contain a significant volume of improvements and validation, all based on feedback from users.

It should be ready to launch!

Deployment

The expert system knowledge base can now go into “production” or “ship”. It’s ready to help real people with real problems in the real world.

Continuous improvement

Opening up your content to your users doesn’t mean you’re done; it’s just the start of the continuous improvement process.

Knowledge engineers and content experts can review data collected from the system – both analytics and qualitative user feedback – to identify improvements. These improvements are created, reviewed and loaded into the system.

Knowledge base curation

Someone should be in control of, and responsible for, all the expert content in an expert system’s knowledge base. This person responds to problems, whether it’s with the platform or with content, and has the final say on all things relating to curation, including things like version control.

Subject matter expert network

When the initial knowledge base creation and loading work is complete, subject matter experts will no longer be engaged in the intensive facilitated knowledge engineering work. But they should stay involved.

Knowledge engineers can create networks of subject matter experts to help with problems and to recommend changes based on new developments, discoveries, changes in the law and so on.

A subject matter expert network may involve a regularly scheduled meeting with subject matter experts every 6 months. The agenda might look like this:

1. Review any errors, problems or weak points identified by users in the last 6 months
2. Identify new developments or changes in this domain in the last 6 months
3. If there is no significant work relating to items #1 or #2, then review a portion of the knowledge base to identify potential improvements