Initiating the Content Creation Process

Creating an expert system’s knowledge base is the heart of knowledge engineering. This description covers 3 of the steps to begin the process.

Early steps include:

• Orientation
• Domain investigation
• Precision scope determination
• Random content generation
• Proto-branch creation
• Knowledge engineering project management

Orientation

Knowledge engineers require orientation to the processes and methodologies that will be used to create the knowledge base for the particular system under construction.

Subject matter experts also require orientation to the idea of expert systems, to the knowledge engineering process and to their role in the creation of the knowledge base.

A presentation covering the various activities and roles, along with sample expert systems in operation are all suitable for orientation purposes.

Domain Investigation

Some early investigation of the domain will improve the actual content creation process.

User experience activities like surveys, focus groups or observations carried out in experts’ offices or call centres will improve the knowledge engineer’s understanding of the way non-experts and experts interact in the domain.

Research activities can further improve the knowledge engineer’s familiarity with the domain. This research should remain relatively light, and can include a general survey of mainstream literature or other sources of recorded knowledge.

During the survey work, the knowledge engineer can create a list of existing resources in case they are needed later in the knowledge engineering process.

The aim of this investigation is not to turn knowledge engineers into experts or reduce the reliance on subject matter experts. It’s more of a process of familiarization to improve the knowledge engineering effort downstream.

Precision scope determination

In this stage, knowledge engineers and subject matter experts work to develop a very precise working definition of the scope of the knowledge base. Anything declared to be “in scope” must be knowledge engineered. Anything “out of scope” must not be knowledge engineered.

This definition of scoping may sound simple. But it can be difficult in practice. Domain knowledge is intangible. It also tends to be un-ordered. It flows from many sources in many directions. During the knowledge engineering process, it will have to be converted into very concrete, ordered, linear logic-based rules.

The working definition of the knowledge base’s scope will determine what does and doesn’t need to be converted into the knowledge base’s rules.

Random content generation

Once the scope of the an expert system’s knowledge base is determined, the next step is random generation of knowledge-based content. Random content generation creates sprouts of the branches that will eventually form decision trees containing the logic-based rules for an expert system knowledge base.

Facilitating random content

Random content generation is a facilitated activity. The knowledge engineer starts by encouraging subject matter experts to name common problems within the domain. This process is best if it is done at a fast pace with no attempt to create any rules or other structure.

The knowledge engineer can prompt subject matter experts with questions like:

• “What’s the most common problem you see?”
• “What are the top 3 problems you encounter?
• “Who comes to ask for your help most often, and what problems do they have?”

The problems can be recorded very quickly on a mind map in ‘floating’ idea or object form. For example, auto repair subject matter experts might identify the following during random problem generation:

Grouping problems

After the random content generation phase ends, the problems can be grouped into broad problem types. In the example used above, the grouping might look like this:

The knowledge engineer can experiment with different types of grouping and to group with multiple levels moving from general to specific. An alternative to our example, with an added level of grouping, might look like this:

Random problem generation doesn’t have to go much further than problem grouping.

Benefits for subject matter experts

In addition to providing the sprouts for the decision tree and the early structure for the knowledge-based rules, random problem generation should hold positive benefits for subject matter experts, including:

• affirmation that they are suitable experts for the knowledge engineering process
• an early sense of involvement and engagement with the knowledge engineering process
• early feelings of ‘ownership’ over the content
• an opportunity for individual subject matter experts in a larger group to validate one other’s knowledge and familiarity with the domain (e.g. one SME names problem type X, causing the other 5 SMEs to all point, nod vigorously, and say “I see that problem all the time too!”)

Validation of the Grouping Exercise

After the randomly generated problems are grouped, the grouping can be validated with the subject matter experts.

They should understand and agree with the problem groupings as much as possible. In the event of any competing approaches between overly technical groupings and approaches non-experts would understand, knowledge engineers should endorse the latter.

Random solution generation

I have experimented with random solution or outcome generation, following a similar approach. Although it can be a good exercise for putting subject matter experts in a solution mindset, it can be difficult to marry the solutions up with the right problems if they are also generated randomly.

In addition, there is a risk that subject matter experts will fall into a pattern of trying to organize the map by solutions our outcomes instead of by problem type. This difference can have a negative impact for non-expert users since they’ll have no notion of potential solutions, or may be unable to discern which solution applies to their circumstance. On Non-experts will, on the other hand, know what problem they are having – and will be capable of matching it up with the problem-based structure of the knowledge base.

Depending on the domain, outcome-based problem generation may be suitable. In many cases it will not.

Proto-branch creation

Following random content creation, the knowledge engineering team can move next to the creation of prototype branches through the domain. These branches will form the basis for the eventual creation of a logic-based decision tree through the domain.

Purpose of proto-branches

These proto-branches will serve a few purposes, including:

• laying down the foundational logic that will support the creation of multiple new branches in the decision tree(s) capturing expert knowledge in the domain
• providing an early indication of the knowledge engineering work, which will help with project planning and estimating
• continuing the gradual orientation to knowledge engineering for subject matter experts and the rest of the team

Proto-branches don’t have to reach the stage of full completion. Once the main knowledge engineering work begins, the proto-branches are likely to undergo many significant alterations.

Build on the random content generation work

The content generated from the earlier random content generation step will be a starting point for the proto-branches.

Knowledge engineers should work with subject matter experts to select 1 or 2 problems to prototype.

Identify a starting point

The expert system’s knowledge base will work on the basis of logic-based rules in the form of a decision tree. The proto-branches need to start at the base of the tree.

The starting point should represent the most general diagnosis or categorization of the subject matter in the domain.

Create the branches’ structure and flow

The proto-branches will generally reflect the following pattern (which will be covered in more depth later):

1. Diagnosis: 2-5 rules with multiple options or branches in the decision tree to diagnose the problem or narrow the subject matter in the domain
2. Information resources: as the problem is diagnosed, small, specific bits of information can be delivered to the user
3. Action, tools or self-help: at least one action-oriented resource to enable self-resolution or problem management for each branch before it terminates
4. Triage: if it’s appropriate for the domain, diagnosis questions can filter out users with particular problems or needs that require special attention and transition some cases to other service providers
5. Streaming: determine the appropriate service or process to direct users to at the conclusion of their interaction with the expert system

Convert to questions and answers

After the basic structure is created, the knowledge engineers can convert it to a question and answer format. A question will be placed at each branch just before it splits. Answers will take users down specific branches.

In the language of expert systems, the question invites the user to provide input to the system. The user’s input causes a specific rule to fire, moving the user through the knowledge base along the branch that corresponds to the answer.

Mock-up informational resources

To further boost realism, informational resources within the proto-branches can be mocked up to express the general idea of the information they contain in the context of the new question and answer format of the content.

These resources don’t have to be complete at this point. They will likely undergo several revisions later.

Create proto-branch endings

Any endings to the proto-branches should be roughly mocked up to serve as the termination points for the expert system.

Load the knowledge into the system

An optional step for proto-branch creation is to load the content into the expert system. Loading it offers a chance for the knowledge engineers and subject matter experts to try the content in the system – giving them a better assessment of the flow of the finished product.

Knowledge Engineering Project Management

Knowledge engineering can be treated like a project. In some respects, the creation of a knowledge base is a one-time collection of activities that pursue a specific end in the creation of a knowledge base. This work will take considerable time and resources. Planning and management should be part of the process.

This post covers some of the unique issues and challenges for knowledge engineering. A general discussion about project management won’t be included here.

Common knowledge engineering project management questions

Some of the questions I get asked before starting a knowledge engineering project come up again and again:

• How long will it take?
• What will it cost?
• How much content will we create?

They’re good questions. It’s tough to give concise answers. Below, are some of the dependencies or variables that will influence the knowledge engineering process, along with strategies to help manage them from a project management perspective.

Domain variables:

An un-engineered domain is a black box. The cost and the amount of time required to knowledge engineer the domain will actually depend on what’s in the domain.

Knowledge engineers aren’t experts in the domain. They don’t know how complex it is and how detailed the knowledge base will be. Unless they’ve done any knowledge engineering before, subject matter experts likely won’t know long it will take or how much it will cost to knowledge engineer a domain.

Problems might lurk in the domain. They might have to do with knowledge or with the process of putting knowledge into a rule-based system that lacks common sense. Finding solutions can chew up time and resources.

Project management strategies:

1. Allow for some proto-knowledge engineering activities to let knowledge engineers and subject matter experts get into the domain and start working on it. They should get an impression of what the domain contains.
2. Manage scope carefully. Allow the team to scope down the work and revise expectations after work begins.
3. Use experienced knowledge engineers to help work through any problems. Sometimes they’ll have seen similar issues and can provide potential solutions quickly for experimentation.
4. If you get in trouble, time-box the work. Set a deadline for hitting a milestone (even if it’s arbitrarily set) and work toward it – even if it doesn’t seem right.

Specificity & granularity variables:

Knowledge content for an expert system knowledge base can vary widely in terms of specificity or granularity. Do you want 1 rule saying “beach” or 1 rule for each grain of sand on that beach?

Project management strategies:

1. Create rough expectations on how granular the knowledge base will be. Revise if necessary as knowledge engineering progresses.
2. Consider using templates that encourage patterns and levels of granularity to manage the volume of work.

Subject matter expert variables:

Strong subject matter experts who understand the point of knowledge engineering will make the work fast. Weak subject matter experts can make the process of translating a domain into logic based rules very slow. Obstructionists who disagree with the exercise can grind it to a halt.

Subject matter experts also need to be available for the work. They’ll often be professionals with lots of pressures and responsibilities.

Project management strategies:

1. Find the best experts you can.
2. Orient the subject matter experts to the process and help them “buy in”. If they don’t accept it, find new experts as soon as possible.
3. Make sure your experts are available for the work, and make the best use of their time.

Knowledge engineer variables: 

Assuming your knowledge engineers are competent in the methodology (that’s a key assumption), their impact on the time and resources required for the project will likely come down to availability.

Project management strategies:

1. Make sure your knowledge engineers have a sufficient number of hours and days per week available to keep the work moving at a decent pace.
2. Encourage your knowledge engineers to be flexible: they have to accommodate last minute scheduling changes that will come from their busy subject matter experts. Rescheduled sessions are 100% more productive than cancelled sessions.
3. Make sure knowledge engineers are working on their own or with other team members to maximize productivity between live facilitated sessions with subject matter experts.

Scheduling variables: 

Open-ended work times lead to more uncertainties around duration and frequency of knowledge engineering work sessions. Team members should at least know how often to be available.

Project management strategies:

1. Set a minimum amount of time for team members. For example:
   

   Subject matter experts: ask them to be available for 2 facilitation sessions a week x 1.5 hours long.

   Knowledge engineers: they have to lead the facilitation 2 sessions a week x 1.5 hours long. They should also have at least 2 extra hours per meeting of pre-work or follow up. Team members playing supporting roles could require much more time.

2. As noted above, allow for flexibility to move these meetings around if subject matter experts (who usually have other jobs) need to change meeting times and dates at the last minute.

Work stage variables: 

It’s easy to think only about creating knowledge base content when we think about knowledge engineering. But it’s only part of the work. Other stages include refinement, review, approval, testing, user feedback evaluation and…project planning.

Project management strategies:

1. Create an end to end work plan covering as many of the stages as possible, starting with conceptualization and ending with ongoing maintenance.
2. Build in some time for experimentation (“what if we try this?”, rework “let’s see if we can make this part better”, and methodology changes “this approach isn’t working for the system we want to build so we’ll have to change it – for all the content”).
3. Be realistic. Be prepared for people who aren’t familiar with the process to tell you “how long they think it should take”.

Back to Table of Contents