This post covers the work phases required to create and deploy a working expert system knowledge base.
Pre-work
We’ve already covered a lot of the steps that happen before knowledge engineering actually starts. They include things like orientation, scoping and prototyping.
Raw content creation
This is the big part! Knowledge engineers work with subject matter experts to capture expert domain knowledge and structure it in the right form for the expert system’s knowledge base.
Subject matter expert validation
After the expert knowledge has been captured and structured in the right form, the knowledge engineering team reviews it with subject matter experts to validate it.
In reviewing the content, knowledge engineers will be asking questions like:
- Is this right?
- Does this structure make sense?
- Is there a better way to do this?
- Will users understand it?
Once the subject matter experts have validated the knowledge base content, it can go to the next stage.
Refinement
A specialist needs to refine all the knowledge base content and refine it for style, grammar, punctuation, layout and overall structure.
Review & approval
A subject matter expert – ideally someone other than the subject matter experts used in the content creation process – needs to review all the knowledge base content for errors, omissions or other inaccuracies.
If you’re knowledge engineering content for legal expert systems like I do, you’ll want this person to be a lawyer with strong expertise in the area.
ETL (extract, transform & load)
This stage is a little bit like programming the system. A specialist takes the content and extracts the rules the expert system needs, transforms it into a language and format the system can understand and loads it into the knowledge base.
Systems that include a “pathway builder” right in the platform may be able to skip this step.
QA (quality assurance) testing
Knowledge content that has been loaded into the system must go through some initial quality assurance testing. Subject matter expUerts and other knowledge engineering team members can run through the knowledge to make sure it flows properly, displays correctly, and is free from obvious errors like typos and formatting.
User testing
Knowledge base content can pass through several stages of user testing. I recommend an “expanding circle” approach:
- Start with a small, highly controlled group of users
- Incorporate the first round of feedback into knowledge base improvements
- Expand the circle of testers and test again
- Incorporate this second round of feedback into knowledge base improvements
- Repeat steps #3 and #4 as required
- Closed beta testing: invite real users to test the system, but in a controlled environment. Rather than make your expert system freely available online, keep its location concealed, and only point the users to it via email or another direct means
- Incorporate closed beta testing into knowledge base improvements
- Open beta testing: make the expert system freely available online, without controlling direct access for users
- Incorporate open beta feedback into knowledge base improvements
By the time knowledge base has passed through the successive rounds of user testing, it will contain a significant volume of improvements and validation, all based on feedback from users.
It should be ready to launch!
Deployment
The expert system knowledge base can now go into “production” or “ship”. It’s ready to help real people with real problems in the real world.
Continuous improvement
Opening up your content to your users doesn’t mean you’re done; it’s just the start of the continuous improvement process.
Knowledge engineers and content experts can review data collected from the system – both analytics and qualitative user feedback – to identify improvements. These improvements are created, reviewed and loaded into the system.
Knowledge base curation
Someone should be in control of, and responsible for, all the expert content in an expert system’s knowledge base. This person responds to problems, whether it’s with the platform or with content, and has the final say on all things relating to curation, including things like version control.
Subject matter expert network
When the initial knowledge base creation and loading work is complete, subject matter experts will no longer be engaged in the intensive facilitated knowledge engineering work. But they should stay involved.
Knowledge engineers can create networks of subject matter experts to help with problems and to recommend changes based on new developments, discoveries, changes in the law and so on.
A subject matter expert network may involve a regularly scheduled meeting with subject matter experts every 6 months. The agenda might look like this:
- Review any errors, problems or weak points identified by users in the last 6 months
- Identify new developments or changes in this domain in the last 6 months
- If there is no significant work relating to items #1 or #2, then review a portion of the knowledge base to identify potential improvements