HowTo: Difference between revisions
(59 intermediate revisions by the same user not shown) | |||
Line 1: | Line 1: | ||
{{language|master page=HowTo|language=en}} | |||
= How to Properly Semantify Your Data Gathering and Querying Needs using a Semantified Wiki to create a fitting Knowledge Graph = | = How to Properly Semantify Your Data Gathering and Querying Needs using a Semantified Wiki to create a fitting Knowledge Graph = | ||
This guide outlines a systematic approach starting with your Needs and ending up with a fully operational Semantic MediaWiki that you may use as a prototype | This guide outlines a systematic approach starting with your Needs and ending up with a fully operational Semantic MediaWiki that you may use as a prototype | ||
or even for production. | |||
== | = Steps = | ||
== Start with gathering your needs == | |||
You can do this systematically ({{Link|target=smartRQM}}), semi-systematically or fully freestyle in natural languages. Just make sure you get to an '''agreement''' between the stakeholders about | |||
what is '''relevant''' and capture the needs in '''computer readable''' form. | |||
== Set Up Your Wiki == | |||
Utilize ProfiWiki and py-mediawikidocker to set up your Semantic MediaWiki environment. | Utilize ProfiWiki and py-mediawikidocker to set up your Semantic MediaWiki environment. | ||
This provides a solid foundation for your semantified wiki in a few minutes. | This provides a solid foundation for your semantified wiki in a few minutes. | ||
== Create a Context for your project == | |||
You'll find examples context right in this wiki at {{Link|target=List of Contexts}} | You'll find examples context right in this wiki at {{Link|target=List of Contexts}} | ||
Currently there are {{#ask: [[Concept:Context]]|format=count}} example Contexts available: | Currently there are {{#ask: [[Concept:Context]]|format=count}} example Contexts available: | ||
Line 26: | Line 26: | ||
|order=ascending | |order=ascending | ||
}} | }} | ||
== Generate Semantic Structure == | |||
Use the y-principle generator to automatically create: | |||
* Concept pages | |||
* Templates | |||
* Forms | |||
* Help pages | |||
* Properties | |||
* List pages | |||
That the Topic/Properties and TopicLinks needed for your Context. | |||
This approach ensures consistency and reduces redundancy in your wiki structure. | |||
You'll get visualizations of your structure in UML and a prototype Wiki content as a result | |||
== Convert Existing Content == | |||
If you have existing content (like the SMWCon List of Attendees), convert it to use the newly created semantic structure. This may involve: | |||
* Transforming existing content to semantic mediawiki pages (e.g by adapting existing templates or importing structured content) | |||
* Replacing free-text information with property assignments | |||
* Creating new pages for concepts that were previously only mentioned in free text | |||
* changing the Structure to faciliated linking between contents to create a true knowled graph | |||
== Maintain Consistency == | |||
Regularly review and update your semantic structure. As your domain evolves, make sure to: | |||
* Update the Context | |||
* Regenerate semantic elements and the diagrams as needed | |||
* Refactor existing content to match the updated structure | |||
= Tools = | |||
All tools being used are open source and have been used for many years in quite a few projects already. | |||
== wikipush Toolkit == | |||
[https://wiki.bitplan.com/index.php/Py-3rdparty-mediawiki wikipush toolkit at github] | |||
== py-sidif == | |||
[https://wiki.bitplan.com/index.php/Py-sidif py-sidif] is used to parse [https://wiki.bitplan.com/index.php/SiDIF '''Si'''mple '''D'''ata '''I'''nterchange ''F''ormat] Triple files | |||
SiDIF is a simplified Triple serialization format in which | |||
<pre>dad loves mom</pre> | |||
is a valid triple declaration and the technical details are a separate concern which is hidden. (note how the [https://en.wikipedia.org/wiki/Resource_Description_Framework Resource Description Framework RDF] does not follow this separation of concerns but hard codes the use the [https://en.wikipedia.org/wiki/HTTP Hypertext Transfer Protocol] into it's design) | |||
== pymediawikidocker == | |||
[https://wiki.bitplan.com/index.php/Pymediawikidocker pymediawikidocker] automates the docker based setups of (Semantic) MediaWikis and supports migration from version to version by running multiple compatible wikis in parallel as a cluster. | |||
== pyMetaModel == | |||
[https://wiki.bitplan.com/index.php/PyMetaModel PyMetaModel] is a python library that can handle Contexts that are defined based on the {{Link|target=MetaModel}} either directly from {{Sidif}} or ProfiWiki input | |||
== py-y-principle generator == | |||
The [https://wiki.bitplan.com/index.php/Py-yprinciple-gen Python based Y-Principle generator] supports separation of concerns along the Y-Principle: domain and technical aspects are combined via a matrix of generation tasks. This way we do not have to manually work on all technical aspects for all domains. The number of tasks is much lower if we only have to supply the rules for the generation. So if you e.g. have 10 domains and 10 technical elements instead of 10x10=100 tasks you only have to do 10 technical descriptions + 10 domain descriptions + 10 mapping rules - that is only 30 steps in total. | |||
== AI / ChatBots == | |||
With AI and Chatbots following this process is even simpler, faster and more effective. | |||
= Examples = | |||
== BookContext == | |||
=== Gathering the Needs === | |||
In the {{Link|target=WikiSon}} explanation | |||
<pre> | |||
{{ Book | Lord of the Rings | 0261102389 }} | |||
</pre> | |||
is given as an example Wiki notation for referencing the book {{Link|target=Lord of the Rings}}. | |||
We have formally gathered the Requirements for this example as: | |||
* Project: {{Link|target=BookProject}} | |||
* Requirements: {{Link|target=BookWithISBN}} | |||
* Acceptance Criterion: {{Link|target=BookLordOfTheRings}} | |||
But could have also simply used some examples in natural language and get the help from an AI Chatbot. | |||
=== Set Up Your Wiki === | |||
We are using this wiki as a demo the {{Link|target=MetaModel}} is already available. | |||
=== Context Creation === | |||
For the Context creation we asked Claude AI: {{Link|target=Book_Experiment}} resulting in {{Link|target=BookContext}} | |||
== CityContext == | |||
=== Gathering the Needs === | |||
[https://www.semantic-mediawiki.org/wiki/Category:City City pages] have been used to explain SemanticMediaWiki since 2012. | |||
To start we copy the Demo Pages locally: | |||
<source lang='bash'> | |||
wikipush -s smw -t contexts -q "[[Category:City]][[Demo:~*]][[Population::+]]" -wi -f | |||
copying 13 pages from smw to contexts | |||
1/13 ( 8%): copying ... Demo:Amsterdam✅ | |||
... | |||
copying image File:Tokyo-Tsukishima-0011.jpg ...✅ | |||
13/13 ( 100%): copying ... Demo:Würzburg✅ | |||
</source> | |||
=== Set Up Your Wiki === | |||
We will use the existing https://smw.bitplan.com partial copy of https://www.semantic-mediawiki.org/. | |||
==== copy needed base ProfWiki pages ==== | |||
First we need to make sure {{Link|target=Property:Needed_for|title=needed pages for ProfiWiki}} are available. | |||
<source lang='bash' highlight='1'> | |||
wikipush -s contexts -t smwcopy -q "[[Needed_for::ProfiWiki]]" -wi -f | |||
copying 23 pages from contexts to smwcopy | |||
1/23 ( 4%): copying ... Template:!✅ | |||
2/23 ( 9%): copying ... File:Actionicon.pngcopying image File:Actionicon.png ...✅ | |||
3/23 ( 13%): copying ... File:Component.pngcopying image File:Component.png ...✅ | |||
4/23 ( 17%): copying ... File:Element into input.pngcopying image File:Element into input.png ...✅ | |||
5/23 ( 22%): copying ... File:Help Icon.pngcopying image File:Help Icon.png ...✅ | |||
6/23 ( 26%): copying ... HowTo/de✅ | |||
7/23 ( 30%): copying ... Template:IconLink✅ | |||
8/23 ( 35%): copying ... Icons✅ | |||
9/23 ( 39%): copying ... File:Index.pngcopying image File:Index.png ...✅ | |||
10/23 ( 43%): copying ... Property:IsA✅ | |||
11/23 ( 48%): copying ... Template:Link✅ | |||
12/23 ( 52%): copying ... Property:Link target✅ | |||
13/23 ( 57%): copying ... Main Page✅ | |||
copying image File:Actionicon.png ...⚠️:file exists (to overwrite existing files enable the ignore parameter) | |||
copying image File:Component.png ...⚠️:file exists (to overwrite existing files enable the ignore parameter) | |||
copying image File:Element into input.png ...⚠️:file exists (to overwrite existing files enable the ignore parameter) | |||
copying image File:Index.png ...⚠️:file exists (to overwrite existing files enable the ignore parameter) | |||
copying image File:Link.png ...✅ | |||
copying image File:Masterlogo.png ...✅ | |||
copying image File:Registry.png ...✅ | |||
14/23 ( 61%): copying ... MetaModel✅ | |||
copying image File:Actionicon.png ...⚠️:file exists (to overwrite existing files enable the ignore parameter) | |||
copying image File:Component.png ...⚠️:file exists (to overwrite existing files enable the ignore parameter) | |||
copying image File:Element into input.png ...⚠️:file exists (to overwrite existing files enable the ignore parameter) | |||
copying image File:Index.png ...⚠️:file exists (to overwrite existing files enable the ignore parameter) | |||
copying image File:Link.png ...⚠️:file exists (to overwrite existing files enable the ignore parameter) | |||
copying image File:Registry.png ...⚠️:file exists (to overwrite existing files enable the ignore parameter) | |||
15/23 ( 65%): copying ... Property:Needed for✅ | |||
16/23 ( 70%): copying ... Template:NeededFor✅ | |||
17/23 ( 74%): copying ... Template:PageSequence✅ | |||
18/23 ( 78%): copying ... ProfiWiki/NeededPages✅ | |||
copying image File:Actionicon.png ...⚠️:file exists (to overwrite existing files enable the ignore parameter) | |||
copying image File:Component.png ...⚠️:file exists (to overwrite existing files enable the ignore parameter) | |||
copying image File:Element into input.png ...⚠️:file exists (to overwrite existing files enable the ignore parameter) | |||
copying image File:Index.png ...⚠️:file exists (to overwrite existing files enable the ignore parameter) | |||
copying image File:Profiwikiicon.png ...✅ | |||
copying image File:Registry.png ...⚠️:file exists (to overwrite existing files enable the ignore parameter) | |||
19/23 ( 83%): copying ... File:Profiwikiicon.pngcopying image File:Profiwikiicon.png ...⚠️:file exists (to overwrite existing files enable the ignore parameter) | |||
20/23 ( 87%): copying ... File:Registry.pngcopying image File:Registry.png ...⚠️:file exists (to overwrite existing files enable the ignore parameter) | |||
21/23 ( 91%): copying ... Template:TemplateSequence✅ | |||
22/23 ( 96%): copying ... Template:UtfIcon✅ | |||
23/23 ( 100%): copying ... Template:WhatLinksHere✅ | |||
</source> | |||
==== generate the MetaModel pages ==== | |||
This can be done via command line or interactively. | |||
=== Create Context === | |||
{{Link|target=CityContext}} is based on the properties of the Demo pages but uses {{WikiSon}} notation. | |||
=== Convert Content === | |||
The pages have properties in traditional notation e.g. [{{fullurl:Demo:Tokyo|oldid=549}} Demo:Tokyo] | |||
<pre> | |||
{{City | |||
|collection = 東京 | |||
}} | |||
Tokyo (also known as [[Has title::東京]] or Tokio) is the capital city, the center of the Greater Tokyo Area and one of the [[Has prefecture::tokyo|47 prefectures]] of [[Located in::Japan]]. Its coordinates are [[Has coordinates::35.6833° N, 139.7667° E]] | |||
The [[Population::8,967,665|population]] of its special wards accounts for over 8 million people, with the total population of the prefecture exceeding 13 million. | |||
[[Category:City]] | |||
</pre> | |||
== SMWCon == | == SMWCon == | ||
SMWCon is the acronym for the Semantic Mediawiki conference | SMWCon is the acronym for the Semantic Mediawiki conference | ||
Line 42: | Line 193: | ||
Over the years, additions were made, but unfortunately, SMWCon didn't fully embrace the power of self -reflective semantification. | Over the years, additions were made, but unfortunately, SMWCon didn't fully embrace the power of self -reflective semantification. | ||
=== Gathering the needs === | |||
We need to build on the existing pages from the categories | |||
* [https://www.semantic-mediawiki.org/wiki/Category:Event Event] | |||
* [https://www.semantic-mediawiki.org/wiki/Category:Talks Talks] | |||
=== | === Creating the Context === | ||
{{Link|target=SMWCon}} is the Context description in computer readable format it is based on | |||
* https://www.semantic-mediawiki.org/wiki/List_of_Attendees | |||
* https://wiki.bitplan.com/index.php/Attendee | |||
and example properties such as: | |||
* [https://www.semantic-mediawiki.org/wiki/Property:Talk_status Talk status] | |||
* [https://www.semantic-mediawiki.org/wiki/Property:Attendee_picture Attendee picture] | |||
As of 2024-08-03 we have simply asked Claude AI to create the wikipage with SiDIF format based on that input which only needed minor corrections and took a few minutes. | |||
== | === Lessons Learned from SMWCon === | ||
# Avoid Redundancy: Don't copy-paste content. Use semantic properties and queries to display information in multiple contexts. | |||
# Eat Your Own Dog Food: If you're promoting semantification, use it extensively in your own wiki. | |||
# Maintain Consistency: Regularly review and update your semantic structure to ensure it remains relevant and efficient. | |||
# Automate Where Possible: Use tools like the y-principle generator to maintain consistency and reduce manual work. | |||
# Embrace Self-Reflection: Implement a system where your wiki's structure is itself described semantically, allowing for easier management and evolution of your knowledge base. | |||
= | = Links = | ||
* [https://www.semantic-mediawiki.org/wiki/Metadata_management Metadata management in Semantic MediaWiki] | * [https://www.semantic-mediawiki.org/wiki/Metadata_management Metadata management in Semantic MediaWiki] | ||
* [https://www.semantic-mediawiki.org/wiki/Help:Properties Working with properties in SMW] | * [https://www.semantic-mediawiki.org/wiki/Help:Properties Working with properties in SMW] | ||
{{LLMHint}} | |||
[[Category:ProfiWiki]] |
Latest revision as of 07:27, 12 August 2024
This page in other languages: de
How to Properly Semantify Your Data Gathering and Querying Needs using a Semantified Wiki to create a fitting Knowledge Graph
This guide outlines a systematic approach starting with your Needs and ending up with a fully operational Semantic MediaWiki that you may use as a prototype or even for production.
Steps
Start with gathering your needs
You can do this systematically (smartRQM), semi-systematically or fully freestyle in natural languages. Just make sure you get to an agreement between the stakeholders about what is relevant and capture the needs in computer readable form.
Set Up Your Wiki
Utilize ProfiWiki and py-mediawikidocker to set up your Semantic MediaWiki environment. This provides a solid foundation for your semantified wiki in a few minutes.
Create a Context for your project
You'll find examples context right in this wiki at List of Contexts Currently there are 16 example Contexts available:
Generate Semantic Structure
Use the y-principle generator to automatically create:
- Concept pages
- Templates
- Forms
- Help pages
- Properties
- List pages
That the Topic/Properties and TopicLinks needed for your Context. This approach ensures consistency and reduces redundancy in your wiki structure.
You'll get visualizations of your structure in UML and a prototype Wiki content as a result
Convert Existing Content
If you have existing content (like the SMWCon List of Attendees), convert it to use the newly created semantic structure. This may involve:
- Transforming existing content to semantic mediawiki pages (e.g by adapting existing templates or importing structured content)
- Replacing free-text information with property assignments
- Creating new pages for concepts that were previously only mentioned in free text
- changing the Structure to faciliated linking between contents to create a true knowled graph
Maintain Consistency
Regularly review and update your semantic structure. As your domain evolves, make sure to:
- Update the Context
- Regenerate semantic elements and the diagrams as needed
- Refactor existing content to match the updated structure
Tools
All tools being used are open source and have been used for many years in quite a few projects already.
wikipush Toolkit
py-sidif
py-sidif is used to parse Simple Data Interchange Format Triple files SiDIF is a simplified Triple serialization format in which
dad loves mom
is a valid triple declaration and the technical details are a separate concern which is hidden. (note how the Resource Description Framework RDF does not follow this separation of concerns but hard codes the use the Hypertext Transfer Protocol into it's design)
pymediawikidocker
pymediawikidocker automates the docker based setups of (Semantic) MediaWikis and supports migration from version to version by running multiple compatible wikis in parallel as a cluster.
pyMetaModel
PyMetaModel is a python library that can handle Contexts that are defined based on the MetaModel either directly from Template:Sidif or ProfiWiki input
py-y-principle generator
The Python based Y-Principle generator supports separation of concerns along the Y-Principle: domain and technical aspects are combined via a matrix of generation tasks. This way we do not have to manually work on all technical aspects for all domains. The number of tasks is much lower if we only have to supply the rules for the generation. So if you e.g. have 10 domains and 10 technical elements instead of 10x10=100 tasks you only have to do 10 technical descriptions + 10 domain descriptions + 10 mapping rules - that is only 30 steps in total.
AI / ChatBots
With AI and Chatbots following this process is even simpler, faster and more effective.
Examples
BookContext
Gathering the Needs
In the WikiSon explanation
{{ Book | Lord of the Rings | 0261102389 }}
is given as an example Wiki notation for referencing the book Lord of the Rings.
We have formally gathered the Requirements for this example as:
- Project: BookProject
- Requirements: BookWithISBN
- Acceptance Criterion: BookLordOfTheRings
But could have also simply used some examples in natural language and get the help from an AI Chatbot.
Set Up Your Wiki
We are using this wiki as a demo the MetaModel is already available.
Context Creation
For the Context creation we asked Claude AI: Book_Experiment resulting in BookContext
CityContext
Gathering the Needs
City pages have been used to explain SemanticMediaWiki since 2012. To start we copy the Demo Pages locally:
wikipush -s smw -t contexts -q "[[Category:City]][[Demo:~*]][[Population::+]]" -wi -f
copying 13 pages from smw to contexts
1/13 ( 8%): copying ... Demo:Amsterdam✅
...
copying image File:Tokyo-Tsukishima-0011.jpg ...✅
13/13 ( 100%): copying ... Demo:Würzburg✅
Set Up Your Wiki
We will use the existing https://smw.bitplan.com partial copy of https://www.semantic-mediawiki.org/.
copy needed base ProfWiki pages
First we need to make sure needed pages for ProfiWiki are available.
wikipush -s contexts -t smwcopy -q "[[Needed_for::ProfiWiki]]" -wi -f
copying 23 pages from contexts to smwcopy
1/23 ( 4%): copying ... Template:!✅
2/23 ( 9%): copying ... File:Actionicon.pngcopying image File:Actionicon.png ...✅
3/23 ( 13%): copying ... File:Component.pngcopying image File:Component.png ...✅
4/23 ( 17%): copying ... File:Element into input.pngcopying image File:Element into input.png ...✅
5/23 ( 22%): copying ... File:Help Icon.pngcopying image File:Help Icon.png ...✅
6/23 ( 26%): copying ... HowTo/de✅
7/23 ( 30%): copying ... Template:IconLink✅
8/23 ( 35%): copying ... Icons✅
9/23 ( 39%): copying ... File:Index.pngcopying image File:Index.png ...✅
10/23 ( 43%): copying ... Property:IsA✅
11/23 ( 48%): copying ... Template:Link✅
12/23 ( 52%): copying ... Property:Link target✅
13/23 ( 57%): copying ... Main Page✅
copying image File:Actionicon.png ...⚠️:file exists (to overwrite existing files enable the ignore parameter)
copying image File:Component.png ...⚠️:file exists (to overwrite existing files enable the ignore parameter)
copying image File:Element into input.png ...⚠️:file exists (to overwrite existing files enable the ignore parameter)
copying image File:Index.png ...⚠️:file exists (to overwrite existing files enable the ignore parameter)
copying image File:Link.png ...✅
copying image File:Masterlogo.png ...✅
copying image File:Registry.png ...✅
14/23 ( 61%): copying ... MetaModel✅
copying image File:Actionicon.png ...⚠️:file exists (to overwrite existing files enable the ignore parameter)
copying image File:Component.png ...⚠️:file exists (to overwrite existing files enable the ignore parameter)
copying image File:Element into input.png ...⚠️:file exists (to overwrite existing files enable the ignore parameter)
copying image File:Index.png ...⚠️:file exists (to overwrite existing files enable the ignore parameter)
copying image File:Link.png ...⚠️:file exists (to overwrite existing files enable the ignore parameter)
copying image File:Registry.png ...⚠️:file exists (to overwrite existing files enable the ignore parameter)
15/23 ( 65%): copying ... Property:Needed for✅
16/23 ( 70%): copying ... Template:NeededFor✅
17/23 ( 74%): copying ... Template:PageSequence✅
18/23 ( 78%): copying ... ProfiWiki/NeededPages✅
copying image File:Actionicon.png ...⚠️:file exists (to overwrite existing files enable the ignore parameter)
copying image File:Component.png ...⚠️:file exists (to overwrite existing files enable the ignore parameter)
copying image File:Element into input.png ...⚠️:file exists (to overwrite existing files enable the ignore parameter)
copying image File:Index.png ...⚠️:file exists (to overwrite existing files enable the ignore parameter)
copying image File:Profiwikiicon.png ...✅
copying image File:Registry.png ...⚠️:file exists (to overwrite existing files enable the ignore parameter)
19/23 ( 83%): copying ... File:Profiwikiicon.pngcopying image File:Profiwikiicon.png ...⚠️:file exists (to overwrite existing files enable the ignore parameter)
20/23 ( 87%): copying ... File:Registry.pngcopying image File:Registry.png ...⚠️:file exists (to overwrite existing files enable the ignore parameter)
21/23 ( 91%): copying ... Template:TemplateSequence✅
22/23 ( 96%): copying ... Template:UtfIcon✅
23/23 ( 100%): copying ... Template:WhatLinksHere✅
generate the MetaModel pages
This can be done via command line or interactively.
Create Context
CityContext is based on the properties of the Demo pages but uses WikiSon notation.
Convert Content
The pages have properties in traditional notation e.g. Demo:Tokyo
{{City |collection = 東京 }} Tokyo (also known as [[Has title::東京]] or Tokio) is the capital city, the center of the Greater Tokyo Area and one of the [[Has prefecture::tokyo|47 prefectures]] of [[Located in::Japan]]. Its coordinates are [[Has coordinates::35.6833° N, 139.7667° E]] The [[Population::8,967,665|population]] of its special wards accounts for over 8 million people, with the total population of the prefecture exceeding 13 million. [[Category:City]]
SMWCon
SMWCon is the acronym for the Semantic Mediawiki conference The lessons learned from the SMWCon experience and best practices in Semantic MediaWiki (SMW) are the basis for this example
Historical Context: SMWCon and List of Attendees
The journey of semantification for SMWCon began with the creation of the first SMWCon page using SemanticMediaWiki properties to capture information about SMWCon Events. https://www.semantic-mediawiki.org/wiki/SMWCon_Fall_2011 is an example for such a SMWCon page.
The proper way to handle the catalog of properties (Metadata management) has been much debated.
A self-reflective, semantified version of the Attendees list, as proposed by Pierre Racine (see Metadata management) of the List of Attendees at SMWCon Fal 2015 was a step forward. Over the years, additions were made, but unfortunately, SMWCon didn't fully embrace the power of self -reflective semantification.
Gathering the needs
We need to build on the existing pages from the categories
Creating the Context
SMWCon is the Context description in computer readable format it is based on
- https://www.semantic-mediawiki.org/wiki/List_of_Attendees
- https://wiki.bitplan.com/index.php/Attendee
and example properties such as:
As of 2024-08-03 we have simply asked Claude AI to create the wikipage with SiDIF format based on that input which only needed minor corrections and took a few minutes.
Lessons Learned from SMWCon
- Avoid Redundancy: Don't copy-paste content. Use semantic properties and queries to display information in multiple contexts.
- Eat Your Own Dog Food: If you're promoting semantification, use it extensively in your own wiki.
- Maintain Consistency: Regularly review and update your semantic structure to ensure it remains relevant and efficient.
- Automate Where Possible: Use tools like the y-principle generator to maintain consistency and reduce manual work.
- Embrace Self-Reflection: Implement a system where your wiki's structure is itself described semantically, allowing for easier management and evolution of your knowledge base.
Links
⚠️ LLM-generated content notice: Parts of this page may have been created or edited with the assistance of a large language model (LLM). The prompts that have been used might be on the page itself, the discussion page or in straight forward cases the prompt was just "Write a mediawiki page on X" with X being the page name. While the content has been reviewed it might still not be accurate or error-free.