Help
Difference between revisions of "SPARQL"
Help:SPARQL gathers a list of basic SPARQL queries in the context of Lingua Libre, demoed and ready to test, together with beginners-friendly knowledges, inline-comments, introductions to concepts, code snippets and few tools. This page allows users not familiar with SPARQL to rapidly learn the basics of SPARQL, query the LinguaLibre database, and to download or directly feed that data to an application. To fit the with most frequent usages, the case of a web developper with basic Javascripts skill is taken.
(→Tools) |
(→Base) |
||
Line 3: | Line 3: | ||
{{Draft|2021/12/10 : Work in progress. '''Please do not translate yet''' as sections are still under active changes. You may help by: reading/fixing the page, testing queries [https://lingualibre.org/bigdata/#query here], replacing Q21 (French) and Q42 (User:0x010C) by a smaller non-western languages and users, harmonising in-line comments, adding the right category to this page. --[[User:Yug|Yug]]}} | {{Draft|2021/12/10 : Work in progress. '''Please do not translate yet''' as sections are still under active changes. You may help by: reading/fixing the page, testing queries [https://lingualibre.org/bigdata/#query here], replacing Q21 (French) and Q42 (User:0x010C) by a smaller non-western languages and users, harmonising in-line comments, adding the right category to this page. --[[User:Yug|Yug]]}} | ||
− | == Base == | + | == Base == |
− | |||
− | |||
* [[Special:ListProperties]] – exhaustive list of LinguaLibre's Wikibase properties. | * [[Special:ListProperties]] – exhaustive list of LinguaLibre's Wikibase properties. | ||
* [[LinguaLibre:List of languages]] – exhaustive list of LinguaLibre's languages | * [[LinguaLibre:List of languages]] – exhaustive list of LinguaLibre's languages | ||
− | * [[ | + | |
− | * [[ | + | === Tools === |
+ | [[File:Wikidata_Query_-_Query_Helper_-_Build_query_from_scratch.webm|thumb|450px|On Wikidata, the WDQS allows to practice SPARQL queries creation in an intuitive way.]] | ||
+ | * [{{SERVER}}/bigdata/#query <span class="mw-ui-button mw-ui-progressive" role="button" aria-disabled="false">Endpoint</span>] LinguaLibre Query Service (LLQS) – run SPARQL Queries upon LinguaLibre. Run, test, download the data as json, csv or tsv. | ||
+ | * [https://query.wikidata.org Wikidata Query Service (WDQS)] – run SPARQL Queries upon Wikidata. Run, test, download the data as json, csv or tsv. | ||
+ | * [https://sinaahmadi.github.io/posts/sparql-query-generator-for-lexicographical-data.html Wikidata Lexeme Queries generators] ([https://jsfiddle.net/hugolpz/rygo9s5b/ hack me]) by @sina_ahm – helps to create queries for Wikidata's Lexeme. | ||
+ | * [[Special:ApiSandbox]] – API queries generator for Lingualibre wikipage and wikibase contents. | ||
+ | |||
+ | === References === | ||
+ | * [https://www.w3.org/TR/sparql11-query/ SPARQL 1.1 Query Language] | ||
+ | * [https://www.iro.umontreal.ca/~lapalme/ift6281/sparql-1_1-cheat-sheet.pdf SPARQL Cheatsheet] | ||
== Code snippets == | == Code snippets == |
Revision as of 19:50, 10 December 2021
2021/12/10 : Work in progress. Please do not translate yet as sections are still under active changes. You may help by: reading/fixing the page, testing queries here, replacing Q21 (French) and Q42 (User:0x010C) by a smaller non-western languages and users, harmonising in-line comments, adding the right category to this page. --Yug
Base
- Special:ListProperties – exhaustive list of LinguaLibre's Wikibase properties.
- LinguaLibre:List of languages – exhaustive list of LinguaLibre's languages
Tools
- LinguaLibre Query Service (LLQS) – run SPARQL Queries upon LinguaLibre. Run, test, download the data as json, csv or tsv.
- Wikidata Query Service (WDQS) – run SPARQL Queries upon Wikidata. Run, test, download the data as json, csv or tsv.
- Wikidata Lexeme Queries generators (hack me) by @sina_ahm – helps to create queries for Wikidata's Lexeme.
- Special:ApiSandbox – API queries generator for Lingualibre wikipage and wikibase contents.
References
Code snippets
Fetch data using SPARQL
LinguaLibre data can be fetched using various coding languages such as Python, Javascript, R and others, returning JSON or other formats.
- For code snippet in your language : open query.wikidata.org (WikiData Query Service, aka WDQS), run your SPARQL query, click "Code" : a pop up window appears with various implementations.
- For downloading data, click "Download".
Javascript:
At least 3 methods exists (code snippet), example:
Query | Result's basic unit |
---|---|
SPARQL:SELECT ?item WHERE { ?item prop:P2 entity:Q5 } LIMIT 10
|
{ … },
{
"item": {
"type": "uri",
"value": "https://lingualibre.org/entity/Q12"
},
"itemLabel": {
"xml:lang": "en",
"type": "literal",
"value": "beginner"
}
},
{ … }
|
Javascript:
var endpoint = 'https://lingualibre.org/sparql';
var sparql = 'SELECT ?item WHERE { ?item prop:P2 entity:Q5 } LIMIT 10';
$.getJSON(endpoint,
{ query: sparql, format: 'json' },
function(data){ console.log('JQuery: ',data)}
);
|
Merging data
Advanced SPARQL queries with COUNT()
and others are often slow (>3secs, sometime >100secs). You are encouraged to do multiple smaller SPARQL queries to then merge their responded data. By example, the complementary Javascript snippet below would help web developers to do so.
// Data from 3 sparql queries.
// Important: One key must be similar in all datasets, here: 'qid'
const langs = [{ qid: 'Q209', label: 'Breton', iso:'bre' }, { qid: 'Q21', label: 'French', iso: 'fra' }],
speakersFemales = [{ qid: 'Q209', genderF: 3, recordsF: 60 }, { qid: 'Q21', genderF: 21, recordsF:15046 }],
speakersMales = [{ qid: 'Q209', genderM: 7, recordsM: 112 }, { qid: 'Q21', genderM: 85, recordsM:82964 }];
// Toolbox for merging data by same id
var merge2ArraysBySameId = function(arr1,arr2,id1){
return arr1.map( item1 => {
var identical = arr2.find(obj => obj[id1] === item1[id1]);
return Object.assign(identical, item1)
} );
}
// Mergings
var step1 = merge2ArraysBySameId(langs,speakersFemales,'qid');
var step2 = merge2ArraysBySameId(step1,speakersMales,'qid');
alert(JSON.stringify(step2))
Lingualibre's ground
✅ Is Language (language/dialect (Q4)) → List existing languages with: LL Qid, ISO 639-3, Name
SELECT ?lang ?iso ?langLabel WHERE {
?lang prop:P2 entity:Q4 .
?lang prop:P13 ?iso .
SERVICE wikibase:label {
bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en" .
}
}
|
|
✅🇶 Is Speaker (speaker (Q3)) → List existing speakers
SELECT ?speaker ?speakerLabel
WHERE {
?speaker prop:P2 entity:Q3 . # Condition 1, P2 'instance of' is Q3 'speaker'.
SERVICE wikibase:label {
bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en" .
}
}
|
|
✅ Is Language level (language level (Q5)) → List existing levels
SELECT ?item ?itemLabel
WHERE {
?item prop:P2 entity:Q5 # Condition 1, P2 'instance of' is Q5 'language level'.
SERVICE wikibase:label {
bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en" .
}
}
|
|
✅ Is Sex or Gender (sex or gender (Q7)) → List existing sexes or genders
SELECT ?item ?itemLabel
WHERE {
?item prop:P2 entity:Q7 # Condition 1, P2 'instance of' is Q7 'sex or gender'.
SERVICE wikibase:label {
bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en" .
}
}
|
|
Speaker
✅ Speaker name(s) → Speaker Qid(s)
SELECT ?speakerName ?speakerId
WHERE {
VALUES ?speakerName { "Yug" "VIGNERON" } # One or multiple values
BIND ( STRLANG(?speakerName, "en") AS ?speakerLabel )
# P2: instance of; Q3: speaker.
?speakerId prop:P2 entity:Q3 ; rdfs:label ?speakerLabel .
}
|
|
✅🇶 Speaker Qid (0x010C (Q42)) → Speaker data, all
# Get Q42 (User:0x010C)'s data
SELECT ?predicate ?object ?objectLabel
WHERE {
entity:Q42 ?predicate ?object .
SERVICE wikibase:label {
bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en" .
}
}
|
|
✅🇶 Speaker Qid (0x010C (Q42)) → Speaker languages (P4)
SELECT ?languages ?languagesLabel
WHERE {
entity:Q42 prop:P4 ?languages .
SERVICE wikibase:label {
bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en" .
}
}
|
|
✅ Speaker Qid (0x010C (Q42)) + Language LL Qid (French (Q21)) → List records
SELECT ?audio ?audioLabel
WHERE {
?audio prop:P5 entity:Q42 . # Filter: P5 Speaker is Q42 User:0x010C
?audio prop:P4 entity:Q21 . # Filter: P4 language is Q21 French
# Labels
SERVICE wikibase:label {
bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en" .
}
}
|
|
✅ Speaker Qid (0x010C (Q42)) + Language LL Qid (French (Q21)) → Count records
SELECT ?language ?speakerLabel (COUNT(?audio) AS ?audio)
WHERE {
VALUES ?language { entity:Q21 }
VALUES ?speaker { entity:Q42 }
?audio prop:P5 ?speaker . # Filter: P5 'speaker' is Q42 '0x010C'
?audio prop:P4 ?language . # Filter: P4 'language' is Q21 'French'
?audio prop:P2 entity:Q2 . # Filter: P2 'instance of' is Q2 'record'
# Labels
SERVICE wikibase:label {bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en"}
}
GROUP BY ?language ?speakerLabel
|
|
Languages
?✅ Language name(s) in English → Language LL Qid(s)
SELECT ?languageId ?languageName
WHERE {
VALUES ?languageName { "Marathi" "Atikamekw" "Central Bikol" } # Target values
?languageId
prop:P2 entity:Q4 ; # Filter: P2 'instance of' is Q4 'language' AND
rdfs:label ?languageLabel . # Assign value label into ?languageLabel
BIND ( STRLANG(?languageName, "en") AS ?languageLabel ) # Bind filter by English
}
|
|
✅ Language ISO-639-3 → Language LL Qid(s), Wikidata Qid, Label
SELECT ?langIso ?langId ?langWDQid ?langIdLabel
WHERE {
VALUES ?langIso { "mar" "bre" "bcl" "atj" "ban" } # Target ISO values
?langId
prop:P2 entity:Q4 ; # Filter P2 'instance of' is Q4 'language' AND
prop:P13 ?langIso ; # Assign value: P13 'Iso-639-3' to ?langIso AND
prop:P12 ?langWDQid . # Assign value: P12 'Iso-639-3' to ?langWDQid
# Labels
SERVICE wikibase:label {bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en"}
}
|
|
✅ Language LL Qid (Q21) → Count items
SELECT ?language (COUNT(?audio) AS ?nbAudio) WHERE {
VALUES ?language { entity:Q21 }
?audio prop:P4 ?language .
}
GROUP BY ?language
|
|
✅ Language LL Qid (Q21) → Count records
SELECT ?language (COUNT(?audio) AS ?audio) WHERE {
VALUES ?language { entity:Q21 }
?audio prop:P2 entity:Q2 . # P2 'instance of' is Q2 'record'
?audio prop:P4 ?language . # P4 'language' is Q21 'French'
}
GROUP BY ?language
|
|
?✅ Language LL Qid (Q21) → Count unique words
SELECT ?language
(COUNT(?audio) as ?audios) # Count and assign value to ?Audio
(COUNT(DISTINCT(?itemLabel)) AS ?words)
(ROUND(10000*?words/?audios)/100 AS ?percent)
WHERE {
VALUES ?language { entity:Q21 }
?audio prop:P2 entity:Q2 . # Filter: P2 'instance of' is Q2 'record'
?audio prop:P4 ?language . # Filter: P4 'language' is Q21 'French'
?audio rdfs:label ?itemLabel. # Assign value: label to ?itemLabel
}
GROUP BY ?language
|
|
✅ Language LL Qid (Q21) → Count speakers
SELECT ?language (COUNT(?audio) AS ?audio) WHERE {
VALUES ?language { entity:Q21 }
?audio prop:P2 entity:Q3 . # P2 'instance of' is Q3 'speaker'
?audio prop:P4 ?language . # P4 'language' is Q21 'French'
}
GROUP BY ?language
|
|
✅ Language LL Qid (Q209) → List speakers
SELECT ?language ?speaker ?speakerLabel WHERE {
VALUES ?language { entity:Q209 }
?speaker prop:P2 entity:Q3 . # P2 'instance of' is Q3 'speaker'
?speaker prop:P4 ?language . # P4 'language' is Q21 'French'
# Labels
SERVICE wikibase:label {
bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en" .
}
}
|
|
✅ Language LL Qid (Breton (Q209)) → Language data, all
'Case: Get for language Q209 'Breton' all its data.
SELECT * WHERE {
# Given Q209 'Breton language', get all properties and values
entity:Q209 ?predicate ?object .
}
|
|
✅ Language LL Qid (Breton (Q209)) → Language data, core
SELECT * WHERE {
# Given Q209 'Breton language', get all properties and values
entity:Q209 ?predicate ?object .
?predicate rdf:type owl:DatatypeProperty .
}
|
|
✅ Language LL Qid (Breton (Q209)) → Property P13 (ISO 639-3)
SELECT * WHERE {
entity:Q209 prop:P13 ?iso . # Assign value : Q209 'Breton', P13 'ISO 639-3', value into ?iso
}
|
|
✅ Languages → List existing languages' iso-639-3
SELECT * WHERE {
?lang prop:P13 ?code .
}
|
|
✅ Language WD Qid → Language data, core
SELECT * WHERE {
?lang prop:P12 "Q12107" . # P12 'Wikidata id' is Wikidata's "Q12107"
?lang ?predicate ?object . #
?predicate rdf:type owl:DatatypeProperty .
}
|
|
Records
✅ Record LL Qid (Cometa (Q500)) → Record data, all
SELECT * WHERE {
entity:Q500 ?predicate ?object .
# ?predicate rdf:type owl:DatatypeProperty .
}
|
|
✅ Record LL Qid (Cometa (Q500)) → Record data, core
SELECT * WHERE {
entity:Q500 ?predicate ?object .
?predicate rdf:type owl:DatatypeProperty .
}
|
|
✅ Language (English (Q22)) + String → Record LL Qid(s)
SELECT ?itemLabel ?item
WHERE {
?item prop:P2 entity:Q2 . # Filter: P2 'instance of' Q3 'record'
?item prop:P4 entity:Q22 . # Filter: P4 'language' is Q22 'English'
?item rdfs:label ?itemLabel. # Assign value: label to ?itemLabel
FILTER(CONTAINS(?itemLabel, "apple"@en)).
} limit 10
|
|
✅ Language (Breton (Q209)) + Speaker (ThonyVezbe (Q584098)) + String (ni) → Record LL Qid
Case: Search in Breton language, with speaker 'ThonyVezbe',
SELECT ?audio
WHERE {
?audio prop:P4 entity:Q209 . # P4 'language' is Q209 'Breton'
?audio prop:P5 entity:Q584098 . # P5 'speaker' is Q584098 'ThonyVezbe'
?audio rdfs:label ?word . #word
FILTER ( STR(?word) = "ni" ) # word = 'ni'
}
|
|
✅ Language (French (Q21)) + Speaker (Justforoc (Q137047)) + String → URL pointer, filename
SELECT ?word ?audio ?urlPointer
(replace(replace(replace(substr(STR(?urlPointer),52),"%20","_"),"%28","("),"%29",")") AS ?filename)
WHERE {
?audio prop:P4 entity:Q21 . # Filter: P4 'language' is Q21 'French'
?audio prop:P5 entity:Q137047 . # Filter: P5 'speaker' is Q137047 'Justforoc'
?audio rdfs:label ?word . # Assign value: label to ?word
#Filter: ?word with 'pomme' in French, non case-sensitive
FILTER REGEX(?word, "pomme"@fr, "i" ) .
?audio prop:P3 ?urlPointer
}
|
|
Heavy queries
Queries below are too large to run on LinguaLibre's wikipages, or even on Lingualibre Query Service).
To do: do smaller sub-queries, with one COUNT()
function.
❌ Languages → Name, Wikidata Qid, LLQid, Iso-639-3, and genders
Query | Result |
---|---|
SELECT ?languageQidLabel ?wdQid ?languageQid ?isoCode
(COUNT(DISTINCT(?record)) AS ?recordCount)
(COUNT(DISTINCT(?speakerLangM)) AS ?speakerM)
(COUNT(DISTINCT(?speakerLangF)) AS ?speakerF)
wWHERE{
?record prop:P2 entity:Q2 . # Filter: items where P2 'instance of' is Q2 'record'
?record prop:P4 ?languageQid . # Assign value: P4 'language' into variable ?language
?languageQid prop:P12 ?wdQid . # Assign value: P12 'wikidata id' into variable ?WD
?languageQid prop:P13 ?isoCode. # Assign value: P13 'iso639-3' into ?isoCode
#?record prop:P5 ?speakerQidM . # Assign value: P5 'speaker' into variable ?speakerQidM
#?speakerQidM prop:P8 entity:Q16 . # Filter: P8 'sex or gender' is Q16 'male
#?speakerQidM prop:P4 ?speakerLangM . # Assign value: P4 'language' into variable ?spakerLangM
?record prop:P5 ?speakerQidF . # Assign value: P5 'speaker' into variable ?speakerQidF
?speakerQidF prop:P8 entity:Q17 . # Filter: P8 'sex or gender' is Q17 'female
?speakerQidF prop:P4 ?speakerLangF . # Assign value: P4 'language' into variable ?spakerLangF
SERVICE wikibase:label { bd:serviceParam wikibase:language "en" . }
}
GROUP BY ?languageQidLabel ?languageQid ?wdQid ?isoCode
ORDER BY DESC(?recordCount)
|
languageQidLabel wdQid languageQid isoCode recordCount speakerM speakerF French Q150 Q21 fra 16761 0 18 Marathi Q1571 Q34 mar 13153 0 5 Polish Q809 Q298 pol 11686 0 1 … |
❌ Is Language (speaker (Q3)) → list all languages with number of unique words and speakers
SELECT ?language (COUNT(?audio) AS ?nbAudio) (COUNT(?speaker) AS ?nbSpeaker) WHERE {
?language prop:P2 entity:Q4 .
?audio prop:P4 ?language .
?speaker prop:P4 ?language .
}
GROUP BY ?language
Others
(These old queries are not assessed yet.)
✅ Language (Breton (Q209)) → Record, speaker's language level
select ?record ?recordLabel ?speakerLabel ?languageLabel ?languageLevelLabel
where {
?record prop:P2 entity:Q2 # Filter: P2 'instance of' is Q2 'record' AND P4
; prop:P4 entity:Q209 . # AND P4 'language' is Q209 'Breton'
?record prop:P5 ?speaker . # Assign value: record's P5 'speaker' into ?speaker
?record prop:P4 ?language . # Assign value: record's P4 'language' into ?language
?speaker llp:P4 ?languageStatement . # P4 'language'
?languageStatement llv:P4 ?language . # P4 'language'
?languageStatement llq:P16 ?languageLevel . # P16 'language level'
# Adds labels
SERVICE wikibase:label {
bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en" .
}
} ORDER BY ?languageLabel ?languageLevelLabel
|
|
✅ Language (Marathi (Q34)) → Records of Wikidata concepts with WD Qid (P12)
SELECT ?languageLabel ?recordLabel ?record ?wid
WHERE {
?record prop:P2 entity:Q2 . # Filter: P2 'instance of' is Q2 'record'
?record prop:P4 entity:Q34 . # Filter: P4 'language' is Q34 'Marathi'
?record prop:P4 ?language . # Assign value: record's P4 'language' to variable ?language
?record prop:P12 ?wid . # Assign value: record's P12 'wikidata id' to variable ?wid
# Add labels capability
SERVICE wikibase:label {
bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en" .
}
}
|
|
❌ Language (Marathi (Q34)) → Wikidata Qid(s) → Geo-coordinates
PREFIX wd: <http://www.wikidata.org/entity/>
PREFIX wdt: <http://www.wikidata.org/prop/direct/>
PREFIX ll: <https://lingualibre.org/entity/>
PREFIX llt: <https://lingualibre.org/prop/direct/>
PREFIX lltn: <https://lingualibre.org/prop/direct-normalized/>
select distinct ?record ?transcription ?languageLabel ?wdQid ?wdQidLabel ?wdLabel ?coord
where {
?record llt:P2 ll:Q2 . # Filter: P2 'instance of' is Q2 'record'
?record llt:P4 ll:Q34 . # Filter: record's P4 'language' is Q34 'Marathi'
?record llt:P4 ?language . # Assign value: record's P4 'language' to variable ?language
?record llt:P7 ?transcription . # Assign value: record's P7 'transcription' to variable ?transcription
?record lltn:P12 ?wdQid . # Assign value: record's P12 'wikidata id' to variable ?wikidataItem
SERVICE <https://query.wikidata.org/sparql> {
OPTIONAL { ?wdQid wdt:P625 ?coord . } # Assign value: wikidata item's wd:P625 'coordinates' to variable ?coord
OPTIONAL {
?wdQid rdfs:label ?wdLabel . # Assign value: wikidata item's label to variable ?wikidataLabel
FILTER (LANG(?wdLabel) = "en") . # Filter: default language, else English
}
}
SERVICE wikibase:label {
bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en" .
}
}
|
Result type:record transcription languageLabel wdQid wdQidLabel wdLabel coord Q196212 Tathavade Marathi Q2719024 Q2719024 Tathavade Point(73.74 18.62) Q428904 Jambavade Marathi Q24894740 Q24894740 Jambavade Point(73.85 18.51) Q428900 Dhangavhan Marathi Q24885008 Q24885008 Dhangavhan Point(73.85 18.52) … |