Help

Difference between revisions of "Download datasets/mk"

Download of Lingualibre's audio datasets allows external reuse of those audios into native or web applications. LinguaLibre's service of periodic generation of dumps is currently staled, volunteer developers are working on it (Jan. 2022). Current, past and future alternatives are documented below. Other tutorials deal with how to clean up the resulting folders and how to rename these files into more practical {language}−{word}.ogg. Be aware of the overall datasize of estimated 40GB for wav format.

(Created page with "Забелешки: * Спсоците пуштени во февруари 2021 г. запреа поради бавноста. * Оваа скрипта е бавна и б...")
(Updating to match new version of source page)
Line 1: Line 1:
{{#Subtitle:{{/Header}}}}
+
{{#Subtitle:{{Help:Download_datasets/Header}}}}
 
<languages/>
 
<languages/>
 
{| class="wikitable right" style="float:right;"
 
{| class="wikitable right" style="float:right;"

Revision as of 22:51, 4 March 2022

Other languages:
Deutsch • ‎English • ‎norsk bokmål • ‎occitan • ‎polski • ‎português do Brasil • ‎svenska • ‎македонски • ‎বাংলা
Податочна големина — 2022/02
Звучни снимки 800,000+
Прос. големина 100kB
Вкуп. големина (проц.) 80GB

Преземање на податочни зборови со стискање

Преземање по јазик:

  1. Open https://lingualibre.org/datasets/
  2. Find your language, naming convention is: {qId}-{iso639-3}-{language_English_name}.zip
  3. Click to download
  4. On your device, unzip.

Пообработка
Refer to the relevant tutorials in #See also to mass rename, mass convert or mass denoise your downloaded audios.

Програмски алатки

The tools below first fetch from one or several Wikimedia Commons categories the list of audio files within them. Some of them allow to filter that list further to focus a single speaker, either by editing their code or by post-processing of the resulting .csv list of audio files. The listed targets are then downloaded at a speed of 500 to 15,000 per hours. Items already present locally and matching the latest Commons version are generally not re-downloaded.

Find your target

Categories on Wikimedia Commons are organized as follow:

Python (current)

Dependencies: Python 3.6+

Petscan and Wikiget allows to download about 15,000 audio files per hour.

  1. Select your category : see Category:Lingua Libre pronunciation and Category:Lingua Libre pronunciation by user, then find your target category,
  2. List target files with Petscan : Given a target category on Commons, provides list of target files. Example.
  3. Download target files with Wikiget : downloads targets files.

Comments:

  • Successful on November 2021, with 730,000 audio downloaded in 20 hours. Sustained average speed : 10 downloads/sec.
  • Some delete files on Commons may cause Wikiget to return an error and pause. The script has to be resumed manually. Occurrence have been reported to be around 1/30,000 files. Fix is underway, support the request on github.
  • WikiGet therefore requires a volunteer to supervise the script while running.
  • As of December 2021, WikiGet does not support multi-thread downloads. Therefore, to increase the efficiency of the download process it is recommended to run the Python Script on 20-30 terminal windows simultaneously. Each terminal running WikiGet would consume an average of 20 Kb/s.
  • WikiGet requires an stable internet connection. Any disruption of 1 second would stop the download process and it requires manual restart of the Python Script.
  • Manual for PetScan
  • Any question about downloading datasets can be made on the Discord Server of Lingua Libre : https://discord.gg/2WECKUHj

NodeJS (наскоро)

Dependencies: git, nodejs, npm.

A WikiapiJS script allows to download target category's files, or a root category, its subcategories and contained files. Downloads about 1,400 audio files per hour.

  1. WikiapiJS is the NodeJS / NPM package allowing scripted API calls upon Wikimedia Commons and LinguaLibre.
  2. Specific script used to do a given task:

Comments, as of December 2021:

  • Successful on December 2021, with 400 audios downloaded in 16 minutes. Sustained average speed : 0.4 downloads/sec.
  • Successfully process single category's files.
  • Successfully process root category and subcategories' files, generating ./isocode/ folders.
  • Scalability tests for resilience with high amounts requests >500 to 100,000 items is required.
  • Performance improvements are under consideration on github.

Python (бавно)

Зависности: python.

CommonsDownloadTool.py is a python script which formerly created datasets for LinguaLibre. It can be hacked and tinkered to your needs. To download all datasets as zips :

Забелешки:

  • Спсоците пуштени во февруари 2021 г. запреа поради бавноста.
  • Оваа скрипта е бавна и беше укината со големиот раст на Lingualibre.
  • Страницата може да се подобри со извесна мера HTML и стилизација.
  • Предлозите одат на https://phabricator.wikimedia.org/tag/lingua_libre/ или во Разговорницата.

Java (неиспробано)

Dependencies:

sudo apt-get install default-jre    # install Java environment

Usage:

  • Open GitHub Wiki-java-tools project page.
  • Find the last Imker release.
  • Download Imker_vxx.xx.xx.zip archive
  • Extract the .zip file
  • Run as follow :
    • On Windows : start the .exe file.
    • On Ubuntu, open shell then :
$java -jar imker-cli.jar -o ./myFolder/ -c 'CategoryName'     # Downloads all medias within Wikimedia Commons's category "CategoryName"

Коментари:

  • Засега не се користи од ниеден член на LinguaLibre. Ако го користите, споделете го вашето искуство со алаткава.

Упатство

Imker -- Wikimedia Commons batch downloading tool.

Usage: java -jar imker-cli.jar [options]
  Options:
    --category, -c
       Use the specified Wiki category as download source.
    --domain, -d
       Wiki domain to fetch from
       Default: commons.wikimedia.org
    --file, -f
       Use the specified local file as download source.
  * --outfolder, -o
       The output folder.
    --page, -p
       Use the specified Wiki page as download source.

The download source must be ONE of the following:
 ↳ A Wiki category (Example: --category=&quot;Denver, Colorado&quot;)
 ↳ A Wiki page (Example: --page=&quot;Sandboarding&quot;)
 ↳ A local file (Example: --file=&quot;Documents/files.txt&quot;; One filename per line!)

Поврзано

Lingua Libre technical helps
Template {{Speakers category}} • {{Recommended lists}} • {{To iso 639-2}} • {{To iso 639-3}} • {{Userbox-records}} • {{Bot steps}}
Audio files How to create a frequency list?Convert files formatsDenoise files with SoXRename and mass rename
Bots Help:BotsLinguaLibre:BotHelp:Log in to Lingua Libre with PywikibotLingua Libre Bot (gh) • OlafbotPamputtBotDragons Bot (gh)
MediaWiki MediaWiki: Help:Documentation opérationelle MediawikiHelp:Database structureHelp:CSSHelp:RenameHelp:OAuthLinguaLibre:User rights (rate limit) • Module:Lingua Libre record & {{Lingua Libre record}}JS scripts: MediaWiki:Common.jsLastAudios.jsSoundLibrary.jsItemsSugar.jsLexemeQueriesGenerator.js (pad) • Sparql2data.js (pad) • LanguagesGallery.js (pad) • Gadgets: Gadget-LinguaImporter.jsGadget-Demo.jsGadget-RecentNonAudio.jsLiLiZip.js
Queries Help:APIsHelp:SPARQLSPARQL (intermediate) (stub) • SPARQL for lexemes (stub) • SPARQL for maintenanceLingualibre:Wikidata (stub) • Help:SPARQL (HAL)
Reuses Help:Download datasetsHelp:Embed audio in HTML
Unstable & tests Help:SPARQL/test
Categories Category:Technical reports