Help:Download datasets/oc

Volum de las donadas - 2022/02
Fichièrs audio	800,000+
Talha mejana	100kB
Talha totala (est.)	80GB

Telecargar los datasets amb un clic

Telecargament per lenga

1. dobrir https://lingualibre.org/datasets/
2. Trapar la vòstra lenga, la convencion de nommage es : {qId}-{iso639-3}-{nom en anglés}.zip
3. Clicar per telecargar
4. Sus lo vòstre terminal, unzip.

Anar al seguent
Referissètz-vos al tutorial concernit dins #See also pels renommages de massa, las conversions de massa o lo debruitage de massa dels audios telecargats.

Aisinas de programacion

Las aisinas çai jos recupèran primièr dins una o mantuna categoria Wikimedia Commons, la lista dels fichièrs audio que contenon.

Tobètz la vòstra cibla

Categorias dins Wikimedia Commons son organizadas coma seguís :

• Commons:Category:Lingua Libre pronunciation by user
• Commons:Category:Lingua Libre pronunciation (per lenga)

Python (actual)

Dependéncias: Python 3.6+

Petscan and Wikiget allows to download about 15,000 audio files per hour.

1. Select your category : see Category:Lingua Libre pronunciation and Category:Lingua Libre pronunciation by user, then find your target category,
2. List target files with Petscan : Given a target category on Commons, provides list of target files. Example.
3. Download target files with Wikiget : downloads targets files.

Comments:

• Successful on November 2021, with 730,000 audio downloaded in 20 hours. Sustained average speed : 10 downloads/sec.
• Some delete files on Commons may cause Wikiget to return an error and pause. The script has to be resumed manually. Occurrence have been reported to be around 1/30,000 files. Fix is underway, support the request on github.
• WikiGet therefore requires a volunteer to supervise the script while running.
• As of December 2021, WikiGet does not support multi-thread downloads. Therefore, to increase the efficiency of the download process it is recommended to run the Python Script on 20-30 terminal windows simultaneously. Each terminal running WikiGet would consume an average of 20 Kb/s.
• WikiGet requires an stable internet connection. Any disruption of 1 second would stop the download process and it requires manual restart of the Python Script.
• Manual for PetScan
• Any question about downloading datasets can be made on the Discord Server of Lingua Libre : https://discord.gg/2WECKUHj

NodeJS (lèu)

Dependencies: git, nodejs, npm.

A WikiapiJS script allows to download target category's files, or a root category, its subcategories and contained files. Downloads about 1,400 audio files per hour.

1. WikiapiJS is the NodeJS / NPM package allowing scripted API calls upon Wikimedia Commons and LinguaLibre.
2. Specific script used to do a given task:
   • Given a category, download all files : https://github.com/hugolpz/WikiapiJS-Eggs/blob/main/wiki-download-many.js
   • Given a root category, list subcategories, download all files: https://github.com/hugolpz/WikiapiJS-Eggs/blob/main/wiki-download_by_root_category-many.js

Comments, as of December 2021:

• Successful on December 2021, with 400 audios downloaded in 16 minutes. Sustained average speed : 0.4 downloads/sec.
• Successfully process single category's files.
• Successfully process root category and subcategories' files, generating ./isocode/ folders.
• Scalability tests for resilience with high amounts requests >500 to 100,000 items is required.
• Performance improvements are under consideration on github.

Python (slow)

Dependencies: python.

CommonsDownloadTool.py is a python script which formerly created datasets for LinguaLibre. It can be hacked and tinkered to your needs. To download all datasets as zips :

• Download scripts :
  • create_datasets.sh - creates CommonsDownloadTool's commands.
  • CommonsDownloadTool/commons_download_tool.py - core script.
• Read them a bit, move them where they fit the best on you computer so they require the minimum of editing
• Edit as needed so the paths are correct, make it work.
• Run create_datasets.sh
• Check if the number of files in the downloaded zips matches the number of files in Commons:Category:Lingua Libre pronunciation

Comments:

• Last ran on February 2021, stopped due to slow speed.
• This script is slow and has been phased out as Lingualibre grown too much.
• The page may gain from some html and styling.
• Proposals go on https://phabricator.wikimedia.org/tag/lingua_libre/ or on the LinguaLibre:Chat room.

Python with UI (Sulochanaviji)

Description to complete, see its github repository.

User:Sulochanaviji coded a Django/Python tool with a HTML/CSS user interface. See its github repository.

Python Script to Download a User's Pronunciations

This script downloads all the pronunciations added by a user into a folder by first querying the Lingua Libre database and then downloading the files from Commons. See its github repository. Languageseeker (talk) 01:57, 24 May 2022 (UTC)

Anki Extension for Lingua Libre

The Lingua Libre and Forvo Addon. It has a number of advanced options to improve search results and can run either as a batch operation or on an individual note.

By default, it first checks Lingua Libre and, if there are no results on Lingua Libre, it then checks Forvo. To run as a pure Lingua Libre extension, you will need to set "disable_Forvo" to True in your configuration section.

Please reports bugs, issues, ideas on github.

Java (not tested)

Dependencies:

sudo apt-get install default-jre    # install Java environment

Usage:

• Open GitHub Wiki-java-tools project page.
• Find the last Imker release.
• Download Imker_vxx.xx.xx.zip archive
• Extract the .zip file
• Run as follow :
  • On Windows : start the .exe file.
  • On Ubuntu, open shell then :

$java -jar imker-cli.jar -o ./myFolder/ -c 'CategoryName'     # Downloads all medias within Wikimedia Commons's category "CategoryName"

Comments :

• Not used yet by any LinguaLibre member. If you do, please share your experience of this tool.

Manual

Imker -- Wikimedia Commons batch downloading tool.

Usage: java -jar imker-cli.jar [options]
  Options:
    --category, -c
       Use the specified Wiki category as download source.
    --domain, -d
       Wiki domain to fetch from
       Default: commons.wikimedia.org
    --file, -f
       Use the specified local file as download source.
  * --outfolder, -o
       The output folder.
    --page, -p
       Use the specified Wiki page as download source.

The download source must be ONE of the following:
 ↳ A Wiki category (Example: --category="Denver, Colorado")
 ↳ A Wiki page (Example: --page="Sandboarding")
 ↳ A local file (Example: --file="Documents/files.txt"; One filename per line!)

See also

• Help:Renaming
• Help:Converting audios
• Help:Embed audio in HTML
• phab:T261519