Help
Difference between revisions of "Download datasets/oc"
Download of Lingualibre's audio datasets allows external reuse of those audios into native or web applications. LinguaLibre's service of periodic generation of dumps is currently staled, volunteer developers are working on it (Jan. 2022). Current, past and future alternatives are documented below. Other tutorials deal with how to clean up the resulting folders and how to rename these files into more practical {language}−{word}.ogg. Be aware of the overall datasize of estimated 40GB for wav format.
(Created page with "Dependéncias: Python 3.6+") |
(Created page with "'''Petscan''' e '''Wikiget''' permeton de telecargar mai o mens 15,000 fichièrs audio files per ora. # '''Causir la vòstra categoria :''' veire :commons:Category:Lingua_Li...") |
||
| (2 intermediate revisions by the same user not shown) | |||
| Line 41: | Line 41: | ||
Dependéncias: Python 3.6+ | Dependéncias: Python 3.6+ | ||
| − | '''Petscan''' | + | '''Petscan''' e '''Wikiget''' permeton de telecargar mai o mens 15,000 fichièrs audio files per ora. |
| − | # ''' | + | # '''Causir la vòstra categoria :''' veire [[:commons:Category:Lingua_Libre_pronunciation|Category:Lingua Libre pronunciation]] e [[:commons:Category:Lingua Libre pronunciation by user|Category:Lingua Libre pronunciation by user]], puèi trobètz la vòstra categoria, |
| − | # ''' | + | # '''listar los fichièrs ciblats amb [https://petscan.wmflabs.org Petscan] :''' Una categoria ciblada sus Commons presenta una lista de fichièrs . [https://petscan.wmflabs.org/?&cb_labels_yes_l=1&cb_labels_no_l=1&edits%5Banons%5D=both&interface_language=en&edits%5Bflagged%5D=both&categories=Lingua%20Libre%20pronunciation-cmn&cb_labels_any_l=1&ns%5B0%5D=1&project=wikimedia&since_rev0=&search_max_results=500&edits%5Bbots%5D=both&ns%5B6%5D=1&language=commons&search_query= Example]. |
| − | # ''' | + | # '''Telecargar los fichièrs ciblats amb [https://pypi.org/project/wikiget/ Wikiget] :''' telecarga los fichièrs. |
Comments: | Comments: | ||
| Line 55: | Line 55: | ||
* Any question about downloading datasets can be made on the Discord Server of Lingua Libre : https://discord.gg/2WECKUHj | * Any question about downloading datasets can be made on the Discord Server of Lingua Libre : https://discord.gg/2WECKUHj | ||
| − | === NodeJS ( | + | === NodeJS (lèu) === |
| − | + | Dependéncias: git, nodejs, npm. | |
A '''WikiapiJS''' script allows to download target category's files, or a root category, its subcategories and contained files. Downloads about 1,400 audio files per hour. | A '''WikiapiJS''' script allows to download target category's files, or a root category, its subcategories and contained files. Downloads about 1,400 audio files per hour. | ||
Latest revision as of 13:36, 8 October 2023
| Volum de las donadas - 2022/02 | |
|---|---|
| Fichièrs audio | 800,000+ |
| Talha mejana | 100kB |
| Talha totala (est.) | 80GB |
Telecargar los datasets amb un clic
Telecargament per lenga
- dobrir https://lingualibre.org/datasets/
- Trapar la vòstra lenga, la convencion de nommage es :
{qId}-{iso639-3}-{nom en anglés}.zip - Clicar per telecargar
- Sus lo vòstre terminal, unzip.
Anar al seguent
Referissètz-vos al tutorial concernit dins #See also pels renommages de massa, las conversions de massa o lo debruitage de massa dels audios telecargats.
Aisinas de programacion
Las aisinas çai jos recupèran primièr dins una o mantuna categoria Wikimedia Commons, la lista dels fichièrs audio que contenon.
Tobètz la vòstra cibla
Categorias dins Wikimedia Commons son organizadas coma seguís :
- Commons:Category:Lingua Libre pronunciation by user
- Commons:Category:Lingua Libre pronunciation (per lenga)
Python (actual)
Dependéncias: Python 3.6+
Petscan e Wikiget permeton de telecargar mai o mens 15,000 fichièrs audio files per ora.
- Causir la vòstra categoria : veire Category:Lingua Libre pronunciation e Category:Lingua Libre pronunciation by user, puèi trobètz la vòstra categoria,
- listar los fichièrs ciblats amb Petscan : Una categoria ciblada sus Commons presenta una lista de fichièrs . Example.
- Telecargar los fichièrs ciblats amb Wikiget : telecarga los fichièrs.
Comments:
- Successful on November 2021, with 730,000 audio downloaded in 20 hours. Sustained average speed : 10 downloads/sec.
- Some delete files on Commons may cause Wikiget to return an error and pause. The script has to be resumed manually. Occurrence have been reported to be around 1/30,000 files. Fix is underway, support the request on github.
- WikiGet therefore requires a volunteer to supervise the script while running.
- As of December 2021, WikiGet does not support multi-thread downloads. Therefore, to increase the efficiency of the download process it is recommended to run the Python Script on 20-30 terminal windows simultaneously. Each terminal running WikiGet would consume an average of 20 Kb/s.
- WikiGet requires an stable internet connection. Any disruption of 1 second would stop the download process and it requires manual restart of the Python Script.
- Manual for PetScan
- Any question about downloading datasets can be made on the Discord Server of Lingua Libre : https://discord.gg/2WECKUHj
NodeJS (lèu)
Dependéncias: git, nodejs, npm.
A WikiapiJS script allows to download target category's files, or a root category, its subcategories and contained files. Downloads about 1,400 audio files per hour.
- WikiapiJS is the NodeJS / NPM package allowing scripted API calls upon Wikimedia Commons and LinguaLibre.
- Specific script used to do a given task:
- Given a category, download all files : https://github.com/hugolpz/WikiapiJS-Eggs/blob/main/wiki-download-many.js
- Given a root category, list subcategories, download all files: https://github.com/hugolpz/WikiapiJS-Eggs/blob/main/wiki-download_by_root_category-many.js
Comments, as of December 2021:
- Successful on December 2021, with 400 audios downloaded in 16 minutes. Sustained average speed : 0.4 downloads/sec.
- Successfully process single category's files.
- Successfully process root category and subcategories' files, generating ./isocode/ folders.
- Scalability tests for resilience with high amounts requests >500 to 100,000 items is required.
- Performance improvements are under consideration on github.
Python (slow)
Dependencies: python.
CommonsDownloadTool.py is a python script which formerly created datasets for LinguaLibre. It can be hacked and tinkered to your needs. To download all datasets as zips :
- Download scripts :
- create_datasets.sh - creates CommonsDownloadTool's commands.
- CommonsDownloadTool/commons_download_tool.py - core script.
- Read them a bit, move them where they fit the best on you computer so they require the minimum of editing
- Edit as needed so the paths are correct, make it work.
- Run
create_datasets.sh - Check if the number of files in the downloaded zips matches the number of files in Commons:Category:Lingua Libre pronunciation
Comments:
- Last ran on February 2021, stopped due to slow speed.
- This script is slow and has been phased out as Lingualibre grown too much.
- The page may gain from some html and styling.
- Proposals go on https://phabricator.wikimedia.org/tag/lingua_libre/ or on the LinguaLibre:Chat room.
- Description to complete, see its github repository.
User:Sulochanaviji coded a Django/Python tool with a HTML/CSS user interface. See its github repository.
Python Script to Download a User's Pronunciations
This script downloads all the pronunciations added by a user into a folder by first querying the Lingua Libre database and then downloading the files from Commons. See its github repository. Languageseeker (talk) 01:57, 24 May 2022 (UTC)
Anki Extension for Lingua Libre
The Lingua Libre and Forvo Addon. It has a number of advanced options to improve search results and can run either as a batch operation or on an individual note.
By default, it first checks Lingua Libre and, if there are no results on Lingua Libre, it then checks Forvo. To run as a pure Lingua Libre extension, you will need to set "disable_Forvo" to True in your configuration section.
Please reports bugs, issues, ideas on github.
Java (not tested)
Dependencies:
sudo apt-get install default-jre # install Java environment
Usage:
- Open GitHub Wiki-java-tools project page.
- Find the last
Imkerrelease. - Download Imker_vxx.xx.xx.zip archive
- Extract the .zip file
- Run as follow :
- On Windows : start the .exe file.
- On Ubuntu, open shell then :
$java -jar imker-cli.jar -o ./myFolder/ -c 'CategoryName' # Downloads all medias within Wikimedia Commons's category "CategoryName"
Comments :
- Not used yet by any LinguaLibre member. If you do, please share your experience of this tool.
Manual
Imker -- Wikimedia Commons batch downloading tool.
Usage: java -jar imker-cli.jar [options]
Options:
--category, -c
Use the specified Wiki category as download source.
--domain, -d
Wiki domain to fetch from
Default: commons.wikimedia.org
--file, -f
Use the specified local file as download source.
* --outfolder, -o
The output folder.
--page, -p
Use the specified Wiki page as download source.
The download source must be ONE of the following:
↳ A Wiki category (Example: --category="Denver, Colorado")
↳ A Wiki page (Example: --page="Sandboarding")
↳ A local file (Example: --file="Documents/files.txt"; One filename per line!)