LinguaLibre

Chat room

Revision as of 02:31, 5 December 2020 by WikiLucas00 (talk | contribs) (→‎Stats page November 2020: Answering to Pamputt)

Welcome to the Chat room! Place used to discuss any and all aspects of Lingua Libre: the project itself, discussions of the operations, policy and proposals, technical issues, etc. Other forums include for code-oriented issues, . Feel free to participate in any language you want to.


Chatroom FAQ

  • How to add missing languages ?
    • Administrators can add new languages, they do so within few days. For users, please provide your language's iso-639-3 code + link to the en.wikipedia.org's article. Optional infos are the common English name and wikidata IQ. For more, see Help:Add a new language.
  • How to keep my wikimedia project up to date ?
  • What IRL event.s are coming ? When ? Where ?
  • How to archive sections which have been answered ?
    • After reviewing the section, add '{{done}} -- can be closed ~~~~' to the top of the section. After few days to 2 weeks, move the section's code to [[LinguaLibre:Chat_room/Archives/year]].

Archives

Archives
Archives by year:


speedy et / ou delete

Bonjour,
Il peut arriver qu'un Élément Qs ne sert plus (enregistrement impropre, page WM différente, titrage incorrect, etc). On peut l'effacer de Commons, mais il subsiste ici. Pour commencer, je propose la création d'une page dédiée à la suppression, avec un/des template/s speedy et/ou delete.
Une de mes créations ne me convenait pas, alors j'ai effacé le fichier sur Commons pour le remplacer par un autre avec mes propres outils pour remettre tout en ordre sur LL. Bref le temps de la nouvelle création, Q309179 avait disparu. Pour speedy et delete, qu'en dites-vous ? Des commentaires ? —Eihel (talk) 17:35, 29 May 2020 (UTC) ps. J'ai déjà ajouté un template. Voir LinguaLibre:Administrators' noticeboard

Salut Eihel, oui pourquoi pas. À noter cependant que si une prononciation est incorrecte, réenregistrer le mot uploadera la nouvelle prononciation sur Commons à la place de l'ancien enregistrement. Par ailleurs vu que les noms des fichiers sont générés automatiquement par Lingua Libre, les cas à traiter devraient être relativement rares. Pamputt (talk) 12:46, 31 May 2020 (UTC)
De manière générale nous avons un point faible dans la gestion dynamique des audios : consultation agreable, renommage, suppression, etc. C'est mentionné (consultation agréable) plus haut dans la comparaison avec Shtooka. Yug (talk) 10:24, 3 June 2020 (UTC)

Discussion

Hi Yug, I guess it should be better to open tickets on Phabricator to keep track of all theses issues and be able to discuss each one more easiyl (structured way). Pamputt (talk) 12:51, 31 May 2020 (UTC)

Thanks Pamputt :) Yug (talk) 07:51, 1 June 2020 (UTC)
+1 to pamputt, phabricator is more appropriate for that for advanced users. — 0x010C ~talk~ 14:36, 1 June 2020 (UTC)

Bugs

Enregistrements accélérés

Bonjour, Mes enregistrements du jour ont été accélérés. Heureusement, je me suis vite rendu compte. Quelques exemples : Q332977 Q332978 Q332979 Q332980 Q332981 Q332982.

PS : Le lien "Commencer une nouvelle discussion" ci-dessus n'a pas l'air de fonctionner.

DSwissK (talk) 08:36, 28 June 2020 (UTC)

Salut DSwissK, problème étrange. J'ai ouvert un ticket sur Phabricator à ce sujet. J'en ai également ouvert un autre à propos de lien « commencer une nouvelle discussion » car je n'ai pas trouvé comment le corriger moi-même. Pamputt (talk) 17:40, 29 June 2020 (UTC)
@DSwissK & Pamputt I got the same feedbacks of speeded up audios from Luilui6666 for Cantonese, today. Yug (talk) 16:24, 16 July 2020 (UTC)
Contributions > Example (corrupted): Q338365 Yug (talk) 16:42, 16 July 2020 (UTC)
Should we review and remove all the bad audios, so it become easier to re-record ? And where should we remove them, here or on Commons ? Yug (talk) 17:44, 16 July 2020 (UTC)
@Yug We can list such items here. Pamputt (talk) 07:44, 18 July 2020 (UTC)

Accès impossible au Wizard dans l'interface en occitan

Bonjour, J'ai toujours un problème pour enregister quand l'interface est en occitan. Je dois faire un atelier public cet été et je suis obligée de passer en français.

  • Si je clique sur le bouton d'enregistrement en haut de la page, j'ai l'erreur suivante :

Fatal error: Maximum execution time of 30 seconds exceeded in /home/www/lingualibre.org/includes/cache/MessageCache.php on line 812

  • Si je clique sur le bouton d'enregistrement en bas de la page d'accueil, j'ai l'erreur suivante :

Fatal error: Maximum execution time of 30 seconds exceeded in /home/www/lingualibre.org/languages/Language.php on line 198 Guilhelma

J'ai ajouté les nouveaux messages d'erreur au ticket Phabricator qui parle des problèmes avec la version en occitan. Pamputt (talk) 08:55, 19 July 2020 (UTC)
@Guilhelma , is this bug confirmed and reoccuring ? Yug (talk) 18:21, 22 September 2020 (UTC)
@Guilhelma , est-ce que ce bug est confirmé et continue de vous géner ? Yug (talk) 18:21, 22 September 2020 (UTC)

Yes, the bug is confirmed Fatal error: Maximum execution time of 30 seconds exceeded in /home/www/lingualibre.org/languages/Language.php on line 4422Guilhelma

Adding list from Wikidata

Hello. It seems the interface has changed since i last used it and i cannot see how to create a word list from a Wikidata query. Could someone tell me the best way of doing this? thanks Jason.nlw (talk) 08:49, 17 August 2020 (UTC)

Hi Jason.nlw, as far as I remember it has never been possible to generate such list but I may be wrong. I opened a feature request on Phabricator. Pamputt (talk) 06:26, 18 August 2020 (UTC)
  • The only workaround is now: run a query--> download the label list as csv-->copy the column-->create a local list on LiLi. This won't remember and link the Wikdiata items though, and the bot won't work either. You can only record the words. --টিটো দত্ত (Titodutta) (কথা) 00:42, 19 September 2020 (UTC)

Datasets out of date

Hello. It seems that the datasets page, although it claims to run every 2 days, is completely out of date: all the available zips are from April 2020 or November 2019 (and the full zip from May 2019). Is this a known problem? Is there a plan to address it? Julien Baley (talk) 23:17, 27 August 2020 (UTC)

Indeed, it seems to have an issue with the dataset updating. I opened a Phabricator ticket about this issue. Pamputt (talk) 18:24, 28 August 2020 (UTC)

Pages translation

I would like to be able to mark pages for translation, but I don't have the user rights (pagetranslation) to do so. This rights are restricted to sysops (see Special:ListGroupRights). Should we create a translation administrator user group? Are there plans for creating a page like this in the future? — WikiLucas (🖋️) 03:08, 13 September 2020 (UTC)

@WikiLucas00 indeed, currently there are not a lot of different user rights available here (bot, admin, bureaucrat). If you think we should have more, please feel free to open a ticket asking for that on Phabricator. About, LinguaLibre:Requests for rights, the same, feel free to create and initialize this page :D Pamputt (talk) 17:06, 13 September 2020 (UTC)
@Pamputt I created a task on Phabricator. Let's first see how it evolves before creating a Request page. — WikiLucas (🖋️) 19:42, 14 September 2020 (UTC)

New admins ?

See also Special:ListUsers/sysop

@Pamputt, WikiLucas00, Titodutta, & Lyokoï I think it would be nice to make WikiLucas an admin. We are a micro-wiki, WikiLucas has proven to be active and knowledgeable, all lights are green to make him a sysop. I would also encourage to have one or two Indian admins. Indian users are the second largest community here, they bring new insights to the projects, let's empower them properly. Any idea who among this later community would need the admin tools ? (page translate, page deletion, language import) Yug (talk) 05:09, 23 September 2020 (UTC)

Checking over Special:ListUsers/sysop I also notice :

  • Bureaucrats: 0x010C is taking a year off; Xenophon is a WMfr staff with bureacrat right for security reasons but barely active here; GrandCelinien... I barely crossed him; it leaves Pamputt as the single active bureaucrat. Not enough. We need at least 3 active bureaucrats. I propose to promote Lyokoï to bureacrat if he is ok. He is a regular contributor and solid bet. We also will need someone on the Indian/Asian side soon. Bureacrats mainly can gives users more rights, such as admin status. It's not much but when we need it we need it, and relying on one single Bureaucrat is no a good practice.
    I’m OK to be a bureaucrat. If you want it, I see no problem. Lyokoï (talk) 16:36, 6 October 2020 (UTC)
  • Admins: WikiLucas is an obvious candidate, he bumped into limitations (page translation right above). I see about 3 Indian contributors quite engaged here, could we promote one ? Yug (talk) 05:24, 23 September 2020 (UTC)
@Yug I am clearly not opposed to have more bureaucrat or admin, neither to have some Indian contributors among them. That being said, I don't think we're in a hurry (the Lingua Libre community is not very active at the moment). I prefer to take some time to give the rights to people involved in Lingua Libre, so that we can be sure that they will use their rights for at least a few months. Yet, if someone requests admin or bureaucrat right, just ask (there is no bureaucratic procedure yet here). Pamputt (talk) 06:25, 23 September 2020 (UTC)
@Yug, Pamputt, Lyokoï, & Titodutta I agree with Yug, I would be more valuable to the project as an admin. As Pamputt has pointed out, the project is not very active for the moment, but in the light of future events -- for instance the training course I will be giving this month with Emma Vadillo to the alumni of the INaLCO in Paris (being able to quickly delete the potential mistakes of the learners would be worthy), or the possible nomination of the project to the Coolest Tool Awards), its outreach will grow, so will the community. — WikiLucas (🖋️) 17:12, 3 October 2020 (UTC)
@Pamputt per the request just above let's grant WikiLucas00 adminship, he is one of the most active here anyway, and admin is just an active user with a toolbox to add languages, block users, add translatable pages.
As for my general argument, I consider that being a small wiki and with most admins/bureaucrats rarely passing by or via occasional sprints (my case), we therefor need a high ratio of admins/bureaucrates so there is always one around and checking upon the Special:RecentChanges.
Last, as for Bureaucrats, we are failing the Bus factor/Bus test : it's a organizational risk we should not fail, ever. I recommend adding one Bureaucrat for sure. Keeping Xenophon as far away backup. I would also recommend to keep the door open for one more, preferably from the East-Asian community (different timezone, human network, strategic opportunity, etc.).
We also need to recruit an admin on Commons able to do mass delete when we provide a list of files. Do we have this already ? @VIGNERON  ? Yug (talk) 19:23, 3 October 2020 (UTC)
Ok, I granted WikiLucas00 as admin. For Commons, it would be indeed interested to have such profile. Pamputt (talk) 21:04, 4 October 2020 (UTC)
@Yug & Pamputt Thank you for your trust. — WikiLucas (🖋️) 21:25, 4 October 2020 (UTC)
Ok for me to give admin statut to WikiLucas. Lyokoï (talk) 16:26, 6 October 2020 (UTC)

@Yug I'm a bit late but yes, obivously, if you need me as a Commons admin, you're welcome. Cheers, VIGNERON (talk) 17:29, 4 December 2020 (UTC)

0x010C year offgrid : preparations

Hello folks, 0x010C anounced by email his soon to be departure from the project for a year+ off grid (he will tell more here if he wish to ;) ). We can't fully replace our benevolent lead developer. But could we brainstorm to see where he was active, and how to best fill the gap ? I'am kick starting this table but I have a biais since I don't know every task 0x010C was taking on nor do I know all active users on the project and your full skillsets. Please help us to fill in the gaps. 0x010C will be available between Oct. 15th and October 30th to pass some know how to who wish to. Let's prepare our questions properly for this transition. Yug (talk) 14:01, 21 September 2020 (UTC)

Critical Task / Aspect Requirement ? Who else knows ?
Who wish to learn ?		 Satisfy
requirements (%)
high Server maintenance 1. Has back-end sysop knowledge
2. Has access rights to WMFr server (see WMFr sysop).
3. Knows how to maintain/restarts scripts and processes.
4. Knows how to restart NGINX server		 NEEDS SOMEONE 0% so far
high Edit recording wizard JS library 1. Has advanced javascript know how.
2. Knows where js code is {link to js repository}
3. Edit and test js code locally .
4. Has access rights to push.		 None or Nicolas Vion ? 0% or 60%
high Deploying fix into production 1. Has back-end sysop knowledge
2. Has access rights to server.
3. Has access rights to pull corrected code.
4. Knows how to rebuild/deploy.		 WMFR's new server sysop ? (to be hired) Uncertain
high Add new language to LinguaLibre 1. Has administrator user rights
2. Can read tutorial {add tutorial link here}		 Has done it: Pamputt, Lyokoy, Yug, ... 90%
high Read Phabricator task, fix code 1. Has background knowledge to understand bug description.
2. Edit code, test locally.
4. Has access rights to push.		 No replacement for real code, code deployment.
Replacements available for CSS, wiki content fixes.		 30%
medium Assign user rights 1. Has bureaucrats status
2. Know how to assign new user rights.		 Special:ListUsers/sysop: 0x010C, GrandCelinien, Pamputt, Xenophôn.
Few more wouldn't hurt to counter unequal activity levels.		 100%
medium Github repository manager 1. Have access to repository {link here}.
2. Has owner status.
3. Can manage userrights		 Has basic understanding: Yug, Pamputt... 70%
medium Create tasks on Phabricator 1. Have account on phabricator.
2. Has background knowledge to write sharp bug / task description.
3. Know to manage Phabricator tasks		 Pamputt, Lyokoï, Yug, ... 70%
low - assumed very stable LinguaLibre -> Wikimedia Commons API communication 1. Knows NodeJS (?) scripts.
Well documented on mw:API.
2. Knows where to edit existing nodes scripts.
3. Can test locally.
4. Has access rights to push.		 Has basic understanding: Yug, others ? 10%
low Update site's CSS 1. Edit MediaWiki:Common.css (hack)
or
1. Edit {git repository page}.		 Has basic understanding of the hack way: Yug, ... 30%
Please help describe where we need help to take on 0x010C's skills. This year long's departure is the opportunity to us to increase our know how in these various fields. See also: Commons:Category:Lingua_Libre.
2018-12 Lingua Libre webrequest flow.png
Thanks for that overview. For now, the worst is there is no developer at all. without that, I think, we can only list all the issues we encounter on Phabricator waiting one is hired by WMFr or other. Considering all the tickets opened on Phabricator, a new developer can be busy for several month, especially if he/she does not very well the project as 0x010C does. Pamputt (talk) 15:56, 21 September 2020 (UTC)
Yes. I'am quite worry about the speeded-up-audios and add-language bugs. The first literally throw to the trash hour-long efforts and pollute existing audios datasets, really bad. The second prevents diversity growth. It would be good to mount an emergency budget to pay 0x010C to fix these 2 critical phabricator issues before he leaves. Yug (talk) 16:42, 21 September 2020 (UTC)

Feedbacks from 0x010C

Ok! So, I just got a correct phone-call with 0x010C! 0x has great projects ahead it's awesome, so happy.
Also, he pointed out the following points:

  • 0x010C will pass by LL's discussion page to add pointers to the table above and answer questions.
    • This will occurs in late October.
  • 0x010C will not be able to inspect the speeding-up bug : we need to find an alternative to fix this.
  • 0x010C underlined the most critical need : a server sysop, able to do server maintenance and restarts processes. Some peripheral routines such as the SPARQL counter occasionally fails and needs to be restarted manually.

First, on the speeding-up bug, we therefor have 2 ways to push forward :

  1. Corner the bug. We currently suspect it to be linked to Chrome. Test more. With various browsers. Record 30 words, listen to them, then report results (see section below)
  2. Hire a freelance asap. The bug is suspected to be within https://github.com/lingua-libre/LinguaRecorder . There, the last 2 modifications and prime suspects are:

Secondly, on the critical server's maintenance, Wikimedia France's server sysop is also leaving soon and the next one is not yet identified. So we may need to send a far reaching call for a server sysop's help, either a volunteer, or the sysop of some friendly chapter ? (UK? DE? IT?) It could equally be the opportunity to open up deeply to non-French member. LL is 3 years old yet most of the LL's admins are French. Not smart. I would especially encourage to open up toward the Indian community, Odia, Tamil, who have been quite active and with high quality feedbacks. They themselves developed a shell-based audio recording tools few years back, so there is there a culture which value orality and acts to protect it. The call shouldn't be limited to this community. The Mediawiki community (techs), Commons, Wikidata, could have some relevant volunters with the needed skillsets. Would be good if we could write together a call to find a volunteer server sysop, together with a call for more diverse contributions to LinguaLibre.

While we can co-write this call asap, I think we should first corner the speeding-up bug before asking people to join in and contribute. So as long as the speeding-up bug is unsolved, we can only call for a server sysop, not for more contributors.

Meanwhile, please update the table above as you feel suit. I'am sure I forgot a lot of things. Yug (talk) 17:52, 22 September 2020 (UTC)

Speeding-up bug : call for testers

Please, we need testers to corner that nasty bug ! Could you test recording with various browsers ? Go to Special:RecordWizard, pick a random language, record 30 words, listen to them, [DO NOT UPLOAD], then report here the resulting pentad ;)

  • Test list (suggested) : List:Kur/Test (10 words)
  • Username : yourusername
  • Speeding bug : true|false
  • Web browser : name-version
  • OS : name-version
  • Microphone : internal|external

Yug (talk) 18:00, 22 September 2020 (UTC) PS: @DSwissK

Tester username Success rate Web browser + version OS + version Microphone type Comments
DSwissK 0% (speeding bug occurs at every recording) Google Chrome 84.0.4183.127 Android 9 internal List:Fra/Dico des Ados (3k+ words)
ClasseNoes 0% (speeding bug occurs at every recording) Google Chrome ChromeOS internal List:Fra/Dico des Ados (3k+ words)
Luilui6666 0% (speeding bug occurs
on one session)		 Google Chrome MacOS 126 (100%) audios of the 04:5*am upload batch are corrupted. This session contains longer than average phrases. Session before and after are ok.
Yug 100% (no bug) Google Chrome 85.0.4183.121 (64-bit) Ubuntu 20.04 internal Observed at step 5. Review before Publish. List:Kur/Test (10 words)
Yug 100% (no bug) Google Chrome 85.0.4183.121 (64-bit) Ubuntu 20.04 external Observed at step 5. Review before Publish. List:Kur/Test (10 words)
Yug 100% (no bug) Chromium 85.0.4183.121 (64-bit) Ubuntu 20.04 internal Observed at step 5. Review before Publish. List:Kur/Test (10 words)
Yug 100% (no bug) Chromium 85.0.4183.121 (64-bit) Ubuntu 20.04 external Observed at step 5. Review before Publish. List:Kur/Test (10 words)
Yug 100% (no bug) Kiwi 77.0.3865.92 (2020-08-15) Android 9 external Observed at step 5. Review before Publish. List:Kur/Test (10 words)
Yug 100% (no bug) Kiwi 77.0.3865.92 (2020-08-15) Android 9 internal Observed at step 5. Review before Publish. List:Kur/Test (10 words)
Yug 100% (no bug) Chrome 80.0.3987.99 Android 9 external Observed at step 5. Review before Publish. List:Kur/Test (10 words)
Yug 100% (no bug) Chrome 80.0.3987.99 Android 9 internal Observed at step 5. Review before Publish. List:Kur/Test (10 words)
Pamputt 100% (no bug) Firefox 78.3.0 esr Mageia Linux 7 internal
DSwissK 100% (no bug) Google Chrome 86.0.4240.75 Android 10 internal List:Fra/Dico des Ados (3k+ words)
DSwissK 100% (no bug) Google Chrome 86.0.4240.99 Android 10 external List:Fra/Dico des Ados/alea (54 words)
DSwissK 0% (all bugged) Google Chrome 86.0.4240.198 Android 10 external List:Fra/Dico des Ados (3k+ words)
DSwissK 100% (no bug) Google Chrome 86.0.4240.198 Android 10 external List:Fra/Dico des Ados/alea (30 words)
<add yourself>

@ClasseNoes & DSwissK It works fine with me on both Chrome and Chromium. Did you and could you try with other OSes ? Yug (talk) 14:59, 6 October 2020 (UTC)

@Titodutta, Lyokoï, & Pamputt , we need help to corner the speed up bug by doing more devices testings. Could you help ? INALCO workshop is on Oct. 17th. Yug (talk) 17:32, 6 October 2020 (UTC)
I've added the information relative to my configuration. Another explanation may be the internet quality (bandwith, latency, etc). In the case of micro-cuts, some software accelerates the voice to make up for the delay. So maybe it could happen more often with 3G connection instead of optical fibers. Pamputt (talk) 19:50, 6 October 2020 (UTC)
0x010C was suggesting a purely client-side issue. The recording into audio data are done client-side. I don't see clear pattern emerge so far. Yug (talk) 21:52, 6 October 2020 (UTC)
It's definitely NOT internet quality for ClasseNoes and myself. We had a good connexion (not over mobile) and the bug occured at several days of difference. DSwissK (talk) 06:19, 8 October 2020 (UTC)
Current assessment: 3 out of 3 of our users with speeding bug used Google Chrome (v84?), on Android 9, ChromeOS, MacOS. The best lead we have so far is a Google Chrome recording API implementation-related, either due to a recent Google Chrome update, or to recent LinguaLibre JS's update done by 0x010C around May. (See above to link to suspected JS code). I took a quick (3mins) look at "Google Chrome + audio recording + speed bug" but nothing conclusive.
More tests?: Could you DSwissK & ClasseNoes* test again on the same devices (hardware, OS) but with different web browser.
@ClasseNoes: could you check your exact Google Chrome's version ?
@DSwissK, after retesting on Android 9 Chrome v.84, do you have the possibility to update so to test Android 9 Chrome to v.85 ?Android 9 Chrome to v.85 works for me on small test lists (~10 items). Could you also comment more : does this speeding bug show up EACH time you use this Android 9 Chrome v.84 pair ? Is there a saturation effect with longer lists ?
*: Luilui6666 is a student who did a paid recording sprint on Cantonese and moved on, he volutarily helped a lot already with the `ratelimit` bug, I can't request further free help from this side. PS: 270€ for 9h and 5000 audios despite the ratelimit bug. Really fruitful experiment! Worth it to boost a language. Yug (talk) 12:18, 8 October 2020 (UTC)
I was thinking maybe it was because of the huge list I'm using (List:Fra/Dico des Ados) but no, it works fine (see last row) on last Chrome version (and Android 10 that I flashed this week-end on the same smartphone). DSwissK (talk) 07:23, 11 October 2020 (UTC)
@DSwissK I suspect Google Chrome v84 to be the issue. But we have not enough details to be conclusive. Yug (talk) 08:49, 15 October 2020 (UTC)
After verification, Luilui6666 had both corrupted batch and non-corrupted batch of audios about 30mins appart. Recordings ok before and after the corrupted session. My Chrome-centered hypothesis is challenged. Must be something else. Yug (talk) 09:45, 17 October 2020 (UTC)
Yug, how did it go on Oct 17 ? Did you encounter that problem with some users ? DSwissK (talk) 11:10, 20 October 2020 (UTC)
@Eavq, WikiLucas00, DSwissK, & Nicolas Lopez de Silanes WMFr Hello DSwissK. I was not there, I'am too far off (near Spain, 800km). Need to ask WikiLucas, Eavq and Nicolas. See also LinguaLibre:Formations_CCWL. Yug (talk) 15:16, 22 October 2020 (UTC)

Lingua Libre Story for September 2020

This is not an official story or newsletter. This is an attempt by the project user(s) to share some updates about the program. There might be more stories which we have missed.

September 2020 was an eventful month and we have seen a lot of activities of uploading new content and also around project-related discussion. Here are some of the best stories from September 2020.

  • 300,000 files: On 10 September 2020 we completed 300,000 pronunciation uploads. After the launch in August 2018, the first 100,000 files were uploaded in April 2019, and the milestone of 200,000 files was reached on January 2020. As of 30 September 2020 there are 366 speakers at this project working in 92 languages.
  • Maximum number of pronunciations in a month: In September 2020, 23,209 files were uploaded. This is the maximum number of files uploaded ever in a particular calendar month (earlier it was 22,963 files in June 2020, and 22,293 files in May 2019).
  • Indian language in top 3 list: This month Bengali language came into the top three languages by the number of files uploaded using Lingua Libre. This is possibly the first time a non-European/Indian language came into the top three most-uploaded languages on the project. As of 30 September there were 26,757 files in Bengali (the top two languages by file count were French: 164,626 files and Esperanto: 28,100)
  • Project chat: Several discussion started on the Chat room, such as Bug testing (you may help), Technical preparations etc.
  • Coming: 1) Oct. 17th's workshop at INALCO University, Paris. This University teach about 105 languages. 2) In late October, User:0x010C willing to share server's know-how before year-long departure off-grid.

That's it. Have a good time. --টিটো দত্ত (Titodutta) (কথা) 16:30, 1 October 2020 (UTC)

Thank you User:Titodutta, it's an interesting format. We can also think of it as collaborative news-letter, edited here, then shareable to our networks. I added a "Coming" section. Yug (talk) 20:31, 4 October 2020 (UTC)

English label and non-English label

Most probably I did not notice this earlier Q389651. Label: this is not English, this is Bengali. My language is set to Bengali as well. Sad thing is: this affects many Bengali files, if not all. I also saw one of the recent uploads in other language: Esperanto: Q389566. For Bengali, and several other languages the script is completely different. (a whole lot of bot work I think). Opinion? --টিটো দত্ত (Titodutta) (কথা) 22:04, 4 October 2020 (UTC)

@Titodutta not sure if this is really a bug. This behaviour comes from the fact Lingua Libre uses Wikibase to handle its own items. And Wikibase allows as many labels as there are languages but actually we do not need any label on Lingua Libre. So, by default it is always English. That's said, I understand it can be weird for some people, so I think the label should be the word that has been recording in English and in the language of the word so that it can be displayed as it when we you use Lingua Libre in your mother tongue. Or maybe it could be the same label for all languages. Anyway, except it is a bit strange, it is not a big deal because these labels are not used by any one. Pamputt (talk) 19:41, 6 October 2020 (UTC)

Other bugs from India

Hi কথা, I was happy to meet you tonight even if it was short. About the bugs you discussed I have created T264790. There is also the problem with the labels discussed above. You talk about a problem about duplicates in the word list but I am not sure I have understood correctly because I was not able to reproduce. So could you open a bug report on Phrabricator to describe what is wrong? If you are uncomfortable with Phabricator, you can describe the problem here and I will open the ticket over there. And there was also another point but I have forgotten it so could you kindly remind me? Pamputt (talk) 19:45, 6 October 2020 (UTC)

  • The 4 bugs we discussed today:
a) Special pages showing errors, you have explained it above.
Check-green.svg Done T264790.
b)Post file move error on Wikimedia Commons: it has 2 types of problems: on Commons, after moved the files are not displayed, b) "Remove words already recorded" don't count those words, so if LL-...Hello.wav is moved to LL-...Bonjour.wav, LinguaLibre does not understand Bonjour is already recorded, and asks to record again.
This is currently an issue. Ideally, recording should not be renamed manually because it happens what you described. The problem comes rom the fact the Commons database and the Lingua Libre Wikibase are not connected. The workaround is to modify manually the Lingua Libre item corresponding to the recording once renaming has been done on Commons.
If Wikimedia France finds some money, a tool allowing to rename and to apply the needed changes in Lingua Libre could be developed. See T264789 for a brief overview (you should develop further what we would like this tool be able to do).
c) some words are being eliminated: I'll try to explain this: a particular word, such as "Paris", when I try to generate from a category from Wikipedia, I get this word and record it. Then I try another option: "Nearby" to generate words, and there also I get the same word. Now, possibly LiLi sometimes fails to understand the word is already recorded. This is not applicable for all files, I have seen this in 3-4 files.
Let me give a clear example, see this edit. This is pretty clear I got the word from Wikipedia. The immediate next edit I used "Nearby" to get a list.
I have checked it for words. I once I can find more samples, I'll report a bug.
Hard to debug. I tried several times with several words and I was not able to reproduce this bug. Are you sure you clicked on "Remove words already recorded"? So not sure opening a bug report is very useful until you have a word that triggers this bug. Did you try to rerecord Q381622 to see wether this bug occurs again?
d) There are actually more bugs, which we did not discuss, that may need quick fix: For example if you use Vector Skin (click here for preview) at the top of the page it says "A maintenance operation is planned for today. ..." I am seeing this message for 2 months now. It might be a minor fix, perhaps we forgot to remove this notice. b) vector skin main page may need more work, as the main page is designed for LiLi skin, the recent files and other nice designs are not working in vector at all.
Lingua Libre supports officially only one skin (BlueLLs). I guess Vector has not been disabled when we moved to the new version of the website. I will open a ticket to ask to remove Vector so that there is only one skin to support. See T265079.
e) Coming to internationalization (which is not a bug): some important pages need to be in English also, as of now, such as LinguaLibre:Privacy_policy. Of course the page can be marked for translation, however until the page is in English also, this might a bit difficult to translate directly from French.
This is known. A working group is planned to work on the documentation page later this month. We will move all pages to English and make them translatable before the end of the year.
Check-green.svg Done
(Fun fact: You saw I used the word "LiLi". Sometime ago I posted on your talk page about the pronunciation of "LinguaLibre". I am aware in Indian community we often use the short form LiLi/Lili in our discussion, which is a female name in multiple language including a few Indian languages.)
This was indeed good to talk you. This was very kind you switched to English briefly. As gradually we are seeing more contribution from India and other countries, possibly we can have a global meet/France-India meet in future.

Regards. --টিটো দত্ত (Titodutta) (কথা) 20:47, 6 October 2020 (UTC)

Support Support I approve using the nickname "LiLi"! — WikiLucas (🖋️) 11:05, 7 October 2020 (UTC)
Yes, it's an elegant nickname. Thanks for the suggestion :) Yug (talk) 20:58, 7 October 2020 (UTC) (The writen "LL" don't translate well when we oralize it in French.)

@Titodutta for e), I've marked the page for translation, and translated it into English :). Please do not hesitate if you see other pages in the same situation. — WikiLucas (🖋️) 13:08, 8 October 2020 (UTC)

@Titodutta I answered point by point in your text to make the discussion more understandable. Feel free to reply below. Pamputt (talk) 19:08, 8 October 2020 (UTC)

Add "Recent changes (non-audio)" to "Tools" menu

1. Check-green.svg Done -- this allow quick access to list of recent changes, without the MASSIVE overload of audio recordings. Yug (talk) 15:21, 8 October 2020 (UTC)

I just found out my User:Yug/common.js doesnt work, doesn't even run a simple console.log("Hello world!"). Any idea why ?

MediaWiki:common.js does work as expected. Yug (talk) 14:52, 8 October 2020 (UTC)

mw:Manual:Interface/JavaScript: "If $wgAllowUserJs is set to true, users can customize the interface for only themselves by creating and importing personal scripts in certain user subpages." Yug (talk) 14:59, 8 October 2020 (UTC)
Damn. I wanted to test on myself before to move to MediaWiki:common.js Yug (talk) 15:00, 8 October 2020 (UTC)
@Yug I reverted what you did. I think it is not a good idea to enable it for everyone because it loads more javascript for something that almost no one uses; it is possible to get the same results in a few clicks. So please, add this code only in your common.js. Pamputt (talk) 19:20, 8 October 2020 (UTC)
Hello Pamputt. As explained above, Special:MyPage/common.js is not activated on LL so I couldn't test it there first and we aren't able to do so. So I went ahead and tested this non-breaking change on the site-wide Common.js.
Ok for the revert. Review and community discussion and approval was required, your input and revert are meaningful parts of this required discussion.
As for the whole rational... Recent change is an access point which mainly allow active users do to patrolling activity.
Current Recent change (Special:RecentChanges) access point just display 50 last changes while Lili records between 300 and 700 audios per days. The stream of recent changes is therefor overfloaded by large amount of audios files which no-one but the speaker actually will create, edit, review. Does someone browser those 3~700 audios changes daily? listen them ? Can you or me review the correctness of Bengali recordings' file names ? Or review their content correctness ? Unlikely. As far as I can see, it's a stream of "Done" things : there is no practical patrolling to do this flow of audio files, nor is it any need to patrol them. This situation is proper to LinguaLibre. Most wikis are text based. Commons:Special:RecentChanges is an hybrid with uploads but also lot of file renaming, editing, discussions, projects pages so the stream is a mix. LinguaLibre file-work is close to 100% only uploading it by the speaker. Then nothing, the stream is dispatched to Commons.
On the other hand, the meaningful changes done on textual pages which require active monitoring is made harder since these textual changes are buried down among the number of audio files. It seems to me that patrolling edited text-pages (documentation, discussion, user pages) seems as or more relevant, but is currently made inaccessible or accessible via a more complex access point.
But definitive adoption would needs consensus, which we haven't. So no quick adoption needed. Yug (talk) 08:57, 9 October 2020 (UTC)
  • Suggestions: a) enable user common.js and common.css pages (I was not aware that these pages are disabled!), AND/OR b) put this as an opt-in gadget at Special:Preferences/Gadgets. The "Gadgets" page is empty now, and gradually gadgets can be added, mostly to be opted in (or manually enabled) by interested users. Kind regards. --টিটো দত্ত (Titodutta) (কথা) 23:20, 9 October 2020 (UTC)
I do not know why user common.js is disable. Anyway, I added a new gadget for this settings. So feel free to enable it. Pamputt (talk) 09:51, 10 October 2020 (UTC)
I think the default php setting is without personal js. The developer has to set $wgAllowUserJs to true, which 0x010C apparently never did because the need never arose before. Yug (talk) 13:35, 12 October 2020 (UTC)

Translation

I translated into English several pages (LinguaLibre:Privacy policy, Help:Your first record and LinguaLibre:About) that were originally written in French, and marked the new versions for translation (I also marked for translation the latest version of LinguaLibre:Stats, which includes the latest crossed thresholds in the description paragraph, and a new row in a table). I think the translation of the pages will be easier for non-French-speakers as from now. Though, every translation of these pages - except in French - is now outdated (for all or part of it).

While translating (or patrolling other people translating), please be careful with the code (one should not change the code on translation pages, only on the main page (/en)). — WikiLucas (🖋️) 11:50, 10 October 2020 (UTC)
PS: for the Stats page, I changed the translation areas, to limit the amount of code in the translation, in order to limit the risks of translators breaking the code — WikiLucas (🖋️) 22:13, 10 October 2020 (UTC)

Perfect. Meanwhile, I finish to translate the main page and I added a button at the bottom to be able to translate this page. So, go ahead :) Pamputt (talk) 15:25, 10 October 2020 (UTC)
@Pamputt Great! All of this is a big step forward Smile. I think we could also take some time to design improvements for the chat room. Like most "Village pumps", we could divide it in two parts: permanent content in one hand, such as the FAQs, but in the other hand, some content such as this topic, archived after a while (for instance every 3-6 months, since it is not too active), in order to make the page lighter and easier to read. Also, the link "Start a new discussion" does not seem to be working. When I look over the text of the Header it's clickable, but not on the actual page... — WikiLucas (🖋️) 21:25, 10 October 2020 (UTC)
@WikiLucas00 Good idea to create a FAQ page in parallel of the chat room; feel free to start one. If you are interested in documentation, there is a meeting end of October (https://framadate.org/1C4aA6vVYWz2izgp). About archiving the chat room, this is done once a year (manually) ; see LinguaLibre:Chat_room/Archives/2018 and LinguaLibre:Chat_room/Archives/2019. Pamputt (talk) 08:25, 11 October 2020 (UTC)
Hi @Pamputt
I archived some topics that were solved since more than a month. Although, I still have a problem regarding the site's blue headers. I can't click on the links they contain (for example, the Start a new discussion button in the current page's header, or the link to access the main page when visiting a subpage such as this one). Is it the same for you? Do you know what could cause this? — WikiLucas (🖋️) 19:23, 10 November 2020 (UTC)
@WikiLucas00 yes this is the same here. About the cause Lyokoï told me that this is because there is some transparent image in front of the text (I did not check) so in principle it should be quite easy to fix (this is probably a CSS problem). Pamputt (talk) 22:32, 10 November 2020 (UTC)

Stats page

Is the Stats page loading now?
Some parts of the source page should not be taken by Fuzzybot to other language pages as it is. For example, the Statistics (En) page had language labels in "Fr" (which should have been in En, as the page was in En), while translating I fixed it. Now it is again broken, and I can not edit the bn page, other than translating. --টিটো দত্ত (Titodutta) (কথা) 22:12, 10 October 2020 (UTC)

@Titodutta You are not able to change the labels while translating? for instance in this section, you could replace the words that are within quotation marks by words in Bengali: <query yearmonth="Date" records="New records" speakers="Active speakers" languages="Active languages"> — WikiLucas (🖋️) 22:16, 10 October 2020 (UTC)
Thanks, yes, translating seems to be the only option. Regards. --টিটো দত্ত (Titodutta) (কথা) 22:38, 10 October 2020 (UTC)
@Titodutta I just understood you were talking about the name of each language in the table, and not the title of the columns. Sorry for this. I added a new section to the translation, it's mostly some code, and in theory the translator only has to insert the language code instead of "en" in the section. The thing is, only "fr" and "en" seem to be working... I set the English stats page and every translated stats page to "en", except for French (it made more sense like this). I don't know where to find the "languageLabel" in order to translate them into other languages...
I tried many changes to the requests and really felt the slowness of the current system while waiting everytime for the tables to load. @Pamputt , do you know if we could be able to add a cache to this pages (to be purged on a regular basis), to avoid having to load the whole request everytime?(for translated stats pages, the waiting time is so long that I don't think many people wait until the end) — WikiLucas (🖋️) 00:54, 11 October 2020 (UTC)
I am not really an expert of the SPARQL system. VIGNERON knows much more about that. I only know there is a bug report about the performance issues. Pamputt (talk) 08:11, 11 October 2020 (UTC)


Priorities of Lingua Libre

I've looked around and I can't seem to find any priorities of this project. It seems that the overall goal is to record pronunciation, but how this will be done is less clear. Based on my experience with Forvo, I think that this will help the project.

Words priority

Focus on pronouncing headwords first. Forvo is flooded with overly specific phrases that only a few uses will use. It would be helpful to scrape a large authoritative dictionary such as the OED, Duden, or TLFI to get a list of words. I don't think that words are under copyright.

We recommend frequency lists and authoritative this, but the copyright status of those are embiguous. Yug (talk) 14:47, 21 October 2020 (UTC)
Could we use wiktionary to help create this official list? Languageseeker (talk) 22:45, 21 October 2020 (UTC)
@Languageseeker see User:Titodutta#কোয়েরি's wiki query. I think we can use wiktionary, but I'am unclear how. Maybe it's even available in the Wizard as a built-in feature when you chose the list. Yug (talk) 20:41, 22 October 2020 (UTC)

Words variations

For each headword, pronounce it with the definite, indefinite, and solo; e.g. "the dog", "a dog", "dog". Also pronounce the declined forms in languages such as Latin or German. Group them all on one page under the headword. For phrases, there's no need to inflect or decline them.

Words variations and verbs are typical to English and western languages. I'am not sure how each language process those questions, but I think we have no recommendation in place. Yug (talk) 14:47, 21 October 2020 (UTC)
This is why I believe it is important to create recommendations in place before the site gets too large. We don't want to have to manually deal with these issues latter. I know that the editors of Forvo are struggling with precisely this issue, especially in English. The best thing to do would be to create a bot to tag alternative spellings, generate files for them, and automatically generating the pages for the alternative spelling. For example, in French, you have électroménager and électro-ménager . If a user pronounces either one of these orthographic variations, the bot should generate files and pages for both variations. Otherwise, we'll be asking to effectively pronounce the same word multiple times. It would also probably be helpful to create a bot to scrape alternative spellings from wiktionary. Languageseeker (talk) 22:44, 21 October 2020 (UTC)
Maybe we should consider "Portals" per language.... with the specific tips, recommended list, active/reference users. Seems a good idea (IF someone ready to attack it XD) Yug (talk) 20:18, 22 October 2020 (UTC)
@Yug You inspired me to create a phabricator ticket on how to do this. Languageseeker (talk) 02:00, 23 October 2020 (UTC)

For verbs

It's best to focus on the irregular and model ones first. It's also makes sense to pronounce them in all the possible permutations. For example, in French, il/elle/on est should have the following entries: "il est", "elle est", "on est", "il/elle/on est", and "est"

As for the previous point, each user is free to record whatever he/she wants. So the question becomes how to manage/organise/browser all the recordings. For that point, everything has to be done so your point of view is more than welcome. Pamputt (talk) 16:06, 21 October 2020 (UTC)

User supplied lists

They are great, but quickly turn into a headache. They require lots of proofreading that can overwhelm editors. Only after we finish pronouncing all the headwords and verbs should we open this to general suggestions.

For "user supplied lists" and "site supplied lists" there are some ongoing efforts this side. An user can create a list which becomes a site supplied list for later users. [EDIT]: We have help pages recommending and demonstrating how to create frequency lists for better impact. See Help:Main> search "frequency". Yug (talk) 14:47, 21 October 2020 (UTC)
If I understand correctly, you suggest to have "official" lists that are proofread so that we can propose these lists in priority to the users. I think it is a good idea because some users do not always know which words to record. We should open a Phrabicator ticket to keep track of that feature request. Pamputt (talk) 16:06, 21 October 2020 (UTC)
Precisely, I'm not against allowing speakers to pronounce whatever they want, but we should have some official list of what we want pronounced. This is why I also suggested the different forms that we should target, especially if we begin to suggest words for users to pronounce. Otherwise, people will begin adding lists with misspelled words or phrases that have no widespread usage. This will create unnecessary work for editors to correct and delete. Do we really want discussions about whether or not "The Pink Adrietic restaurant will closed today at 9:30 due to an alien invasjon" should be on the official list at this stage? What will we do if a user adds 60,000 of these? Languageseeker (talk) 22:56, 21 October 2020 (UTC)
(Note: I edited my paragraph above.) Yug (talk) 14:43, 22 October 2020 (UTC)

Site supplied lists

Such lists of words that usersy can pronounce are better from a project management standpoint. This list should be randomized at each refresh in case the user is not interested in the selection. Let users see the list first and then make them log in.

I am not sure to understand exactly what you mean. Is it related to the previous item? Could you give an example? Pamputt (talk) 16:06, 21 October 2020 (UTC)
It's related to the previous item. If we create official lists, we should not adopt the last-in/first-pronounced model of Forvo. On Forvo, the last word added is the first word in the list for users to pronounce. Speakers have no option to change the way that the list is generated. I believe that we should have more flexibility. Instead of displaying the last words first, I'm proposing that LL randomly sorts the list by default. We could also add a drop-down menu with: Random, Newest, and Oldest. Languageseeker (talk) 23:00, 21 October 2020 (UTC) Edit: Official lists can also help prevent unnecessary duplication of effort. Look at the entry for "arbre" on French Wiktionary. Do we really need 32 pronunciations of "arbre"? Does it make sense to add 32 different sound files to one wiktionary page? Languageseeker (talk) 16:06, 22 October 2020 (UTC)
Note: There are various schools of though here on LinguaLibre :
  • the linguists, which would be happy to have 200,000 version of "arbre" so one could study the variability.
  • the language teachers/learners/learning apps developpers, who one one clear and standard speaker for 10~30,000 most frequent words, just once each and with zero hole in the dataset.
Yug (talk) 20:45, 22 October 2020 (UTC)
I'm not for limiting the maximum number of pronunciations for an item, but I do not want a situation where "arbre" has 200,000 pronunciations and "cigale de mer" has zero. A site supplied list will make it more likely that rarer words will receive at least one pronunciation. Languageseeker (talk) 02:07, 23 October 2020 (UTC)
+1. Maybe a specific label within the list's pagename. Ex: List:CMN/HSK-0001-to-8868_(RECOMMENDED) ?
This decision could be made within a language community via its Portal. Yug (talk) 18:11, 23 October 2020 (UTC)
+1. I like the idea of adding labels. I think this should help also avoid any copyright claims. We can add tags such as "HSK 1.1" (HSK Revision 1 Level 1) "HSK 2.1" (HSK Revision 2 Level 1) and "HSK 3.1" (HSK Revision 3 Level 1) for the various iterations of HSK and that way a user can easily tag see all the word pronounced for a given language list. Languageseeker (talk) 00:48, 24 October 2020 (UTC)
I also support this idea. Currently, all the lists of a given language are proposed to the contributor. For the French language, it starts to become a bit messy to find what we are interested in. So I think the system of local list of the Record Wizard should be improved to highlight some specific lists. I will open a Phabricator ticket to keep track of this idea. Pamputt (talk) 13:46, 1 November 2020 (UTC)

Accents matter

See also: Help:Renaming (using metadata tags).

They should be tagged as part of the filename. For example, LL-Q1860_(eng)-Commander_Keane-phonate.wav contains no accent information. LL-Q1860_(eng_Au)-Commander_Keane-phonate.wav would be better. Also, allow users to filter by accent codes. We would need to think about and propose a list of accents for each language.

The choice has been done to tag the location in the wiki metadata. Indeed, people are not always aware that they have an accent and an accent can highly vary in a given country or region. So it becomes difficult to find the good granularity. So that, saving the location allow to write some query to get exactly the recordings we are interested in. Pamputt (talk) 16:06, 21 October 2020 (UTC)
But, geographic location is not an accurate predictor for an accent because people move around and some have speaking impediments. For example, you can have an American living in Paris or a Parisian living in America. Who will speak the words more accurately? A person living in London could speak the Queen's English or have a Cockney accent. Especially for language learning, accents are important. We're not judging accents, but merely tagging their existence per speaker. If a user has an incorrect accent listed, I believe that mods should be able to change it annd Lingua Libre will automatically retag all their pronunciations. Languageseeker (talk) 23:06, 21 October 2020 (UTC)
@Languageseeker : I believe information on accent is conserved thanks to the file metadata. To make them more visible, see Help:Renaming and fish metadata tag' value to push it into the new filename of your desired shape. Yug (talk) 14:48, 22 October 2020 (UTC)
@Yug I don't see accents on either user profiles or on Wiki Commons. See: speaker Q141723 for instance. Accents should be easily visible and filterable. For Example, on commons there should be a category for Lingua Libre pronunciation in French (Parisian Accent). On the description page for a file on Commons, there should be "AccendId" under "languageId" Languageseeker (talk) 16:00, 22 October 2020 (UTC)
The profile contains the location of where you learnt a language. City and country, if I remember well. Yug (talk) 19:59, 22 October 2020 (UTC)
@Yug Correct, but this does not necessarily translate into accent. We're making an assumption that an accent can be geographically located and that all individuals that inhabit that geographic location will have a particular accent. Also, an accent can span across geographic region as well. What's wrong with adding a third parameter called accent? Is there any reason that you feel that it would be detrimental? Languageseeker (talk) 20:10, 22 October 2020 (UTC)
The filenames are already 3 times too long in my opinion.........
I don't remember clearly. But maybe we then assumed the *speaker* to be the data marking the accent. @Lyokoï may remember. --Yug (talk) 20:16, 22 October 2020 (UTC)
Could we set an advance option in the user profile, where users can decide how they wish the language files to be named based on metadata? That way users can have fine-tune the way they see the files?
My point is that this is an invalid assumption. People speak with dialects and may have a speech disorder that Lingua Libre should have a metadata tag for. It's easier to add at first, then to have to manually add later. It shouldn't be that hard to implement. Furthermore, it will also make it easier for users to filter pronunciations. Take, for instance, Russian that has three major accents with twenty-four sub-divisions in Russia. Each accents occurs in hundreds of cities and villages. If we don't have dialect metadata, then we need to create a list with all of these geographical locations to group the pronunciations in the same dialect. If we have dialect metadata, then I can just filter by "Central Russian" or "Chukhloma enclave." All we would need to do is add the following three options to person's user page "Dialect" (required); "Dialect Sub-group" (optional); "Speech Disorder" (optional) and then have that propagate automatically to all their pronunciations. To help users and promote standardization, we can make these drop down menus with an option for custom. Languageseeker (talk) 01:12, 24 October 2020 (UTC)
I really think that accent are subjective. In France, the "standard" accent is the one used in Paris, so people from Paris think they have no accent. Someone living in the North or the East of France may think he/she has no accent bu people from other regions think they have one. So, you are not always aware that we speak with an accent and if we are aware, we do not not know how to name it. That is why the information of the location, even if not perfect, is not so bad.
That's said, to fit your need, there is this ticket that asked for custom categories. I think it can be used for this purpose. Pamputt (talk) 13:43, 1 November 2020 (UTC)

Authoritative lists (& copyrights)

Lists such as HSK or JLPT should be an high priority. This would help language learners the most.

No problem to import such lists if they are not copyrighted. No idea on that. Pamputt (talk) 16:06, 21 October 2020 (UTC)
@Languageseeker & Pamputt Actually, we are not Wikipedia nor Commons. So we could set up our own copyright rule in accordance with the server's geographic localisation and associated laws. We can consider, like Anki and Skritter.com do, that we are just hosting content uploaded by the user, who is the legally responsible party, and as the European and French law commands: we will take down any content following a formal complain. This is the true legal requirement we have in France. Our (LinguaLibre) rules don't have to be as pro-active as Wikipedia or Commons, which decided to go beyond what is legally required. Where we put the cursor is really up to us. It's a matter of internal policies. Some tolerance such as the one I cited above could be greatly advantageous to LinguaLibre's objectives. Indeed, it's the strategy that took Anki, Skritter, Memzine, Duolinguo, and many other actor of online language learning. Yug (talk) 14:55, 22 October 2020 (UTC)
@Yug & Pamputt Agreed, we shouldn't worry too much about copyrights until someone submits a claim and then remove the entries. I think it'd be very difficult to file a copyright claim for "chien." We could also automatically reorder list to prevent an argument that the specific order makes them somehow under copyright. We could also rely on corpuses in the PD at first. Languageseeker (talk) 16:04, 22 October 2020 (UTC)
Copyrights violations are not possible for stand alone lexemes such "狗" (gǒu: dog) from HSK.
List are different : lists are specific creation of one's mind so its author can claim copyrights on a given list.
Still, the Chinese Ministry of Education which authored the HSK LIST OF SELECTED WORDS and could legally file a copyright complain yet never filed copyrights violations on any for-profit companies to remove their online HSK lists. Then why would the MoE do so for a non-profit ? And if they do, we can simply delete the said list(s).
Shuffling is not enough to claim difference and originality, especially when your page is named "List:cmn/HSK1" ^^
To claim originality the minimum would be to substantially edit the list. In order to not loose data, it lead to adding words. The HSK 1 to 6's 8800+ words could indeed be extended to 11000 via a merge with a relevant frequency lists, around this threshold of difference we could start to claim originality. The algo could be ligthly more complex, with ranking... You see the idea. But I thing the fair-use option is more practical and relevant for us (see below).
Sum up: as I shared above, I think we could advise and state that :
  • Our policy should be based on the law of the land (Europe & France's laws) and current observed online practices.
  • Our users make fair-use judgement and uploads the lists
  • Our admins and/or bureaucrats (?) receives the copyrights claims, one of them do a rapid review process on a case by case basis, then remove the list if the complain has merit.
This seems a good balance satisfying both relevant laws together with our project's objectives and interests. Can we push this way ? Yug (talk) 19:40, 22 October 2020 (UTC)
@Yug This is an awesome idea. I think that it would make a lot of sense to combine a set of frequency lists with official lists, then deduplicate them. This should prevent any copyright claims. We can even think about whether it makes sense to create frequency lists based on data in Project Gutenberg as part of this process. Then, ve should also write a script to automatically add inflections, conjugations, articles, etc. based on wiktionary data. These lists could be the first set of official lists for LL. Languageseeker (talk) 22:06, 22 October 2020 (UTC)
We don't have the human resources to find and merge those list as I wished above. It need knowledge of the target language, of the available resources (only major, official languages such as EN/FR/ES/DE/JA/CN/KO have HSK/JLPT-like lists), and programming skills. Then add free time and willingness. Having them all is very rare. Better to redefine our copyright rule toward more tolerance so we accept any list there is. Yug (talk) 18:09, 23 October 2020 (UTC)
Agreed. I do think that this might take too much time and dev energy for a temporary and potentially problematic taks. I'm against user supplied lists at this stage for the same reason. I don't think that we have the resources to proofread and merge multiple lists. I'd propose focusing on creating an official list from the various languages of Wiktionary because that will have no copyright claims and will, eventually, contain all the words and phrases in a language. If we create a list from that, it will probably contain around 500, 000 to 600, 000 items per language which will be enough for an initial set and then we can open it up to user suggestions afterwards. Languageseeker (talk) 00:39, 24 October 2020 (UTC)

Recording quality and post recording clean ups matters

Words pronounced with lots of static or background hum should be deleted unless it's a truly rare language.

We have no or weak process for that. There are documented methods to denoise audios, see : Help:SoX. Yug (talk) 14:47, 21 October 2020 (UTC)
Yes, some tools have to been developed to control that. Ideas are welcomed. About denoising, a ticket already exists to add by default in the Record Wizard. Pamputt (talk) 16:06, 21 October 2020 (UTC)
Most of this has to do with recording equipment. On Forvo, recording quality is usually pretty fairly consistent for users. We may have to flag users with recording quality issues and delete their files. It could be a manually review process for a few files or batch deletion. The same would be true for users with terrible or fake accents. (Think Dick Van Dyke in Mary Poppins) Languageseeker (talk) 23:11, 21 October 2020 (UTC)
Personnally, I think we should argue more for good microphones and silent room. We frequently run after users and get lightly-noisy audios which are not satisfying. Yug (talk) 15:03, 22 October 2020 (UTC)
@Yug It seems that we need a statement on our expectations for recording quality. In the end, we only need one good pronunciation entry per language + accent. Terrible quality recording help nobody unless they are the only one that we have. It might be worth having a voting system such as on Forvo to help flag good/bad speakers. We can even toy with the idea of a speaker of the week or month to reward those who really help us out. Languageseeker (talk) 16:46, 22 October 2020 (UTC)
+1. We need a quality statement somewhere. It's not perfect but it sets the tone a bit so we may require more quality from contributors. Yug (talk) 20:11, 22 October 2020 (UTC)
I decided to create a draft of standard for Libre Lingua in phabricator. Languageseeker (talk) 03:36, 23 October 2020 (UTC)
@Languageseeker Not dev related, to move back here. Also we need to check we may have something already in place making most of the job. Yug (talk) 18:03, 23 October 2020 (UTC)

Minimal threshold ?

I also recommend to push for sets of more that 1000 words. Basically, in the download page, we should compile list by language by all datasets (one speaker one language) of less than 1000 audios should be ignored. We assume that sets of 1000+ audios are autoconfirmed. While sets of <1000 audios are potential beginners and likely noisy play-around (as for myself ! My audios are just test and not good !). Ideally we would have a download page such as :

Download audios
Language All audios Top 1 speaker Autoconfirmed speakers (≥1000) Other speakers (≤999)
French 88,934 audios by 34 speakers 47,076 audios by speaker Tom Smith 76,567 audios by 4 speakers 12,367 audios by 30 speakers
Gascon ... ... ... ...

Yug (talk) 20:11, 22 October 2020 (UTC)

Disagree, I don't think we should require a minimum number because that would discourage users. This would especially impact language with fewer speaker and can perpetuate oppression. Instead, I think we should not divide audio into datasets by speaker. Moreover, a system to vote on pronunciations and report them can help to flag problems. As a final resort, we can vote on whether or not to batch delete the pronunciation of speakers that are particularly horrible. Languageseeker (talk) 03:36, 23 October 2020 (UTC)
@Languageseeker datasets are grouped by languages via downloadable zips, then by speakers.
My proposal is to create variable packagings for one languages : All / Top speaker / Autoconfirmed speakers / Non-autoconfirmed speakers.
The ranking system / API would be great yes. Right now when I review a list of words I have to copy the filename(-filepath), store it, to then send a message "this audio is to redo". Not right. Maybe a smart template could do as of now. There is also the question of synch between Lili and Commons to keep in mind. Yug (talk) 18:00, 23 October 2020 (UTC)

Volume normalization

Check-green.svg Done -- feature request on Phabricator. Yug (talk) 20:05, 23 October 2020 (UTC)

It should be automatically provided across the entire corpus. We don't want one word at 140db and the other at 20db.

Volume normalization : we already have some normalization, we reject low db and high db recordings. But I cannot specify the exact mechanism : per file ? per recording set? I think it's the former. Yug (talk) 14:47, 21 October 2020 (UTC)
This has already been proposed and is saved in ticket. Pamputt (talk) 16:06, 21 October 2020 (UTC)

Anki plugin

Check-green.svg Done a proposal have been documented on phabricator. To keep in mind and follow through there. Yug (talk) 20:30, 22 October 2020 (UTC)

It would be great to develop an Anki plugin that would enable users to automatically add audio to flashcards. This is the biggest downside to Forvo that requires users to manually add one word at a time. Providing an Anki plugin will help to popularize this project attracting new users. Since, Anki is python based, this can be based on the French bot. Having a large group of testers can help to identify how the metadata of these files can be improved.

Anki plugin: YES, it's about 1~2 day work. Maybe Anki folks could help.
  1. Read Anki's documentation for Anki decks folder's syntax
  2. Download the target language folder
  3. Create a bash script to pick up the filepaths, the words, and create the Anki decks file with the proper syntax.
  4. Document process to share file with Anki community Yug (talk) 14:47, 21 October 2020 (UTC)
Please, feel free to open a Phabricator ticket to keep in mind this need. Pamputt (talk) 16:06, 21 October 2020 (UTC)
Appreciated. I'll write up a full proposal in the next few days and open a Phabricator ticket.
[EDIT]:Ticket created for Anki Plugin. Languageseeker (talk) 05:22, 22 October 2020 (UTC)
Awesome. Thank you :) Yug (talk) 15:06, 22 October 2020 (UTC)

Flac

Files should be uploaded and stored as flac files to enable tagging and reduce file user. All modern browsers use flac and it's emerged as the default lossless audio compression format that is widely supported across different devices. Tagging will help keep the metadata with the file and enable easier renaming by end user or bots.

Tagging and flacs : we do tag files in the files' code; we already have long discussion about file formats and chose to keep .wav together with sharing scripts to mass convert to alternative formats. See Help:Convert_audios?. Yug (talk) 14:47, 21 October 2020 (UTC)
I was not aware about the reason of choosing the wave format. Concerning FLAC, there is a ticket explaining why it is interesting (in addition of the points given here). Pamputt (talk) 16:06, 21 October 2020 (UTC)
It's a decision by Mr. Vion and @Lyokoï if i remember well. Maybe worth creating a Help:Formats.
Side note: it also seems to be the time to create categories to class our help pages via several dimensions : scripts, lists, guidelines, recording, ... ? Yug (talk) 20:32, 22 October 2020 (UTC)

Discussions

These are my few thoughts and I'd love to hear any feedback. Languageseeker (talk) 19:30, 20 October 2020 (UTC) Edited on 20 October 2020, 23:35 (UTC)

@Languageseeker Hi there, thank you for this review. I edited your points and added bold so the key topics are more visible and we agree on naming for the discussion to continue. I will try to answer to several of your points Yug (talk) 14:47, 21 October 2020 (UTC)
I reorganised by section and move your answer, Yug, so that it will be easier to follow the different points (I hope so). I will add some answers as well. Pamputt (talk) 15:49, 21 October 2020 (UTC)
Pamputt Thank you!! Languageseeker (talk) 23:12, 21 October 2020 (UTC)
Thanks, good by me. As long as it improves and is in good faith it's a good practice to allow. ;) Yug (talk) 19:21, 22 October 2020 (UTC)

Datasets to download

Hello the team, I just noticed that https://lingualibre.org/datasets/ which is central for external developers such Anki's community to reuse our audios work has been lost in the recent UI revamp. Any idea where to put it back so it stays highly visible ? Yug (talk) 14:50, 21 October 2020 (UTC)

The dates visible are also mainly from 2019. Any idea what they are ? First compilation ? Last compilation ? Yug (talk) 14:53, 21 October 2020 (UTC)
Yea, there's a message redirecting towards the datasets in DataViz:Records but I agree they would be more visible under "Actions" (upper left) or "Tools" (bottom left) - Eavq 18:40, 29 October 2020 (UTC)
Another idea would be to add a menu besides Record Wizard, Discussion, Statistics, Help and About. You could name it "Dataset". That's said, I do not know whether we can do it by ourselves or whether we need access to the server. Pamputt (talk) 13:30, 1 November 2020 (UTC)

CSS fixes

See MediaWiki:Common.css

There is clearly some CSS to update. Most notably for the H2, H3, H4 section titles *within wikipedia pages*. I will test some solutions soon. Feel free to test other CSS aspects in MediaWiki:Common.css (admin only?) or in User:Yourname/common.css. Yug (talk) 14:55, 21 October 2020 (UTC)

Proposed Community Standards

Based on a conversation with Yug, I've decided to draft a community compact statement that sets out our standards. Any and all feedback welcome.


Lingua Libre is dedicated to providing free, high-quality recording of words and phrases in all languages. To achieve this, we ask you to abide by the following community standards

  1. All people and accents are welcome. However, do not assume an accent that you do not normally use.
  2. Please, fill out your profile information accurately. This enables the correct usage of your pronunciations.
  3. Do not upload pronunciations that you did not create or are posted with a restrictive license elsewhere.
  4. Please, record in a quiet room with no background noise audible when listening with headphones.
  5. Do not include excessive silence before or after your pronunciations.
  6. Record your pronunciations in a relaxed, neutral tone. If you wish to pronounce them fast, slow, or emphatically, please tag your pronunciations appropriately.
  7. Discrimination, cyberbullying, harassment, stalking, or any other form of intimidation on the base of age, accent, class, disability, ethnic identity, gender identity, geographical location, marital status, native language, political beliefs, race, religious identity, religious beliefs, sex, sexual orientation, or any other category will not be tolerated. Any such behavior will result in a permanent ban.
  8. Lingua Libre reserves the right to delete any pronunciations that do not follow these guidelines.

Languageseeker (talk) 00:34, 24 October 2020 (UTC)

  • Hello, points are OK, except #8, for various reasons such as a) LL pronunciation are uploaded on Wikimedia Commons and follow Wikimedia Commons guidelines, several of the points mentioned above are not reasons to delete on Wikimedia Commons. Regards. --টিটো দত্ত (Titodutta) (কথা) 21:14, 25 October 2020 (UTC)
I appreciate your feedback. I think that we need to review files before adding to the Wikimedia Commons because otherwise we have the potential to flood Commons with recordings of extremely poor quality. I've heard some files with truly horrific sound quality issues on Forvo and have also seen files in which the speaker did not properly read the text. In any language, pronouncing the headwords and all their variants will amount to somewhere between 200,000 to 1,000,000 unique entries. We don't want to that many files that are unusable on Wikimedia. I'm ok with just flagging the files and then deleting them when better quality versions are available. Of course, for rare languages or places of more limited means, we won't delete files just because of sound quality.
I mainly want to have a deletion policy for users that pronounce word in a stereotypical manner that perpetuates discrimination. For example, the white person speaking "black," a straight person attempting to sound "gay," etc. These should be collective judgement calls by administrations with bans and deletions. Languageseeker (talk) 00:42, 26 October 2020 (UTC)
Actually the problem right now is we do not have any sysop tool to check the quality of the files and to manage these fileS. It may evolve in the future but currently we have no mean to apply the policy we discuss here. Pamputt (talk) 13:27, 1 November 2020 (UTC)

Lingua Libre bot on Wiktionary

Most probably LinguaLibre bot is working on 2–3 Wiktionary projects now, such as French, Occitan etc. What are the steps to enable the bot for other projests such as Bengali Wiktionary? --টিটো দত্ত (Titodutta) (কথা) 15:14, 27 October 2020 (UTC)

@Titodutta Do you have the bot name under hand, could you share it ? I think the bot master was 0x010C, who is leaving for a year nearly off grid (scientific mission in isolated place).
We need to call for new botmasters and bot hacker/coder. Yug (talk) 20:26, 31 October 2020 (UTC)
The bot's name is Lingua Libre Bot. — WikiLucas (🖋️) 00:27, 1 November 2020 (UTC)
I do not think Lingua Libre Bot will support new Wiktionaries soon because 0x010C is away for a long time. That's said, the code is public and it is possible to use it to run its own bot. You can adapt the code from already supported Wiktionary to your Wiktionary. Pamputt (talk) 13:24, 1 November 2020 (UTC)

Template:Welcome

Check-green.svg Done
It would be good to create a Template:Welcome (see a similar template on French Wiktionary fr:wikt:Modèle:Bienvenue or English Wikipedia en:Template:Welcome). We see editors joining this site, we have seen more participation from South-East Asia, and India recently. It would be good to send a welcome message that will have links to important pages, including this Project Chat page. --টিটো দত্ত (Titodutta) (কথা) 14:04, 6 November 2020 (UTC)

I've imported Template:Welcome from Wikidata (it is an international template). Now, we need to write the welcome text specific to Lingua Libre. It happens here. You are more than welcome to make a nice template. Pamputt (talk) 15:37, 6 November 2020 (UTC)
@Pamputt I made a very first draft, which needs styling improvements, in order to make it fit in a smaller space, like a box. I had issues with the translation tags, especially when it came to links (the <tvar> tag for links in translations does not seem to work, do you know why?).
Do you agree with the content? Do you see important information to add? — WikiLucas (🖋️) 20:50, 8 November 2020 (UTC)
  • Hello, one stable version can be created on a sandbox, and other versions can be translated from that. Links like Special:MyPage works for any language. Any link of the main text preferably should have a filter of "Special:MyLanguage". I would like to see 2 points, a) please make sure to listen to all the pronunciation before uploading, b) consider using a microphone, and upload audio without background noise (or in other words, it would be good to add some details about the best practices). Regards. --টিটো দত্ত (Titodutta) (কথা) 21:42, 8 November 2020 (UTC)
I changed appearance a little and made this sandbox for the text, but I think we all agree that it's not nice. We need someone with good skills to design the style of this page, once we have all agreed on its content. @Titodutta we already give advice about best practices in the Help pages, but this of course still needs to be improved. Regards 🙂 — WikiLucas (🖋️) 02:18, 9 November 2020 (UTC)

To @Pamputt, Titodutta, Yug, Languageseeker, Eavq, & Nicolas Lopez de Silanes WMFr , ...
Hello everyone. Could we talk about the information we want to display on this Welcome template ? I created this collaborative page that we could all edit to add (or remove) points that we find interesting. This is only about the content, and the page is currently in English (we will focus on appearance and other languages later). Regarding the appearance, I made this suggestion (not definitive). Please tell me your thoughts about it.
Best regards — WikiLucas (🖋️) 23:30, 15 November 2020 (UTC)

Nice job WikiLucas00. I really like User:WikiLucas00/Welcome/text/sandbox. The current content looks good to me. Pamputt (talk) 07:03, 16 November 2020 (UTC)
  • Great work indeed WikiLucas00. It looks fantastic. I have added a couple of subst options. Test results can be seen here. Using {{subst:User:WikiLucas00/Welcome/text/sandbox}} will the call the BASEPAGENAME, and put the signature I think. I have also linked the logo as it is under CC-SA (not CC0), it needs a link back for attribution details. Feel free to revert as it is your sandbox. Regards. --টিটো দত্ত (Titodutta) (কথা) 10:13, 16 November 2020 (UTC)
@WikiLucas00 & Titodutta I moved what WikiLucas00 has prepared to Template:Welcome/text so that it is right now usable with Template:Welcome. So if the content looks good to you, you can translate it to your language. Pamputt (talk) 20:37, 17 November 2020 (UTC)
Great job! We are now able to greet properly future newcomers 🙂 How can we watch/be alerted when new contributors contribute here for the first time? — WikiLucas (🖋️) 22:20, 17 November 2020 (UTC)
I do not know a perfect way to be aware of new user here but you can watch the changes in the User namespace. When you see a line such as "User:XYZ/RecordWizard.json", it means that this user set its personal config. So it is a good start to see whether he/she already received a Welcome message. Pamputt (talk) 16:43, 18 November 2020 (UTC)
@Pamputt & Titodutta Thank you! — WikiLucas (🖋️) 19:04, 18 November 2020 (UTC)
PS: When you want to use the template to welcome a new contributor, please use it this way: {{subst:welcome|--~~~~}}. Thank you! — WikiLucas (🖋️) 22:28, 17 November 2020 (UTC)

Users

Check-green.svg Done
Feel free to make it a level 3 header, however this idea came to my mind when I was checking text of this template
It would be good to register User:Example and User:Lingua Libre. User:Example userpage and its subpages can be used for demo user pages or other demos. Lingua Libre username can be taken as that is the project name. Preferably someone (admin/from Wikimedia Fr) should register the names. --টিটো দত্ত (Titodutta) (কথা) 11:28, 16 November 2020 (UTC)

Because we can log in Lingua Libre only via an account on any Wikimedia website, I checked what is the status of User:Example and User:Lingua Libre. The first is blocked and only used as an example on Wikimedia Commons. So here, I think we can use this page (User:Example) for our purpose and I will protect it so that only admin can edit. About User:Lingua Libre, there is already a real account. It has been created in 2017 by someone involved (I guess) but I do not know who. So the best is not to use this account because it could be used, in theory, and we work only with User:Example. Thus, I have copied the content of User:WikiLucas00/User_Page_Demo to User:Example. Pamputt (talk) 17:07, 16 November 2020 (UTC)


Update: The template is now fully useable and translatable. I also imported this template from the French-speaking Wikisource, in order to explain how to use the babel template, on this demo user page: User:Example (also translatable, but using LangSwitch --you have to create a subpage named with your language code, and list it on the main page). — WikiLucas (🖋️) 19:15, 18 November 2020 (UTC)

About the exclusion of already recorded words

Hi, I think the option to exclude words that I have already recorded is broken. This morning, I start a recording session and LL proposes me words that I registered two days ago. For example, I already registered Belorusino two days ago, but it does not disappear when I click exclude words already recorded. And notice the two versions of the file, which I already re-recorded it. Can someone fix this? Lepticed7 (talk) 10:07, 15 November 2020 (UTC)

I have opened a Phabricator ticket. It may be fixed in the coming months but not sure. Pamputt (talk) 20:05, 15 November 2020 (UTC)


A Wikimedian in Residence for Lingua Libre

Dear all,

After an epic (but domestic) brainstorming, Sebleouf, Noé and WikiLucas00 proudly come to announce the first residence dedicated to Lingua Libre!

To fund this residence, which will take the form of an internship, we take advantage of the on-going project Dictionnaire des francophones which plans to reuse Lingua Libre recordings.

WikiLucas00 will be hosted as an intern at the Lyon 3 University, at the Institut international pour la Francophonie under the supervision of Sebleouf.

From May to August 2021, WikiLucas plans to implement suggested tasks, in order to facilitate individual collection and recording-workshops, and will explore solutions for patrolling new recordings.

The full presentation is here

Best Regards,

Sebleouf (talk), Noé (talk) and WikiLucas00 (talk) 14:28, 20 November 2020 (UTC)

That's really AWESOME news !!! Congrats. All the best. DSwissK (talk) 16:15, 22 November 2020 (UTC)

Stats page November 2020

The stats page is most possibly not working for several days and showing a week's old statistics. Regards. --টিটো দত্ত (Titodutta) (কথা) 23:40, 26 November 2020 (UTC)

There is possibly a kind of cache. How do you know that there is a week lag? Pamputt (talk) 07:00, 27 November 2020 (UTC)
... because the stats are not getting updated for close to 10 days now and showing old stat disregarding recent uploads, regardless of device, browser etc. Let's see sometime tomorrow. Regards. --টিটো দত্ত (Titodutta) (কথা) 06:07, 30 November 2020 (UTC)
Same for me, counter blocked at 333606 items for a few days. — WikiLucas (🖋️) 10:23, 1 December 2020 (UTC)
Can this be fixed User:Pamputt? --টিটো দত্ত (Titodutta) (কথা) 23:42, 4 December 2020 (UTC)
We had an online meeting today with the WMFrance team and reported the issue, things should be fixed in the next days/weeks. — WikiLucas (🖋️) 02:31, 5 December 2020 (UTC)

Meta Community Wishlist for 2021

Hello @Pamputt, Yug, & Titodutta and everyone else!
For the Community Wishlist Survey 2021 on Meta, I posted a request that had already been posted last year, and which got 40 support votes but didn't make it to the top 5. It is about adopting Lingua Libre Bot as a WMF Tool. Please tell me your thoughts about it, we still have a short amount of time to modify it before the opening of the votes. Regards — WikiLucas (🖋️) 17:33, 29 November 2020 (UTC)

Sounds good for me. Thanks for the proposal. Pamputt (talk) 07:06, 30 November 2020 (UTC)

New interwiki link for Lingua Libre

Good news!
Pamputt's request on Meta in May 2020, related to the creation of an interwiki link for LiLi, has been accepted! We now have to wait (a few days or weeks) for the next cache update in order to be able to use links with the [[LinguaLibre:...]] prefix on any wiki! — WikiLucas (🖋️) 11:13, 1 December 2020 (UTC)

Retrieved from "https://lingualibre.org/index.php?title=LinguaLibre:Chat_room&oldid=365707"