View source for How to create a frequency list?

{{#Subtitle:Words lists sorted by frequency are a very good way to cover one language methodically. After reading this page you will be able to '''find or create your own frequency list''', clean and split it into easy-to-handle files.
}}
{| class="wikitable"
! Reminder : to start a recording session you need
|-
|
# One LinguaLibre user,
# One willing speaker, and
# '''One list of items to record''' with one item by line.<br> One item can be any easy to read sign, word, sentence or paragraph. The most common use-case is to record a comprehensive words list for your target language.
|}
== Reusing open license frequency lists ==

=== Hermite Dave's lists ===
Hermite Dave created 61 frequency lists from OpenSubtitle data, covering most major languages under CC license. This data requires minor clean up, example with Korean (<code>ko</code>) :

<pre>
mkdir -p ./clean                                                              # create a folder
google-chrome github.com/hermitdave/FrequencyWords/tree/master/content/2018   # open in web-browse to browse available languages
iso=ko                                                                        # defined your target language
curl https://raw.githubusercontent.com/hermitdave/FrequencyWords/master/content/2018/${iso}/${iso}_50k.txt | sort -k 2,2 -n -r | cut -d' ' -f1 | sed -E 's/^/# /g' > ./clean/${iso}-all.txt
# download, sort by 2nd column numerical value descendant, cut by space then keep first field, add # to make a list, print all to file.
split -d -l 5000  --additional-suffix=".txt" ./clean/${iso}-all.txt ./clean/${iso}-words-by-frequency-
# split in files of 5000 items
</pre>

On LinguaLibre.org, [[Special:MyLanguage/Help:Create_your_own_lists#Create_a_new_list|create your lists]] as <code>List:{Iso3}/words-by-frequency-00001-to-5000</code>, etc. Ex. <code>[[List:Pol/words-by-frequency-00001-to-02000]]</code>. <br>

After creating the list on LinguaLibre, add the following to its talkpage:
<pre>
==== Source ====
{{Hermite Dave}} 
</pre>

=== UNILEX's lists ===
UNILEX is an Unicode Consortium project which curates 999 languages. As many frequency lists are available under GNU-like license. This data requires minor clean up, example with Igbo (<code>ig</code>) :
<pre>
mkdir -p ./clean                                                              # create a folder
google-chrome github.com/lingua-libre/unilex/tree/master/data/frequency       # open in web-browse to browse available languages
iso=ig                                                                        # defined your target language
curl https://raw.githubusercontent.com/lingua-libre/unilex/master/data/frequency/${iso}.txt | tail -n +5 | sort -k 2,2 -n -r | cut -d$'\t' -f1 | sed -E 's/^/# /g' > ./clean/${iso}-all.txt
# download, remove first 5 lines, sort by 2nd column numerical value descendant, cut and keep first field, add # to make a list, print all to file.
split -d -l 5000  --additional-suffix=".txt" ./clean/${iso}-all.txt ./clean/${iso}-words-by-frequency-
# split in files of 5000 items
</pre>

On LinguaLibre.org, [[Special:MyLanguage/Help:Create_your_own_lists#Create_a_new_list|create your lists]] as <code>List:{Iso3}/words-by-frequency-00001-to-5000</code>, etc. Ex. <code>[[List:Pol/words-by-frequency-00001-to-02000]]</code>. <br>

After creating the list on LinguaLibre, add the following to its talkpage:
<pre>
==== Source ====
{{UNILEX License}} 
</pre>

=== Subtlex's lists ===
The [[:en:Word_lists_by_frequency#SUBTLEX_movement|Subtlex movement]], a group of academic frequency list studies based on open subtitles, also provides about 10 of the highest quality frequency lists. Items are better cleaned up, etc. These resources are published under various licenses. Their usage on LinguaLibre must be on a case by case basis.

== Corpus ==
Requirements for relevant corpus :
*Size: 2M+ words.
*Type: raw text.
*Language: monolingual or close to be.

==== Download a corpus ====
You can download available corpuses in your language or collect your own corpus via some datamining. Corpura are easily available for about 60 languages. Corpuses for rare language are likely missing, you will likely have to do some data mining.

Some research centers are curating the web to provide large corpura to linguists and netizens alike. 

* Corpus Crawler is an open-source crawler for building corpora in 1000+ languages (easy to add more, send pull requests); https://github.com/googlei18n/corpuscrawler
* P. Lison and J. Tiedemann (2016), ''"OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles"'', http://stp.lingfil.uu.se/~joerg/paper/opensubs2016.pdf . In Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC 2016)
** "The corpora is made freely available to the research community on the OPUS website" − Lison and Tiedemann (2016).
** http://opus.nlpl.eu/OpenSubtitles2018.php

==== Wiki(p)edia dumps ====
One possibility is to harvest Wikipedia's contents. See:
* [https://github.com/hugolpz/introduction-wikipedia-corpus Introduction-wikipedia-corpus]
* [https://github.com/google/corpuscrawler/issues/78 Discussion on harvesting Wikipedia]

== From corpus to frequency data `{occurences} {item}` ==
Main tools will be <code>[https://ss64.com/bash/grep.html grep]</code> to grab the text strings, <code>[https://ss64.com/bash/awk.html awk]</code> to count them, <code>[https://ss64.com/bash/sort.html sort]</code> to sort and rank them.

==== Characters frequency (+sorted) ====
<pre class="console">
$ grep -o '\S' longtext.txt | awk '{a[$1]++}END{for(k in a)print a[k],k}' | sort -n -r -t' ' -k1,1 &gt; sorted-letters.txt
</pre>

==== Space-separated Words frequency (+sorted): ====
<pre class="console">
$ grep -o '\w*' longtext.txt | awk '{a[$1]++}END{for(k in a)print a[k],k}' | sort -n -r -t' ' -k1,1  &gt; sorted-words.txt
# or 
$ awk '{a[$1]++}END{for(k in a)print a[k],k}' RS=&quot; |\n&quot; myfile.txt | sort -n -r -t' ' -k1,1 &gt; sorted-words.txt
</pre>

==== Loop on all .txt, recursively within folders ====
<pre class="console">
find -iname '*.txt' -exec cat {} \; | grep -o '\w*' | awk '{a[$1]++}END{for(k in a)print a[k],k}' | sort -n -r -t' ' -k1,1 &gt; sorted-words.txt
</pre>

==== Output ====
<pre class="console">
39626 aš
35938 ir
33361 tai
28520 tu'21th
26213 kad'toto
...
</pre>

== Cleaning up frequency lists ==
Most sources provide wordlists with <code>{number_of_apparitions}{separator}{item}</code> or its mirror <code>{item}{separator}{number_of_apparitions}</code>, already sorted from most frequent to less ones. We what to keep the field <code>{item}</code> and drop the <code>{separator}</code> and <code>{number_of_apparitions}</code>.

{| class="wikitable" style="width:100%"
! Display input we have || Display output we want
|- valign="top"
|
<pre class="console">
$ cat frequency-list.txt
39626 aš
35938 ir
33361 tai
28520 tu'21th
26213 kad'toto
...
</pre>
|
<pre class="console">
$ cat words-list.txt
# aš
# ir
# tai
# tu'21th
# kad'toto
# ...
</pre>
|-
!colspan=2| Command
|- 
|colspan=2|
<pre class="console">
cut frequency-list.txt -d$' ' -f2 | sed -E 's/^/# /g' > words-list.txt
# load file line by line, cut by space then keep field 2, replace start of line by # on all lines, print in file.
</pre>
|}
This final result is what you want for LinguaLibre [[Help:Create your own lists]].

== Additional helpers ==
=== Sort command ===
See <code>[https://ss64.com/bash/sort.html man sort]</code> for details. 
:<code>-n:</code> numeric sort
:<code>-r:</code> reverse (descending)
:<code>-t:</code> changes field separator to ' ' character
:<code>-k:</code> as <code>-k:1,1</code>, sort key starts on field 1 and ends on field 1

=== Counting lines of a file ===
<pre class="console">wc -l filename.txt       # -l : lines</pre>

=== See sample of a file ===
<pre class="console">head -n 50 filename.txt       # -n : number of line</pre>

=== Splitting a very long file ===
<pre class="console">
split -d -l 2000 --additional-suffix=".txt" YUE-words-by-frequency.txt  YUE-words-by-frequency-
</pre>
Words-lists files generally are be over 10k lines long, thus not convenient to run recording sessions. Given 1000 recordings per hour via LinguaLibre and 3 hours sessions being quite good and intense, we recommend sub-files of : 
* 1000 lines, so you use 1, 2 or 3 files per session 
* 3000 lines, so you use 1 file per session and kill it off like a warrior ... if your speaker and yourself survives.

See [https://stackoverflow.com/a/2016918 How to split a large text file into smaller files with equal number of lines in terminal?]
<!-- 
[How to select range of line from file via terminal ?](https://stackoverflow.com/questions/83329/how-can-i-extract-a-predetermined-range-of-lines-from-a-text-file-on-unix)
```
sed -n 1,1999p words-list.txt > words-list-0001-1999.txt
```
-->

=== Convert encoding ===
<pre class="console">
iconv -f "GB18030" -t "UTF-8" SUBTLEX-CH-WF.csv -o $iso2-words.txt
</pre>

=== Create frequency list from [[:en:Dragon]] ===
:''See also [https://codepen.io/hugolpz/pen/ByoKOK 101 Wikidata/Wikipedia API via JS]''
<pre class="console">
curl 'https://en.wikipedia.org/w/api.php?action=query&titles=Dragon&prop=extracts&explaintext&redirects&converttitles&callback=?&format=xml' | tr '\040' '\012' | sort | uniq -c | sort -k 1,1 -n -r > output.txt
</pre>

=== How to compare lists ? ===
:[Section status: Draft, to continue.] ([https://github.com/hugolpz/audio-cmn/blob/master/hsk-missing-audios.bash example]).

<pre>
 comm - compare two sorted files line by line
</pre>

== See also ==
* https://dumps.wikimedia.org

[[Category:Lingua Libre:Help]]