User
Psubhashish/tools/Prepare words for Lingua Libre
< User:Psubhashish
The HTML code below is for a converter to prepare words for Lingua Libre. It is created considering a language written in a writing system other than Latin. It has five major functions:
- Remove punctuation and convert words into new lines
- Remove words typed in Latin script
- Remove duplicates
- Sort alphabetically
- Replace lines with # signs
How to use this
Use any text editor to copy the HTML code in the box below. Save the file as a .HTML file (the default extension would be .txt otherwise). Open the .HTML file using any browser.
Your source text can be copied into the "Input Text" box below. You can then press 1. Remove punctuation & words to new line followed by the other buttons. After the fifth step, click "Copy text" below the "Output Text" box. Go to LinguaLibre's RecordWizard and paste the copied text to generate a word list.
<!DOCTYPE html>
<html>
<head>
<title>Prepare words for Lingua Libre</title>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
</head>
<body>
<h1>Prepare words for Lingua Libre</h1>
<label for="input-text">Input Text:</label><br>
<textarea id="input-text" rows="10" cols="70"></textarea><br>
<button onclick="removePunctuationAndWords()">1. Remove punctuation & words to new line</button><br/><br/>
<button onclick="removeLatinWords()">2. Remove Latin character words</button><br/><br/>
<button onclick="removeDuplicates()">3. Remove duplicates</button><br/><br/>
<button onclick="sortAlphabetically()">4. Sort alphabetically</button><br/><br/>
<button onclick="replaceLinesWithHash()">5. Replace lines with #</button><br/><br/>
<label for="output-text">Output Text:</label><br>
<textarea id="output-text" rows="10" cols="70"></textarea><br>
<button onclick="copyText()">Copy text</button><br><br/>
<script>
function removePunctuationAndWords() {
var inputText = document.getElementById("input-text").value;
// Replace punctuation with a space, except for hyphen and apostrophe
var textWithSpaces = inputText.replace(/[।“”\(\)\[\]\{\}<>.,\/#!$%\^&\*;:{}=\-_`~]+/g, " ").replace(/(^[-])|([-]$)/g, "");
// Convert spaces to new line breaks
var textWithLineBreaks = textWithSpaces.replace(/[\s]+/g, "\n");
document.getElementById("output-text").value = textWithLineBreaks;
}
function removeLatinWords() {
var inputText = document.getElementById("output-text").value;
var outputText = inputText.replace(/[^\u0B00-\u0B7F\s]+/g, "");
document.getElementById("output-text").value = outputText.trim();
}
function removeDuplicates() {
var inputText = document.getElementById("output-text").value;
var words = inputText.split(/\s+/);
var uniqueWords = [];
for (var i = 0; i < words.length; i++) {
if (uniqueWords.indexOf(words[i]) === -1) {
uniqueWords.push(words[i]);
}
}
var outputText = uniqueWords.join("\n");
document.getElementById("output-text").value = outputText.trim();
}
function sortAlphabetically() {
var inputText = document.getElementById("output-text").value;
var words = inputText.split(/\s+/);
words.sort();
var outputText = words.join("\n");
document.getElementById("output-text").value = outputText.trim();
}
function replaceLinesWithHash() {
var inputText = document.getElementById("output-text").value;
var outputText = inputText.replace(/\n+/g, "#");
document.getElementById("output-text").value = outputText.trim();
}
function copyText() {
var outputText = document.getElementById("output-text");
outputText.select();
document.execCommand("copy");
alert("Text copied to clipboard!");
}
</script>
</body>
</html>
Note
- This tool works for the Odia language, written in the Odia script. You'll need to find the Unicode block for your language in case your language does not use the Latin script, and edit the line number 36:
var outputText = inputText.replace(/[^\u0B00-\u0B7F\s]+/g, "");and replace "u0B00" and "u0B7F" with the ranges for your script.