Santhosh Thottingal

Goodbye LJ!

Sun, 14 Jun 2009 06:36:41 GMT

Good Bye Livejournal..
I am moving to my new home : http://thottingal.in/blog
Friends, Please update your bookmarks, feed subscriptions etc..
Migrating all livejournal posts was too easy with new version of wordpress.

Openoffice Indic Regional Language group

Tue, 26 May 2009 15:59:23 GMT

We just formed Indic Regional Language group for Openoffice. This is as per the Openoffice Native Language Consortium Plans. The objectives of such groups can be read from here. Basically the group is meant for better coordination among Indic languages to make Openoffice experience in our language better.
The announcement of this group is here

Thanks to Charles-H. Schulz, we got a mailing list indic@native-lang.openoffice.org. To subscribe login to http://native-lang.openoffice.org

We just started, and I will soon setup a wiki page there. To start with , I will collect the list of issues pending for Indian languages from people from various languages and will find out people from various languages as point of contacts. Feel free to contact me for anything related to Openoffice in your language.

Update: June 3, 2009: This is our wiki page

In solidarity

Wed, 13 May 2009 10:04:23 GMT

Python isalpha is buggy

Sun, 29 Mar 2009 12:09:00 GMT

This code

#!/usr/bin/env python
# -*- coding: utf-8 -*-
ml_string=u"സന്തോഷ്  हिन्दी"
for ch in ml_string:
    if(ch.isalpha()):
        print ch

gives this output

സ
ന
ത
ഷ
ह
न
द

And fails for all mathra signs of Indian languages. This is a known bug in glibc. Does anybody know whether python internally use glibc functions for this basic string operations or use separate character database llke QT does?

N-gram Visualization Experiment

Sat, 28 Mar 2009 14:04:59 GMT

Following image shows the python-graphviz generated visualization of N-Gram representation of first paragraph this article from Hindi wikipedia. The image represents the possible paths through which a sentence can be constructed if we start from a word भारत.
Click to view the enlarged image

Localization: What are we missing?

Wed, 25 Mar 2009 17:10:42 GMT

[This blog post is kind of self criticism and written not forgetting the valuable contribution that l10n communities are doing. ]
Some observations on the Localized desktops in Indian Languages
* Not all localization team members try the application that he/she translate at least once before working on the PO file. Result: If somebody does the localization without understanding what the application does and try the en_US interface, he/she miss the context of the strings. An example I have seen : the string "Querying" was translated to xx_IN language string which means "Questioning" instead of the required string corresponding to "Searching". Sometimes we miss to understand how much space the string is going to take in the screen and we translate a small English word to a long xx_IN string to make the meaning clear. Result: Ugly interface.

Tamil gedit from Ubuntu 8.10(Click to enlarge)
* Not all localization team members *try* the application that he/she translated after completing the PO file or even after the application is released. This happens when he/she translates many applications(sometimes if it is part his/her job).
* Practically, there is no process called *testing* localized desktop in our SDLC. L10N members translates a PO file and sometimes he/she translates it as text file rather than a user interface. It is must that we should bring some process to make sure that the localized desktop is tested for usability, contextually correct translation, spelling mistakes, wrong short cut keys, fuzzy strings , non translated strings in main interface etc etc.
* Since the ratio between the total number of applications in a desktop environment and number of team members is very less, we end up in translating one application by many people. Result: inconsistent translation and no ownership for ensuring the translation quality. Ramadoss from Tamil team was suggesting that ideally , for each application there should be a person from each language , who is responsible for timely translation, testing. He can take more than one application responsibility but not more than say, 10. Practically, this requires a big l10n community per language and unfortunately we don't have it as of now.
* Peer review, one of the important and mandatory process in l10n is not happening properly when the release date is approaching. L10N communities often try to meet the percentage of completion somehow. IMHO, the new l10n tools frameworks often miss to give importance for peer review in the workflow they design. FOSS community, being inclusive in nature often welcomes new l10n contributors. I have seen many members improving their l10n skills after making the corrections as per the review comments from others. When a new l10n workflow allows every contributor to submit their translated PO file without the peer review from community, the ultimate result is very bad user interface. We have seen this many times with Rosetta translations of Ubuntu. Everybody going there tries out the Rosetta "features" and leave few strings "translated" there. And Ubuntu takes those strings for their immediate release. Upstream translations are never taken on time or the "translated" strings are never submitted to upstream. Result: Very bad localized desktop with many spelling mistakes, inconstant translations etc.. We ml_IN team used to watch who is "contributing" through Rosetta and make him work with the community. I hope the new translation frameworks will give sufficient attention to this problem. If we are not keeping a balance between newbie translation and quality assuarance , our localized desktops will not improve.

Again Tamil gedit, but from Debian Lenny. Compare it with the Ubuntu version shown above(Click to enlarge)

* User feedback: The number of users who use the desktop in their own mother tongue, even though the % of translation is more than 80% for many languages, is very less. IMHO, It is because of a 'dependency conflict' of the following things
a) A person who is not good in English
b) A person want to use computer in mother language for some "purpose"
c) A person who is capable of spending Rs ~20K for a computer
Most of the cases, there is a conflict between any 2 of the following and that ends up in a) Person use his desktop in english b) Person not using computer at all. I am sure that if there are good number of users, we will not end up in interfaces I showed in the screen shots.
* One inconsistency I noticed across localized desktops is regarding the shortcut keys/accelerator keys. Some languages use English short cut keys and give at the end of the word in Brackets for eg: അടയ്ക്കുക (C). As you can see in screen shots sometimes we have small letter and sometimes capital letter for that. Some languages use letters in xx_IN itself. But there is no consistency. For Control and Alt keys, some language translate them, some others keep it in English. What is the problem with English short cut key? For using English short cut key , the user should be using English layout keyboard. For shortcut keys in xx_IN, one should be using xx_IN keyboard layout. For a user(assume that he use xx_IN desktop since he is not good in English) typing in xx_IN in gedit using xx_IN keyboard, is it possible to use the short cut keys if we give in English? Are we expecting that for using short cut key while typing the document, he change switch his keyboard layout ? (btw, anybody noticed that Apple doesn't use Accelerator keys in its OS?)

bn_IN gedit in Ubuntu 8.10(Click to enlarge)

ml_IN gnome dictionary client in Ubuntu 8.10(Click to enlarge)

Suggestion/Ideas are welcome... How can we make our localized desktop more beautiful and user friendly?

Updates...

Sun, 11 Jan 2009 05:53:38 GMT

Praveen prepared videos from the matrix screen savers in 6 languages
This video is translated to Malayalam. For those who are interested in how to do that refer this
I prepared the glibc collation table for Malayalam . But still some more bugs to be fixed
We friends are working on adding Saka year system to KDE calendar system and it is almost ready . And here is the video : Saka calendar in KDE
Dict based english-malayalam dictionary is in developement and we are ready for a beta release. Rajeesh did a woderful job in preparing it
KDE Malayalam team is doing great job
Worked on hunspell to handle the agglutinative nature of Indian languages and found some problems with hunspell. Hunspell developer Nemth is looking into that
Sujith added Malayalam support to KLetters
Planning to attend fossmeet @ Model Engineering College, Cochin

KDE Indic Screensavers

Sun, 21 Dec 2008 16:32:03 GMT

I ported all of the Matrix screensavers with Indian language glyphs to KDE4. For details about the screensavers please read:

Download the binary packages: Deb package, and RPM package

There are 6 screensavers in that package, for Malayalam, Hindi, Oriya , Bengali, Tamil and Gujarati. After installation, goto KDE system settings->Desktop->Screensaver and select any of this.

Screenshots(click to get the image in original size):

KDE Screensaver configuration for Hindi:

Enjoy...!

Hyphenation of Indian Languages in Webpages

Tue, 16 Dec 2008 17:04:15 GMT

In my last blogpost I explained hyphenation of Indian language text in openoffice. In this blogpost I will explain how hyphenation can be done in webpages.

As I explained importance of hyphenation come into picture when we justify the text. The length of the lines are controlled by the parent tags.... Unicode had defined a special character called soft hyphen for hyphenation denoted by  . In HTML, the plain hyphen is represented by the "-" character (- or-). The soft hyphen is represented by the character entity reference  ( or )

User agents-browsers can break the line whenever a soft hyphen is found. So if we have a javascript based implemenation, which insert the softhyphen in between the words based on language specific rules, we can achieve hyphenation in webpages too.

Hyphenator is a project which does exactly the same. "Hyphenator.js brings client-side hyphenation of HTML-Documents on to every browser by inserting soft hyphens using hyphenation patterns and Frank M. Liangs hyphenation algorithm commonly known from LaTeX and Openoffice. "

Hyphenator was not tested for any non-latin languages so far. I tried to add support for Indian languages and the result was satisfactory. I used the same rules I defined for openoffice. Unlike latin languages, the number of hyphenation patterns for Indian languages is very less and the performance is good because of that.

I have added Malayalam, Tamil, Hindi, Oriya, Kannda, Telugu, Bengali, Gujarati and Panjabi support to it. You can see a working example here. (I wanted to embed one example here. But livejournal doesnot allow javascript inside blog body ). The column layout is done by CSS. Try resizing the browser windows and try a print preview too..

Don't forget to read the source code of that page. It is very simple. If you want hyphenation in your webpage, all you need is to include the javascript as done in the example. We need to provide the lang attributes for nodes so that the required patterns for that language can be loaded. I placed the new language patterns temporarily in download area of SMC. I will ask the author of Hyphenator to include it in upstream itself. Code is available here

Update(18-Dec-2008):Thanks to Mathias Nater, author of hyphenator, the patterns were added to upstream.

Hyphenation of Indian Languages and Openoffice

Sat, 13 Dec 2008 14:11:25 GMT

What is Hiphenation?

Hyphenation is the process inserting hyphens in between the syllables of a word so that when the text is justified, maximum space is utilized.

Hiphenation is an important feature that DTP softwares provide. For Indian languages there is no good DTP softwares available. XeTex is the only choice to work with unicode and professional quality page layout. But xetex and DTP are not exactly same. Inkscape can be used as temporary solution. But only for small scale works. There is a project going on to add Harfbuzz backend to Scribus, the freedomware DTP package.

Hiphenation is also requred in many other places. Actually it is required where ever we 'justify' a block of text in openoffice or any wordprocessors. Same is the case of webpages. If we justify a block of text in ml_IN, let is see what is happening now

Note the long gaps between words. This is a screenshot taken from firefox. The default hiphenation just breaking the lines in space characters. And no doubt that it makes the pages ugly. The problem becomes worse if the length of the word is more and column width is less.

So what is the solution?

Ideal solution : Applications should be aware of the language, its hiphenation rules and should to the hiphenation wherever required.

Openoffice can take hiphenation dictionaries just like spell checkers. But for Indian languages, we are yet to prepare hiphenation dictionaries(more on that later.) . CSS3 draft of w3c has a provision for hyphenate. But it is stil in draft stage

Algorithm For Hiphenation

The basic for all hyphenation algorithms is the hyphenation algorithm, designed by Frank Liang in 1983, which is adopted in TeX. Wikipedia artcle of TeX explain this with very simple example

If TeX must find the acceptable hyphenation positions in the word encyclopedia, for example, it will consider all the subwords of the extended word .encyclopedia., where . is a special marker to indicate the beginning or end of the word. The list of subwords include all the subwords of length 1 (., e, n, c, y, etc), of length 2 (.e, en, nc, etc), etc, up to the subword of length 14, which is the word itself, including the markers. TeX will then look into its list of hyphenation patterns, and find subwords for which it has calculated the desirability of hyphenation at each position. In the case of our word, 11 such patterns can be matched, namely 1c4l4, 1cy, 1d4i3a, 4edi, e3dia, 2i1a, ope5d, 2p2ed, 3pedi, pedia4, y1c. For each position in the word, TeX will calculate the maximum value obtained among all matching pattern, yielding en1cy1c4l4o3p4e5d4i3a4. Finally, the acceptable positions are those indicated by an odd number, yielding the acceptable hyphenations en-cy-clo-pe-di-a. This system based on subwords allows the definition of very general patterns (such as 2i1a), with low indicative numbers (either odd or even), which can then be superseded by more specific patterns (such as 1d4i3a) if necessary. These patterns find about 90% of the hyphens in the original dictionary; more importantly, they do not insert any spurious hyphen. In addition, a list of exceptions (words for which the patterns do not predict the correct hyphenation) are included with the Plain TeX format; additional ones can be specified by the user.

For more details about the algorithm used in Openoffice read this paper by Nemeth Laszlo

Hiphenation in Indian languages.

Unlike English or any other languages, hiphenation in Indian languages are not that much complex. In general following are the rules

[consonant][vowel][consonat] can be hiphenated as [consonant][vowel] - [consonat] if vowel is not a virama or halant
Dont split a word after ZWJ
We can split a word after ZWNJ
plus any language specific rules. For eg: in ml_IN a line should not start with a chillu letter.

Hiphenation Dictionaries for Indian languages.

Based on the above mentioned rules, Let us try to create hiphenation dictionaries for Indian languages. I will explain this with the help of a Hindi word example: अनुपल्ब्ध. We have to define the following rules in the dictionary for this
अ1 -> 1 is odd number , ie. word can be splitterd after अ
ु1 -> 1 is odd number , ie. word can be splitterd after ु
1ल -> 1 is odd number , ie. word can be splitterd before ल
1प -> 1 is odd number , ie. word can be splitterd before प
1ब -> 1 is odd number , ie. word can be splitterd before ब
्2 -> 2 is even number , ie. word can NOT be splitterd after ्
1ध -> 1 is odd number , ie. word can be splitterd before ध
So the end result is अ+नु+प+ल्ब्ध

Same way we can create the Hyphenation dictionaries for all other languages. I have prepared the Hyphenation dictionaries for 8 Indian Languages. Download it from the git repo of the SMC.
How to Install a xx_IN hyphenation dictionary.

Copy the hyphenation dictionay hyph_xx_IN to /usr/share/myspell/dicts folder.
Create a file at /usr/share/myspell/infos/ooo/ folder named openoffice.org-hyphenation-xx with one line content
HYPH xx IN hyph_xx_IN
Run this command sudo update-openoffice-dicts

Open the openoffice writer, Open some fille in your language or type some text. Justify the text. Set the language of the selection by using Tools->Language menu Hiphenate it by using Tools->Language->Hiphenation menu.

Hope it works :). I tested only Hindi and Malayalam. For other languages , inform me if you see any problems or if it is not working . Here is the hyphenated Malayalam paragraph. Compare it with the image I showed at the beginning of this blogpost

Ok. so after testing these hyphenation dictionaries, if we provide them to upstream and packaged, hyiphenation problems in openoffice is solved. :)

But.... How to solve this problem in web pages?!. We will discuss it in next blogpost!
PS: Thanks to Nemeth Laszlo , author of Hunspell and Openoffice Hyphenation for helping me to prepare the hyphenation tables.

Update(Apr 16,2009) The hyphenation dictionaries were packaged for Fedora and will be part of Fedora 11

Yahoo search bug

Fri, 05 Dec 2008 14:49:41 GMT

None of the search engines can handle Indian languages very well. Google removes the zero width joiners, non joiners , that are used in many languages. Yahoo doesnot remove it. But a UI bug in webpage makes the results wrong..
See the below image:

The bottom half of the image is the source code. We can clearly see that the closing bold tag is placed in between the word instead of putting at the end of the word. As a result, the word is rendered wrong in the page.
This happens for all languages which use ZWJ, ZWNJ, ZWS etc. It breaks the word just before the zwnj/zwj and puts the end of bold tag to highlight the search result..

I showed this to Gopal and told me that he filed a bug on that.

KDE spellchecker not working for Indian Languages

Sun, 30 Nov 2008 12:33:30 GMT

As I mentioned in my blog post on Language detection the sonnet spellchecker of KDE is not working. I read the code of the Sonnet and found that it fails to determine the word boundaries in a sentence (or string buffer) and passes the parts of the words to backend spellcheckers like aspell or hunspell. And eventually we get all words wrong. This is the logic used in Sonnet to recognize the word boundaries

Loop through the chars of the word, until the current char is not a letter/ anymore.

And for this , it use the QChar::.isLetter() function. This functions fails for Matra signs of our languages.

A screenshot from a text area in Konqueror:

For example


#include <QtCore/QString>
#include <stdlib.h>
int main(){
	QChar letter ;
	letter = 'அ';
	fprintf(stdout,"%d\n", letter.isLetter());
	letter = 'ी';
	fprintf(stdout,"%d\n", letter.isLetter());
}

In this program, you will get true as output for அ and false for ी.

When I showed this to Sayamindu during foss.in , he showed me a bug in glibc . Eventhough the bug is about Bengali, it is applicable for all languages. It is assigned to Pravin Satpute and he told me that he got a solution and will be submitting soon to glibc.

But I am wondering why this bug in KDE unnoticed so far? Nobody used spellcheck for Indian languages in KDE?!

Let me explain why this is not happening in GNOME spellchecker if this is a glibc bug. In gnome, this word splitting will be done in application itself using gtk_text_iter_* and these iteration through words are done by pango words boundary detection algorithms.

Filed a bug in KDE to track it.

Youtube to MPEG or Ogg video conversion

Sat, 22 Nov 2008 17:12:48 GMT

Here is the two line method to convert a youtube video to oggvorbis video.
Locate clive and ffmpeg2theora in your package and install
$clive http://in.youtube.com/watch?v=6JeZ5oeAEyU (replace this with the youtube address you want) It will create a flv file.
Convert to mpeg video file
$ffmpeg -i AmericaAmerica.flv AmericaAmerica.mpg
Convert to ogg video file
$ffmpeg2theora AmericaAmerica.mpg (replace it with the name of the flv file the previous command created)
Done. You can see the .ogg file in the directory from where you executed the above commands

Dhvani 0.94 Released

Sat, 15 Nov 2008 15:35:32 GMT

A new version of Dhvani -The Indian Language Text to Speech System is available now. The new version comes with the following improvements/features

Support for 11 languages- Hindi, Panjabi, Gujarati, Marati, Bengali, Oriya, Telugu, Kannada, Tamil , Malayalam and Pashto(Afganistan)
Pitch and Tempo modification for speech
Direct ogg-vorbis speech output and optional wav output format

C/C++ APIs for applications to use dhvani as a shared library.
Generic driver for Speech-dispatcher and Integration to Orca through speech dispatcher
Python binding through speech dispatcher
Improved language detection algorithm

Dhvani documentation is available here. Binary packages and source code are available here
Thanks

Rahul Bhalerao for Marathi module and patches
Zabeehkhan for Pashto Module
Nirupama, CDAC Chennai and CDAC Noida people for testing and reporting bugs
NRCFOSS Chennai, Krishnakanth Mane and many others for feedbacks
Amida Simputer team for patches on Telugu module especially the Telugu number reading logic
Debayan and Roshan for testing and informing problems

There was good amount of code change in this version. Still there are many improvements to do in language modules and synthesizer. Some of the language modules requires developers who speak that language. Syntheziser got some improvements and require some amount of research to make the speech more natural. So your feedbacks, suggestions, bug reports and patches are valuable.

PS: A note for quick usage after installation from binary: After installing deb or rpm, Open gedit, edit->preferences->plugins, enable external tools. Dhvani will be available as a plugin there. Select some text in any of the supporting languages and click the Dhvani menu.

Language Detection and Spellcheckers

Thu, 13 Nov 2008 16:15:06 GMT

A few weeks back there was a discussion on #indlinux IRC channel about automatic language detection. The idea is, spellcheckers or any language tools should not ask the users to select a language. Instead, they should detect the language automatically. The idea is not new. There is a KDE bug hereand Ubuntu has this as an brainstorm idea. It seems M$ word already have this.

A sample use case can be this: "While preparing a document in Openoffice, I want to write in English as well as in Hindi. For doing spellcheck, I need to manually change the language rather than the application detect it automatically"

Regarding the algorithm behind automatic language detection, there are many approaches. Statistical approaches are effective for languages sharing same script(For eg: languages which use latin script or Hindi and Marathi). N-gram based methods are used in statistical approach. Here is a 'patented' idea . And this page explains a character trigram approach. Google has a language detection service(http://www.google.com/uds/samples/language/detect.html) and it seems it is still in development or 'learning stage'.

Here is an example of statistical language detection: languid(It did not work for me when I tried, But you can download the source code and check)

Sonnet is the spellchecker framework of KDE written by J. Rideout. It is also trying to provide the language detection feature. Here is an old article in linux.com about that. It is based on n-gram based text categorization and is a port of languid. From the article:

A gram is a segment of text made of N number of characters. Sonnet uses trigrams, made from three characters. By analyzing the popularity of any given trigram within a text, one may make assumptions about the language the text is written in. Rideout gives an example: "The top trigram for our English model is '_th' and for Spanish '_de'. Therefore, if the text contains many words that start with 'th' and no words that start with 'de,' it is more likely the text is in English [than Spanish]. Additionally, there are several optimizations which include only checking the language against languages with similar scripts and some heuristics that use the language of neighboring text as a hint."

(I tried sonnet and could not get it working for ml_IN. Instead of words, it was iterating through letters. Anyway I will check this problem later.)

As far as Indian languages are concerned, Unicode code range based language detection will work for most of the cases. Most of the languages has its own script and Unicode code point range. For example, detecting Malayalam is a matter of checking the letters are in the Malayalam Unicode range. But for Devanagari script it is not straight forward. Hindi , Marathi etc use Devanagari script. Dhvani, the text to speech system for Indian languages use a simple algorithm for language detection(http://dhvani.sourceforge.net/doc/language-detection.html). There the Hindi and Marathi is identified by giving a priority for LANG environment variable. But it will fail if somebody try to use Marathi in an English desktop(Users can specify the language to be used – In that case language detection will not be done.).

In the case of spell checkers other than LANG environment variable there are other options. When you type in gedit or any text editors, detecting the keyboard layout will be one way of detecting the language. But it depends which IME the users uses. It can be xkb or scim or even a copy-paste.

Anyway, it is pretty clear that the current natural language features in the free desktops requires more improvements. Based on a discussion we had in #indlinux IRC, we had setup a wiki page here to discuss on this.

As a proof of concept, I tried to write a spellchecker for Gedit texteditor with language detection for Indian languages. Basically it uses Unicode character range. It is a gedit plugin written in python. And it uses pyenchant spellcheck wrapper library. Install python-enchant using your package manager if it is not already installed. Download the plugin and python module to ~/.gnome2/gedit/plugins folder and restart gedit. Enable external tools and new Spellchecker plugin in edit->preferences->plugins. It does not have the pango error style underline or suggestions in context menu as of now. It just prints the results and suggestions in the console of gedit. And ‘Add to Dictionary’ etc are not there now.

I would like to request interested developers to come forward and make this feature ready to use in free desktops. Suggestions are welcome. We need good algorithms for detecting the language too.
A sample use case: "System locale is English and I am typing a document in Hindi and want to write some Marathi sentences in between. Without manually changing the language, system detect the language of each word and check the spelling against corresponding dictionaries."

PS: Because of the inflectional and agglutinative nature of some of the Indian languages, the spell checking is not at all effective. I will write on that later.

Gedit plugin for showing unicode codepoints

Tue, 11 Nov 2008 17:26:01 GMT

While working with Unicode text, it is often required to get the Unicode code points of text for debugging. Using python, it is very easy to get the unicode codepoints of the text. Following examples illustrates it.


>>> "സന്തോഷ്".decode("utf-8")
u'\u0d38\u0d28\u0d4d\u0d24\u0d4b\u0d37\u0d4d'


>>> str=u"സന്തോഷ്"
>>> print repr(str)
u'\u0d38\u0d28\u0d4d\u0d24\u0d4b\u0d37\u0d4d'

Well, But we need to take python console and type/paste the text etc..How can we make it more easy? What if pressing F12 key after selecting some text gives the codepoints?
So I wrote a plugin for gedit. I never knew that writing a gedit plugin is too easy. This tutorial gives all the required information.
Download the plugin file and python module and place it in .gnome2/gedit/plugins folder inside your home folder. And restart gedit. Enable the plugin from Edit->Preferences->Plugins menu. Note that you need to enable the External tools plugin too.

Select some text and press F12. If text is not selected, entire content of the document will be used.

Screensavers in your language

Mon, 27 Oct 2008 11:17:26 GMT

I had written a blog post about hacking the glmatrix screensaver with the glyphs of our languages.

Now I have those screensavers in the following languages:

Hindi : Deb Package , RPM

Gujarati : Deb Package , RPM

Bengali : Deb Package , RPM

Oriya: Deb Package , RPM

Tamil : Deb Package , RPM

Malayalam: Deb Package , RPM

Try it and enjoy !!
ps: I used the default fonts of Fedora 9 for these. If you have any specific font to be used please let me know. I used Dyuthi calligraphic font for Malayalam.

Swanalekha M17N based Input Method for 11 Languages

Mon, 27 Oct 2008 07:31:00 GMT

Swanalekha is an Input method originally designed for Malayalam. It is works with scim. as well as m17n. The input method scheme is transliteration based and it has a unique feature of candidate list menu(which I will explain shortly). Now I have extended it to 10 other Indian languages.

Before explaining how swanalekha is different from other phonetic/transliteration based input methods, let me explain some of the characteristics of transliteration. Transliteration based input methods were following a strict one to one mapping from english letters to another Indian language. For eg: The ka=क ,pa = प , ti = टि etc.. when you write bharath, you will easily transliterate it to hindi as भारत. But for a rule based transliteration system it is भरत unless the english is bhaarath. Some times it may be Bhaarat too.. See another example: Kartik. it should be transliterated to കാര്‍ത്തിക് in Malayalam. So some people write it as Karthik, and some others write it as karthick too. All these are based on personal preferences. But when it use transliteration based input methods, people find difficulty with using a strict rule based writing method. There they have to write kaa for കാ or કા or கா or কা. Users like to get what they mean without the difficulty following the strict rules of transliteration. In an Intelligent transliteration based system when somebody write linux they should be able to map it to लिनक्स . Some times a choice to select लैनक्स is also preferable. This is what google transliteration does. No rules, no learning.. just type in english...

Google's Transliteration is based on machine learning and statistical approach. And it works only when we are online and only in webpages. Now I will explain how swanalekha tries to provide a solution for the above problem.
For each english letter or pattern , we saw that there are multiple choices . ka can be क, का . ga can be ക, ഗ, ഖ, ഘ, ഗാ in Malayalam. sa can be स, श etc.. So swanalekha provides all these candidates as a suggestion menu under the cursor while typing. See the below image of Hindi swanalekha version.

The differences between google transliteration and swanalekha are:
a) Google transliterate is web based and works in webpages when you are online. Swanalekha works in all applications in your GNU/Linux desktop such as gedit, openoffice, kwrite, firefox...
b) Google transliterate gives suggestions as words, but swanalekha works in letter level (not exactly a single letter. but like का, કા etc. )
c) Google transliterate is machine learning based. But swanalekha is rule based with 'one to many' pattern mapping in m17n

The candidates are mapped to English string patterns inside the source code- the m17n input method files - .mim files.
You can download the .mim files from here. Icons for each language is also provided. You can see .mim files for Malayalam, Hindi, Telugu, Oriya, Tamil, Bengali, Assamese, Panjabi, Gujarati, Marathi and Kannada. Note that other than Malayalam all other source files are not complete. They are generated using a small python script from Malayalam mapping file. They are just templates with approximate mapping. And should be corrected and modified by a person who know that language very well. Malayalam mapping is tested and it is already packaged for Fedora and already present in m17n upstream as part of m17n-contrib package. It is widely used by GNU/Linux users in Kerala too.
Candidate selection based Input methods are very common in CJK(Chinese, Japanese, Korean) languages. Swanalekha is first implementation of candidate list outside CJK using scim and m17n.

So if anybody is interested in testing and correcting the mappings for your language, please continue reading :)

How to Install :
download the tar ball containing all .mim files and icons from here. Extract it and copy all .mim to /usr/share/m17n
sudo cp *.mim /usr/share/m17n
sudo cp *.png /usr/share/m17n/icons

Note that you need to install scim-m17n before doing this. Most of the distros will have it pre installed
After copying these , restart your X by pressing alt+ctrl+del or do a logout+login
Open gedit, select input method as scim, and select your language from the scim menu. Start typing

How to correct the maps?
Open the .mim file for your language using any text editor.
You will see lines in lisp syntax. No, You need not know Lisp :)
For example in hi-swanalekha.mim, you will see a line like this
("sa" (("स") ("श")))
This means, for 'sa', show स and श as candidates with स as default option. If you want to add सा as third option just change the line like this
("sa" (("स") ("श") ("सा")))
If any pattern is not found in .mim file just add one more line there following the above syntax. Only thing is you should be careful about opening and closing parenthesis since it is Lisp.

Once you are done, install it by just copying it to /usr/share/m17n folder. Restarting X is required to restart scim. or even a 'killall scim' will do sometimes
Don't change any other code(code for candidate selection using up/down arrow, and using number keys) unless you know what you are doing.

Let me know if you face any issues..

Happy Hacking and Happy Deepavali !!!

Geo-visualisation, the FOSS way

Thu, 04 Sep 2008 16:38:05 GMT

My friend Jaisen Nedumpala has been developing a Geo-visualisation system for Cheruvannoor Grama Panchayath(Page in ml_IN) of Kerala. The system, developed using FOSS tools is available here

"Development of effective geo-visualisation based decision support system (DSS) involved primarily data compilation from collateral sources, setting up appropriate hardware configuration, design of database and design of a spatial DSS. "

Jaisen used softwares like GRASS, UMN MapServer and ka-Map. He has written a detailed documentation(English) on how he developed this and what are all the tools used.

UTF8Decoder

Sun, 31 Aug 2008 15:56:21 GMT

zabeehkhan was trying to code a Pashto (ps_AF) module for dhvani. And he told me that "it is not saying anything" :). So I took the code and found the problem. Dhvani has a UTF-8 decoder and UTF-16 converter. It was written by Dr. Ramesh Hariharan and was tested only with the unicode range of the languages in India. It was buggy for most of the other languages and there by the language detection logic and text parsing logic was failing. So I did some googling, went through the code tables of gucharmap and got some helpful information from here and here
So here is my new UTF8Decoder and converter

/*
UTF8Decoder.c
This program converts a utf-8 encoded string to utf-16 hexadecimal code sequence

UTF-8 is a variable-width encoding of Unicode.
UTF-16 is a fixed width encoding of two bytes

A UTF-8 decoder must not accept UTF-8 sequences that are longer than necessary to
encode a character. For example, the character U+000A (line feed) must be accepted from
a UTF-8 stream only in the form 0x0A, but not in any of the following five possible overlong forms:

  0xC0 0x8A
  0xE0 0x80 0x8A
  0xF0 0x80 0x80 0x8A
  0xF8 0x80 0x80 0x80 0x8A
  0xFC 0x80 0x80 0x80 0x80 0x8A

Ref: UTF-8 and Unicode FAQ for Unix/Linux http://www.cl.cam.ac.uk/~mgk25/unicode.html

Author: Santhosh Thottingal <santhosh.thottingal at gmail.com>
License: This program is licensed under GPLv3 or later version(at your choice)
*/
#include<stdlib.h>
#include<stdio.h>
#include<string.h>
unsigned short
utf8_to_utf16 (unsigned char *text, int *ptr)
{

  unsigned short c;		/*utf-16 character */
  int i = 0;
  int trailing = 0;
  if (text[*ptr] < 0x80)	/*ascii character till 128 */
    {
      trailing = 0;
      c = text[(*ptr)++];
    }
  else if (text[*ptr] >> 7)
    {
      if (text[*ptr] < 0xE0)
	{
	  c = text[*ptr] & 0x1F;
	  trailing = 1;
	}
      else if (text[*ptr] < 0xF8)
	{
	  c = text[*ptr] & 0x07;
	  trailing = 3;
	}

      for (; trailing; trailing--)
	{
	  if ((((text[++*ptr]) & 0xC0) != 0x80))
	    break;
	  c <<= 6;
	  c |= text[*ptr] & 0x3F;
	}

    }
  return c;

}


/* for testing */
int
main ()
{
  char *instr = "സന്തോഷ് തോട്ടിങ്ങല്‍";	/* my name :) */
  int length = strlen (instr);
  int i = 0;

  for (; i < length;)
    {
      printf ("0x%.4x ", utf8_to_utf16 (instr, &i));
    }
  printf ("\n");
/* output is:
0x0d38 0x0d28 0x0d4d 0x0d24 0x0d4b 0x0d37 0x0d4d 0x0020 0x0d24 0x0d4b 0x0d1f 0x0d4d 0x0d1f 0x0d3f 0x0d19 0x0d4d 0x0d19 0x0d32 0x0d4d 0x200d 
*/

  return 0;
}

There may be already existing libraries for this, but writing a simple one ourself is fun and good learning experience. For example, in python, to get the UTF-16 code sequence for a unicode string, we can use this:


str=u"സന്തോഷ്‌"

print repr(str)

This gives the following output


u'\u0d38\u0d28\u0d4d\u0d24\u0d4b\u0d37\u0d4d'

Say NO to Software Patents

Tue, 19 Aug 2008 15:12:52 GMT

say_namaskaar.c

Sat, 26 Jul 2008 16:33:31 GMT

/* say_namaskaar.c
 *  This is a sample C code using dhvani text to speech API which I am 
 *  developing now and planning to release soon. New version of dhvani 
 *  will provide a shared library libdhvani and it allows other C or C++
 *  applications to use dhvani synthesizer. Tamil and Marathi modules, pitch, tempo 
 *  control etc are the features for the coming release.
 *  I need to prepare documentation, fix many bugs, test, commit the files in cvs ...
 *  Looking for some free time for all these...
 *  Visit http://dhvani.sourceforge.net
 */

/* compile with gcc -ldhvani -o namaskaar say_namaskaar.c */
#include <dhvani/dhvani_lib.h>
int main(int argc, char *argv[]) {
    dhvani_options options;
    /* Set the pitch and tempo of the speech */
    options.tempo = -10.0; /* reduce the speed by 10%  */
    options.pitch = 2.0;    /* increase the pitch b 2 semitons */
    options.rate = 16000;  /* 16KHz Sampling rate */
    /* Initialize dhvani */
    dhvani_init(&options);
    /* Say Namaskar */
    dhvani_say("नमसकार",  &options);
    /* close the synthesizer */
    dhvani_close();
    return 0;
}
 
/*  We can write a blog post in C too :P . Syntax highlighted by Code2HTML */

Dhvani Now Speaks Marathi

Tue, 24 Jun 2008 16:48:31 GMT

Thanks to Rahul Bhalerao , he wrote the Marathi module for dhvani- The Indian Language Text to speech System. Dhvani can speak 10 Indian languages now: Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Oriya, Panjabi, Tamil, and Telugu.

Rahul also gave some patches for hindi module and for some other bugs. The code is available in CVS.

The automatic language detection algorithm will not work for Marathi since it uses the devanagari script and I have assigned the unicode range used for language detection to Hindi. So it requires a langauge switch like "dhvani -l mr inputfile"

Many new features for dhvani are in development, incluiding pitch and tempo control of the generated speech. And I am trying to improve the code quality too.

I had demonstrated the tamil module at NRCFOSS, AU-KBC centre, chennai a few days back and Amachu offered help for improving the tamil pronunciaton rules.

For those are interested in Marathi module, I have some sample speech files generated by dhvani in ogg format. The text is taken from an article about Marathi langauge in Marathi wikipedia. Here is the article and here is the exact text used for the speech
1. With default pitch and tempo- Male voice
2. Female voice by positive pitch shift- A feature in development

Dhvani has an IRC channel now: #dhvani at freenode

Canonical Equivalence in Unicode: Some notes

Mon, 02 Jun 2008 16:51:43 GMT

Some Notes on Canonical Equivalence in Unicode:
Unicode defines canonical equivalence as follows: From UAX #15

Canonical Equivalence
This section describes the relationship of normalization to respecting (or preserving) canonical equivalence. A process (or function) respects canonical equivalence when canonical-equivalent inputs always produce canonical-equivalent outputs. For a function that transforms one string into another, this may also be called preserving canonical equivalence. There are a number of important aspects to this concept:

The outputs are not required to be identical, only canonically equivalent.

Not all processes are required to respect canonical equivalence. For example:

A function that collects a set of the General_Category values present in a string will and should produce a different value for < angstrom sign, semicolon > than for < A, combining ring above, greek question mark >, even though they are canonically equivalent.

A function that does a binary comparison of strings will also find these two sequences different.

Higher-level processes that transform or compare strings, or that perform other higher-level functions, must respect canonical equivalence or problems will result.

One good example for this is Ά = U+0386 GREEK CAPITAL LETTER ALPHA WITH TONOS and its canonical decomposition is defined as U+0391 GREEK CAPITAL LETTER ALPHA + U+0301 COMBINING ACUTE ACCENT = Α + ‍́ Following are the defined canonically equivalent unicode sequences for some Indic Languages
Bengali:

U+09CB BENGALI VOWEL SIGN O = U+09C7 BENGALI VOWEL SIGN E + U+09BE BENGALI VOWEL SIGN AA
U+09CC BENGALI VOWEL SIGN AU = U+09C7 BENGALI VOWEL SIGN E + U+09D7 BENGALI AU LENGTH MARK
U+09DC BENGALI LETTER RRA = U+09A1 BENGALI LETTER DDA + U+09BC BENGALI SIGN NUKTA
U+09DD BENGALI LETTER RHA = U+09A2 BENGALI LETTER DDHA + U+09BC BENGALI SIGN NUKTA
U+09DF BENGALI LETTER YYA = U+09AF BENGALI LETTER YA + U+09BC BENGALI SIGN NUKTA

Devanagari

U+0929 DEVANAGARI LETTER NNNA= U+0928 DEVANAGARI LETTER NA + U+093C DEVANAGARI SIGN NUKTA
U+0931 DEVANAGARI LETTER RRA = U+0930 DEVANAGARI LETTER RA + U+093C DEVANAGARI SIGN NUKTA
U+0934 DEVANAGARI LETTER LLLA = U+0933 DEVANAGARI LETTER LLA + U+093C DEVANAGARI SIGN NUKTA
U+0958 DEVANAGARI LETTER QA = U+0915 DEVANAGARI LETTER KA + U+093C DEVANAGARI SIGN NUKTA
U+0959 DEVANAGARI LETTER KHHA = U+0916 DEVANAGARI LETTER KHA + U+093C DEVANAGARI SIGN NUKTA
U+095A DEVANAGARI LETTER GHHA = U+0917 DEVANAGARI LETTER GA + U+093C DEVANAGARI SIGN NUKTA
U+095B DEVANAGARI LETTER ZA = U+091C DEVANAGARI LETTER JA + U+093C DEVANAGARI SIGN NUKTA
U+095C DEVANAGARI LETTER DDDHA = U+0921 DEVANAGARI LETTER DDA + U+093C DEVANAGARI SIGN NUKTA
U+095D DEVANAGARI LETTER RHA = U+0922 DEVANAGARI LETTER DDHA + U+093C DEVANAGARI SIGN NUKTA
U+095E DEVANAGARI LETTER FA = U+092B DEVANAGARI LETTER PHA + U+093C DEVANAGARI SIGN NUKTA
U+095F DEVANAGARI LETTER YYA = U+092F DEVANAGARI LETTER YA + U+093C DEVANAGARI SIGN NUKTA

(Note: I saw the ॻ U+097B DEVANAGARI LETTER GGA, ॼ U+097C DEVANAGARI LETTER JJA , and ॾ U+097E DEVANAGARI LETTER DDDA in Debanagari. I am not sure where these letters will be used and whether these are related to GHA, JHA and DDHA )

Gujarati:
Gujarati doesnot have any characters with canonically equivalent sequence.

Gurmukhi :

U+0A36 GURMUKHI LETTER SHA = U+0A38 GURMUKHI LETTER SA + U+0A3C GURMUKHI SIGN NUKTA

U+0A59 GURMUKHI LETTER KHHA = U+0A16 GURMUKHI LETTER KHA + U+0A3C GURMUKHI SIGN NUKTA

U+0A5A GURMUKHI LETTER GHHA = U+0A17 GURMUKHI LETTER GA + U+0A3C GURMUKHI SIGN NUKTA

U+0A5B GURMUKHI LETTER ZA = U+0A1C GURMUKHI LETTER JA + U+0A3C GURMUKHI SIGN NUKTA

U+0A5E GURMUKHI LETTER FA = U+0A2B GURMUKHI LETTER PHA + U+0A3C GURMUKHI SIGN NUKTA

Kannada:

U+0CC0 KANNADA VOWEL SIGN II = U+0CBF KANNADA VOWEL SIGN I + U+0CD5 KANNADA LENGTH MARK

U+0CC7 KANNADA VOWEL SIGN EE = U+0CC6 KANNADA VOWEL SIGN E + U+0CD5 KANNADA LENGTH MARK

U+0CC8 KANNADA VOWEL SIGN AI = U+0CC6 KANNADA VOWEL SIGN E + U+0CD6 KANNADA AI LENGTH MARK

U+0CCA KANNADA VOWEL SIGN O = U+0CC6 KANNADA VOWEL SIGN E + U+0CC2 KANNADA VOWEL SIGN UU

U+0CCB KANNADA VOWEL SIGN OO = U+0CC6 KANNADA VOWEL SIGN E + U+0CC2 KANNADA VOWEL SIGN UU + U+0CD5 KANNADA LENGTH MARK

Malayalam :

U+0D4A MALAYALAM VOWEL SIGN O= U+0D46 MALAYALAM VOWEL SIGN E + U+0D3E MALAYALAM VOWEL SIGN AA

U+0D4B MALAYALAM VOWEL SIGN OO = U+0D47 MALAYALAM VOWEL SIGN EE + U+0D3E MALAYALAM VOWEL SIGN AA

U+0D4C MALAYALAM VOWEL SIGN AU = U+0D46 MALAYALAM VOWEL SIGN E + U+0D57 MALAYALAM AU LENGTH MARK

Oriya :

U+0B48 ORIYA VOWEL SIGN AI = U+0B47 ORIYA VOWEL SIGN E + U+0B56 ORIYA AI LENGTH MARK

U+0B4B ORIYA VOWEL SIGN O = U+0B47 ORIYA VOWEL SIGN E + U+0B3E ORIYA VOWEL SIGN AA

U+0B4C ORIYA VOWEL SIGN AU = U+0B47 ORIYA VOWEL SIGN E + U+0B57 ORIYA AU LENGTH MARK

U+0B5C ORIYA LETTER RRA = U+0B21 ORIYA LETTER DDA + U+0B3C ORIYA SIGN NUKTA

U+0B5D ORIYA LETTER RHA = U+0B22 ORIYA LETTER DDHA + U+0B3C ORIYA SIGN NUKTA

Tamil:

U+0B94 TAMIL LETTER AU = U+0B92 TAMIL LETTER O + U+0BD7 TAMIL AU LENGTH MARK

U+0BCA TAMIL VOWEL SIGN O = U+0BC6 TAMIL VOWEL SIGN E + U+0BBE TAMIL VOWEL SIGN AA

U+0BCB TAMIL VOWEL SIGN OO = U+0BC7 TAMIL VOWEL SIGN EE + U+0BBE TAMIL VOWEL SIGN AA

U+0BCC TAMIL VOWEL SIGN AU = U+0BC6 TAMIL VOWEL SIGN E + U+0BD7 TAMIL AU LENGTH MARK

Notes:

When you search a decomposed form of codepoints, you are also getting the search results of the atomic codepoint. Eg:
മ + േ + ാ == മോ and മ + ോ == മോ
Eventhough the code points are differnt for both of them, since there is a canonical equivalence between them, when you search one, you should get the second.

When you sort the words with canonically equivalent codes, they should come adjacent.

There are some languages where unicode defined the atomic code points obsolete and corresponding sequence as valid one without defining the canonical equivalence. One example is Khmer language. There U+17A8 KHMER INDEPENDENT VOWEL QUK was the atomically enocoded letter for the sequence U+17A7 KHMER INDEPENDENT VOWEL QU U+1780 KHMER LETTER KA. But now the description of U+17A8 KHMER INDEPENDENT VOWEL QUK says " obsolete ligature for the sequence U+17A7 KHMER INDEPENDENT VOWEL QU U+1780 KHMER LETTER KA , use of the sequence is now preferred "

Unicode 5.1 defined new codepoints for the 6 chillu letters in Malayalam which are currently represented by consonant + virama + ZWJ. So a single letter can be represented in both ways. But unicode did not define the canonical equivalence for them. This results in dual encoding and users will not able to get the search results with one representation if they search with other representation.

Windows does not implement the canonical equivalence at all(when tested in windows XP Sp2 for the above mentioned scripts).

Google search gives the correct search results for words with canonical equivalent sequence. Example used is മോഷണം <=> മോഷണം

Yahoo search gives the correct search results for words with canonical equivalent sequence. Example word used is മോഷണം <=> മോഷണം. But unlike google, the search key highlighted in search results uses a decomposed sequence. i.e Yahoo replaces the search key in the search results with decomposed unicode sequence while showing the results.

References:

http://en.wikipedia.org/wiki/Canonical_equivalence

http://unicode.org/reports/tr15/

Firefox spellcheck bugs...

Mon, 02 Jun 2008 16:44:54 GMT

Firefox spellcheck feature requires some volunteers to fix the
tokenization issue. There are two bugs related to the tokenization

Bug 434044 – The tokenization of words for spellcheck is wrong when there is a ZWJ/ZWNJ/ZWS in the word. - Reported: 2008-05-16 07:49 PDT by Santhosh Thottingal

Bug 318040 – Spell checker flags words containing full stops (periods) Reported: 2005-11-28 12:45 PDT by Joseph Wright