Author Topic: [Classic Rexx] Translation database  (Read 14176 times)

Jan-Erik Lärka

  • Global Moderator
  • Sr. Member
  • *****
  • Posts: 363
  • Karma: +14/-0
[Classic Rexx] Translation database
« on: May 25, 2025, 10:50:24 pm »
I've created a script to assemble words and phrases into a database (sqlite3) that can be viewes with DB Browser for SQLite (A huge Thank you TeLLie! for porting it!).

One just need to type
Code: [Select]
set mode=INITand then
Code: [Select]
T2NLV
It use wget to download a dictionary from wikdict (default english to the language your system run as) so it not aimed at those that use en_US or en_UK or similar.

It then build a database and feed the data to it.

The script can also handle translations by Goran Ivankovic such as .DCT and .INI but also .MSG files.
I've included (though renamed .DCT files provided in Gorans translation tool for reference).

Code: [Select]
set mode=IMPORTand then
Code: [Select]
T2NLV en-sv.DCT(change "sv" for your language if available)

Do not try to feed it the file "Words.txt" unless you're ready to let it spend (a lot of) time fetching translations from google.

I would appreciate your help to improve the script and database!

How about a translation tool to go with it to manage all the translation it should be able to swallow.
For those interested may want to poke on it and look att the table "CLASSIFICATION" as well.

Type
Code: [Select]
Set fromlang=frto set the source language by hand
or
Code: [Select]
Set tolang=nlto set destination language by hand.

Test and create a function on the side in another script that output numerals, so such words in the database can be classified.
That code has to be added, but the db contain them already.
rxNumerals: PROCEDURE EXPOSE this.
IF ARG(1) = 0 THEN
   RETURN 'zero'
IF ARG(1) < 20 THEN
   RETURN WORD( 'one two three four ...
« Last Edit: May 29, 2025, 05:14:52 pm by Jan-Erik Lärka »

Jan-Erik Lärka

  • Global Moderator
  • Sr. Member
  • *****
  • Posts: 363
  • Karma: +14/-0
Re: [Classic Rexx] Translation database
« Reply #1 on: May 26, 2025, 12:44:19 pm »
You need two files if you want to add text from .msg and .ini files, one with the source language and another built the same way but with text for the destination language.
The script handle missing translations, but may require a watchful eye afterward.

Please do provide an url (and possibly code) or two, for other translation services than google one could use (as backup) as well.
See the script for one code example as of now.

It would also be very convenient if one could add support for more file formats... perhaps system specific files?!
OS/2 system files often need to be disassembled before one can extract the useful stuff.

Do explore and present command line resource (de)compilers along with other tools and scripts to extract text.

The translation team have the knowledge and it would be most useful if they'd share it and we could include such info into a tool so not everyone need to be a rocket scientist, on top of everything else, but can focus on writing good translations.

Jan-Erik Lärka

  • Global Moderator
  • Sr. Member
  • *****
  • Posts: 363
  • Karma: +14/-0
Re: [Classic Rexx] Translation database
« Reply #2 on: May 26, 2025, 10:27:35 pm »
Time is up!
Here's the update that add Numerals.
This now also show the word for to_language for prefix, suffix, language and numerals.
The view PHRASE_VW has been adjusted to allow that.

What files should one take on next?
Hmmm, lip packages while we're at it?

Jan-Erik Lärka

  • Global Moderator
  • Sr. Member
  • *****
  • Posts: 363
  • Karma: +14/-0
Re: [Classic Rexx] Translation database
« Reply #3 on: November 03, 2025, 08:53:43 pm »
Related topic: [Classic Rexx] How can one pipe data to application?

I've been bussy to rewrite the script.
It take ISO files with OS/2-eCS-ArcaOS, source language and target language, extract and interpret various file formats/resoruces and add translations to a database (SQLite), but also use online resources (database and google) when possible to translate text.
This version is more capable, but I still need your help.

I need your help to find more translation sources and add how to parse and add them.

This script is currently available in English and Swedish... and with your help soon also in your language ;)
It is built as a menu system that use .msg files to output text, thus need your time to translate it. (See files in the folder TRN_MSG_TXT)

It is intended to be able to translate itself, but it is unlikely right now since the word-to-word translations available, focus on other phrases and words.

The current sources for translations are:
https://download.wikdict.com/dictionaries/sqlite/
http://translate.googleapis.com/translate_a/
saldo.xml or swesaurus.xml specifically for Swedish
and the OS/2 centric ftp://ftp.zx.net.nz/pub/archive/service.boulder.ibm.com/ps/products/os2/fixes/

Someone may want to create a function that can parse either the 19Gb large jsonl within https://kaikki.org/dictionary/raw-wiktextract-data.jsonl.gz or the smaller kaikki.org-dictionary-<Language>.jsonl and help me here! and/or perhaps mare the related python scripts available... whatever it can be used for.

What is lacking that also need your assistance to solve:
Caseless comparison of phrases/words within the sqlite database when used with national characters.
Code to use additional online and offline translation sources.
SQL views within the database that can be useful to output translations into various useful file formats.
Graphical software to add and use translations!

Good to have tool: DB Browser for SQLite by TeLLie


Interactive guide
=============

 1. Information
 2. Preparations
 3. Usage
 4. Change default language/country
 5. Start

 0. Exit

Type a number from the list above: 5


Start
====

 1. Combine translation of OS/2, eComStation or ArcaOS
 2. Combine translation between en and fr
 3. Create database for translation between en and fr
 4. Import text to translate between en and fr

 5. Change source language from en to translate from
 6. Change target language from fr to translate to

 0. Exit

Type a number from the list above (Enter to return): 1


Source ISO-file of OS/2, eComStation or ArcaOS in en: C:\var\ArcaOS-5.0.7.iso
The Process Identification Number is 5474.
Path to ISO-file of OS/2, eComStation or ArcaOS in fr (or Empty): E:\Operatingsystem\OS2 Warp 4\Warp_4_fr.ISO
The Process Identification Number is 5481.


Alfredo Fernández Díaz

  • Jr. Member
  • **
  • Posts: 94
  • Karma: +6/-0
Re: [Classic Rexx] Translation database
« Reply #4 on: March 29, 2026, 10:35:47 pm »
Hello Jan-Erik,

The translation team have the knowledge and it would be most useful if they'd share it and we could include such info into a tool so not everyone need to be a rocket scientist, on top of everything else, but can focus on writing good translations.

I am in touch with the translation team, so I may be able to help, just what do you need? (I haven't got round to having a look at your script yet, sorry.)

Meanwhile, I have posted this over at the REXX tutorials thread:

[...]Hobbyists and enthusiasts still use old versions of OS/2 which have zero support today. Imagine you are one and find your system messages show typos or some nastier error(s) like wrong numbers, swapped messages, etc. Since IBM released a free toolkit, in theory you could use it to compile new system message files, except that you don't have sources. Fortunately, the OS/2 message file format is simple enough that some quick REXX can be used to generate compatible source from the files themselves.[...]

which may be of interest for you too ;)

Regarding this:
[...]
What is lacking that also need your assistance to solve:
Caseless comparison of phrases/words within the sqlite database when used with national characters.

Once you get something out of the database, you probably want to do that with Alex Taylor's RxULS (long included with ArcaOS) -- it exposes Unicode handling functions of the OS/2 API to REXX. In particular, ULSConvertCodepage and ULSTransform are your friends, I think.

Jan-Erik Lärka

  • Global Moderator
  • Sr. Member
  • *****
  • Posts: 363
  • Karma: +14/-0
Re: [Classic Rexx] Translation database
« Reply #5 on: March 30, 2026, 07:08:09 am »
Hello Alfredo,

As for translating the screen output for the script itself, just take the text source for an existing file, rename it and type those new phrases. I have the scripts ready to compile the files automatically.
It can be done with google translate, but it's better if more actually try to use the script and try something easy to begin with.

Extracting resources from OS/2 should have been done centrally, so that translators could focus on ... translations, and not all the different tools and file formats on the OS/2 installation CD's and the dictionaries in file formats not known to the os2 world as they focus on nix and win. This script take care of the OS/2 resources found on installation cd's. Result: One database with from-to phrases.

The script also use Alex Taylor's Unicode conversion, as it's needed  when exchanging data with sqlite etc.

Jsonl text extraction from dictionaries still missing, but the script assemble a large portion of os/2 translatable text as it is. So once you translate the data collected , it contain your NLS version of OS/2.

As for caseless comparisons in the database and many tools: That is not unique to OS/2, but still annoying.

Alfredo Fernández Díaz

  • Jr. Member
  • **
  • Posts: 94
  • Karma: +6/-0
Re: [Classic Rexx] Translation database
« Reply #6 on: April 01, 2026, 09:44:24 pm »
OK, I have fixed several typos in your English message file, and here you have a preliminary Spanish translation for the script. Lacking proper testing I'll probably have questions to ask further down the road. Meanwhile, I have two observations:
1. Regarding ArcaOS in particular, you will find OS/2 resources living in at least two places: the MCP2 base, and the 'reorganized resources' package to be selected during install, where most of the updates to the UI text including newer NLV translations are included for logistical reasons AFAICT.
2. When extracting OS/2 resources, it is frequent enough (at least with the core OS itself) to get big RCDATA "blobs" ("custom resource" according to IBM specs) in .rc files which are actually a heap of null-terminated string that a "custom" application (IBM's OS applet) knows about and are usually for MRIs et al, so extracting with rdc does not get you all text. Example: resource #5000 in PMWPMRI.dll.

Jan-Erik Lärka

  • Global Moderator
  • Sr. Member
  • *****
  • Posts: 363
  • Karma: +14/-0
Re: [Classic Rexx] Translation database
« Reply #7 on: April 03, 2026, 09:57:48 am »
Thank you Alfredo,

Thank you for correcting the english translation!

Very good points.
The script know nothing about specific locations of data, but list and scan the whole cd-images for "known" file formats to extract (see example image).

So how can one do anything useful whith such rcdata, while other rcdata contain something else?

Could you write 3 of functions for it?
1. Read a portion of data from the file (name) provided and sample the extracted data to determine if the file contain data that can be extracted in an orderly fashion
2. Read and if needed separate data from garbage to be entered to the translation database
3. If present in the target language, extract the equivalent translation to complete the from-to translation pair, otherwise it will translate online and indicate that in the database.

rcdata as file format allow for a vast variety of content.

I'd need structural info to know where to add it to the large script:
Would your code perhaps handle data as "osapplet"? (or suggest other, one per file-format-handler-function to call)
Some file formats contain translation pairs, others in separate files with the same base name, while yet other on the same location in each cd-image. I assume that these rcdata text is of the latter.

trn_es.txt changed id 47 set to: Español
« Last Edit: April 03, 2026, 10:17:01 am by Jan-Erik Lärka »

Alfredo Fernández Díaz

  • Jr. Member
  • **
  • Posts: 94
  • Karma: +6/-0
Re: [Classic Rexx] Translation database
« Reply #8 on: April 11, 2026, 12:01:19 pm »
Thank you for correcting the English translation!
You're welcome, but that's just one first step, I expect this will have to be refined as I get familiar with the whole thing ^^
More on this all throughout the post.

Quote
Very good points.
The script knows nothing about specific locations of data, but lists and scans the whole cd-images for "known" file formats to extract (see example image).

So how can one do anything useful with such rcdata, while other rcdata contain something else?

Could you write 3 of functions for it?
1. Read a portion of data from the file (name) provided and sample the extracted data to determine if the file contain data that can be extracted in an orderly fashion
2. Read and if needed separate data from garbage to be entered to the translation database
3. If present in the target language, extract the equivalent translation to complete the from-to translation pair, otherwise it will translate online and indicate that in the database.

rcdata as file format allow for a vast variety of content.

I'd need structural info to know where to add it to the large script:
Would your code perhaps handle data as "osapplet"? (or suggest other, one per file-format-handler-function to call)
Some file formats contain translation pairs, others in separate files with the same base name, while yet other on the same location in each cd-image. I assume that these rcdata text is of the latter.

Mmmh, let start with the coding part. IBM's RCDATA blobs which are just heaps of strings are very easy to parse. Example: save the strings.cmd attached, extract the famous resource #5000 from PMWPMRI.DLL with "resmgr -x PMWPMRI.DLL 5000 mystrings.bin" and process with  "strings.cmd mystrings.bin" or "strings.cmd mystrings.bin > mystrings.txt".

As for the question of what string blobs to parse like this, or leave alone, I have no easy answer, there may be none. You could try some heuristics (f.e. if an rcdata blob contains only text data, i.e. charcodes >= 32 strings followed by short null terminator runs --ideally single nulls, mybe two at the end-- it very likely is a string "heap"), but not even that will tell you much about what to do with what you get. By this I mean that after processing you could still have anything: a string array where stuff will be picked by index (so check for equal array length on both sides of the translation), or some table. For example, take glocmri.dll. As all MRIs, this says 'translate me'. However, there you will find an rcdata string blob that will translate to a table of language codes and names; one column should be translated, the other not. How to tell programmatically? How to tell it is even a table?

Quote
trn_es.txt changed id 47 set to: Espa¤ol
OK, but you did it on an editor that changes OS/2 newlines (CR+LF) to unix-style ones, please see the attached pic and do it again ; )

Anyway, this brings me back and full circle again onto the most important question, as often overlooked in translation projects.

How would I know that message #47 should translate as "Español" and not "English"? There is no way even the best translator in the world would, except by familiarity with the project, knowing where translations will end exactly, and probably having a couple of goes at it.

The problem is, stuff like the above, which any translation database project will inevitably keep bumping into, is extremely difficult to handle, let alone get right, in some generic way -- it will take time and familiarity with project specifics, so please take this is as a very friendly warning to be patient and persevere. We'll get stuff out of the way, one step at a time. : )
« Last Edit: April 11, 2026, 12:03:00 pm by Alfredo Fernández Díaz »

Jan-Erik Lärka

  • Global Moderator
  • Sr. Member
  • *****
  • Posts: 363
  • Karma: +14/-0
Re: [Classic Rexx] Translation database
« Reply #9 on: April 13, 2026, 09:48:42 pm »
Here's the translation with crlf.

below is the output of rdc.
So how should the comment be parsed/divided?

RCDATA 5000
BEGIN
  0x656A624FL, 0x46007463L, 0x20656C69L, 0x74737953L, /* Object.File Syst */
  0x44006D65L, 0x20617461L, 0x656C6946L, 0x6F725000L, /* em.Data File.Pro */
  0x6D617267L, 0x6C694620L, 0x6F460065L, 0x7265646CL, /* gram File.Folder */
  0x69724400L, 0x00736576L, 0x65726853L, 0x72656464L, /* .Drives.Shredder */
  0x61745300L, 0x70757472L, 0x69725000L, 0x7265746EL, /* .Startup.Printer */
  0x6E695700L, 0x20776F64L, 0x7473694CL, 0x73694400L, /* .Window List.Dis */
  0x654E006BL, 0x726F7774L, 0x6241006BL, 0x61727473L, /* k.Network.Abstra */
etc.