LDC2006T02
Arabic Gigaword Second Edition

LDC2006S01
CSLU:  Voices

LDC2006T04
Multiple Translation Chinese (MTC) Part 4

The Linguistic Data Consortium (LDC) would is please to announce the availability of three new publications.

New LDC Publications

(1)  Arabic Gigaword Second Edition is is a comprehensive archive of newswire text data that has been acquired from Arabic news sources by the Linguistic Data Consortium (LDC).  Arabic Gigaword Second Edition includes all of the content of the first edition of Arabic Gigaword (LDC2003T12) as well as new data.

Arabic Gigaword contains five distinct sources of Arabic newswire

The seven-letter codes in the parentheses above consist of the three-character source name IDs and the three-character language code ("arb") separated by an underscore ("_") character. The three-letter language code represents the standard Arabic in the ISO 639-3 standard. In the first edition of the Arabic Gigaword corpus, a simpler three-character-code scheme was used to identify both the source and the language. The new convention allows us to distinguish data sets by source and language more naturally when a single newswire provider distributes data in multiple languages.

Ummah Press is a new source added to the Second Edition. The following table shows the new data that appear for the first time in the Second Edition.

There are 423 files, totaling approximately 1.4GB in compressed form (5359 MB uncompressed and 1591983 K-words). 

*

(2)  The CSLU: Voices corpus contains 12 speakers reading 50 phonetically rich sentences. The recording procedure involved a "mimicking" approach which resulted in a high degree of natural time-alignment between different speakers. The acoustic wave and the concurrent laryngograph signal were recorded for 1 "free" and 2 "mimicked" renditions of each sentence. Pitch marks, calculated from the laryngograph signal, and time marks, the output of a forced-alignment algorithm, have been added to the corpus. 

(3)  Multiple-Translation Chinese (MTC) Part 4 supports the development of automatic means for evaluating translation quality.  The LDC was sponsored to solicit four sets of human translations for a single set of Chinese source materials. The LDC was also asked to produce translations from various commercial-off-the-shelf-systems (COTS, including commercial Machine Translation (MT) systems as well as MT systems available on the Internet). There are a total of five sets of COTS outputs, and six output sets from TIDES 2003 MT Evaluation participants.

To see if automatic evaluation systems, such as BLEU, track human assessment, the LDC has also performed human assessment on one COTS output and the 6 TIDES research systems. The corpus includes the assessment results for one of the 5 COTS systems, the assessment result for the 6 TIDES research systems, and the specifications used for conducting the assessments.

Multiple-Translation Chinese (MTC) Part 4 contains two sources of journalistic Chinese text:

- Xinhua News Agency: 50 news stories
- AFP News Service: 50 news stories

There are 100 source files, and 1,100 translation files. All source data were drawn from LDC's January and February 2003 collection of Xinhua news Chinese data and AFP Chinese data.  For the Chinese data, there are approximately 21K-words, while for the English translations, there are 396K-words in total and 16K unique words.