[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]
Subject: Re: [xliff] XLIFF 2.0 spec - dF sanity check of translation candidates module
Thanks Yves, I know that this discussion was on, but IMHO it did not conclude, it was rather interrupted with Xmas vacations. We all know that the scores [not discerning them now for simplicity's sake] tools are using are not comparable and won't until a standard is developed [this should be fairly simple for similarity, but quite elusive in case of quality] AND widely implemented in the industry [I am personally skeptical in this respect]. Now, my concern is that we decided not to allow extensibility (except mda) in the translation candidates, SO the basic small set of attributes MUST be expressive enough to allow for encoding and passing on the commonly used metadata. Otherwise the industry will not adopt the mechanism. If we conclude that the presently used values are meaningless across tools, this is IMHO a reason to expand the basic set of attributes with tool information rather than dropping the scores altogether. In your elaboration of the feature on the wiki you describe all three similarity, quality, and overall score (to be used for ordering them), which I think is a great idea, otherwise the ordering of matches will purely depend on implementers decision, without possible resort to a standard way of conveying the order, which I think is too bad. I think it is important to realize that we are not in the business of providing algorithms in this area, just vehicles for moving around universally used metadata in a way as useful as possible. The baseline is that the current set of attributes is not enough to pass around the state of the art information: similarity [OK but covering TM only], origin [underspecified, possibly overloaded], type [happy with Yves list not yet in spec], subtype [was agreed and is good for private values, not sure if we need xlf: values here], We do not provide any way of conveying quality information whatsoever, and origin is underspecified if its goal is to ensure comparability of scores produced by by one specific tool [maybe with specific settings, training corpus, runtime glossary etc.] Now I think that there are two possible solutions, one making things simpler, and the other making things more complex 1) drop similarity and subsume similarity under a general suitability attribute, make origin more specific to make sure that it is used for reporting unambiguous tool information, such as a URI, or introduce a tool attribute. 2) use three attributes to describe match suitability in the spirit of the wiki feature description by Yves, i.e. similarity, quality, and an ordering score. Also this wider array would need tool information, either in origin or in dedicated attribute, same as in 1). What I think we cannot afford is to stick to similarity only. Cheers dF Dr. David Filip ======================= LRC | CNGL | LT-Web | CSIS University of Limerick, Ireland telephone: +353-6120-2781 cellphone: +353-86-0222-158 facsimile: +353-6120-2734 mailto: david.filip@ul.ie On Tue, Mar 5, 2013 at 3:54 AM, Yves Savourel <ysavourel@enlaso.com> wrote: > The TC went through that discussion once already (many months ago). > The outcome, as far as I recall was an agreement that there are two distinct pieces of information: > > a) how close the source of the candidate is from the source of the source of the original (similarity) > and b) how good is the translation of the candidate (quality) > > Those two values little to do with each other. > > - MT candidates for example are always perfectly similar (they are the MT translation of the original source), but their quality varies widely. > - You can have an exact match for a candidate coming from a not-reviewed yet pair, so its quality is to be verified > - You can have > > Yes, it would be nice to have some sort of composite value that's computed from both the similarity and the quality, but it's hard to come up with (e.g. which is better: a medium-quality translation for a highly similar source or a high-quality translation of a not exactly similar source?). > > I'm not against an extra 'quality' attribute that would measure how good/bad the translation is. It's just that the result of the previous discussion (if I recall correctly) was to drop such attribute. > > cheers, > -ys > > > > -----Original Message----- > From: Dr. David Filip [mailto:David.Filip@ul.ie] > Sent: Monday, March 04, 2013 1:35 PM > To: Yves Savourel > Cc: xliff@lists.oasis-open.org > Subject: Re: [xliff] XLIFF 2.0 spec - dF sanity check of translation candidates module > > Thanks Yves, > > answering inline.. > >> I think the text/attributes/values in the current draft don't reflect the later discussions. >> See for example >> https://lists.oasis-open.org/archives/xliff/201212/msg00065.html >> and some of the emails (before and after) from Ryan, Shirley, Helena, etc. > OK, I see I will update the spec with the reference during this week. > But the issue of types and subtypes is still entangled with the scores issue below.. Yet, I see value in changing the spec to reflect the discussion for now. >> >> As for similarity: we had that discussion long ago and agreed that it was not meant to represent how good the translation was, just how close the source text of the candidate was from the source text of the original. And for the 'match-quality' (score) , if I recall well at the time the consensus was that the value was not meaningful across tool so we didn't have one. > > I agree with you and Helena that scores are meaningless across tools [until a matching standard is specified, by ULI? when?], no matter if you call them similarity or quality etc. > IMHO the solution is not to ban the attribute but make a tool identifier mandatory if the scores are present. > In my view both similarity and match-quality can be subsumed under something like match suitability, both are values between 0 and 100, so I do not see value in splitting them.. Where would I store the mtconfidence given the current proposals? > >> Some of those ideas are in >> https://wiki.oasis-open.org/xliff/XLIFF2.0/Feature/Translation%20Propo >> sals >> >> cheers, >> -yves >> >> >> -----Original Message----- >> From: xliff@lists.oasis-open.org [mailto:xliff@lists.oasis-open.org] >> On Behalf Of Dr. David Filip >> Sent: Monday, March 04, 2013 11:07 AM >> To: xliff@lists.oasis-open.org >> Subject: [xliff] XLIFF 2.0 spec - dF sanity check of translation >> candidates module >> >> Hi all, I have an issue with semantics of the attributes on this module. >> >> Clearly a match can carry much less attributes than an alt-trans can. >> This is in principle good because we do not want loads of garbage that no one is using. Metadata module is allowed here, but NOT general extensibility, so with the minimum of predefined attributes, we bettwr have them right. >> >> Apart from id and reference flag, we have three attributes that can be considered three dimensions specifying the match. >> We have free text origin that can be probably best used to identify the leverage source within a business process, so we are actually left with similarity and type to describe the match. >> >> I have an issue with similarity, as I think that it is too narrow. >> alt-trans had a match-quality, which was nevertheless defined as similarity of sources. >> >> Now, the issue is that match quality defined as similarity of sources is strictly speaking only relevant for one the match types, i.e. the tm. >> Value Description >> am Assembled Match >> ebm Example-based Machine Translation >> idm ID-based Match >> ice In-Context Exact Match >> mt Machine Translation >> tm Translation Memory Match >> >> For all match types except the TM match, similarity of source is largely irrelevant. >> >> I think there is an easy fix. Call the attribute "match-quality" and use the source similarity just as an example of a valid use of this one.[1] Other match types might have their methods of determining match-quality, e.g. machine translation systems are capable of reporting their confidence that however has nothing to do with similarity of sources. >> >> One of the motivations to do this, would be to be able to map the ITS >> 2.0 metadata category mtConfidence. >> >> In this scenario I can use "origin" to point to the specific MT engine that self-reports its confidence, the "mt" "type" to indicate that the source is MT, and not TM or something else. But "similarity", as defined now in the spec, does not make sense. If it was defined broader as quality if match and named "match-quality", it could be actually used to report mtConfidence. >> >> BTW, why are matches allowed on <unit>? Does it make any sense with at least one segment obligatory? >> >> Thanks for your attention >> dF >> >> >> [1] [proposed new text for the replacement attribute] >> >> B.1.3.2 match-quality >> >> Match-quality - indicates how suitable the match might be for eventual use as translated <target> in the parent <segment>. The most common usage would be to indicate the similarity level between the content of the <source> child of a <match> element and the translatable text being matched. Another usage might be to self-report MT confidence on machine translated translation candidates. >> >> Value description: a decimal number between 0.0 and 100.0. >> >> Default value: undefined >> >> Used in: <match>. >> >> >> Dr. David Filip >> ======================= >> LRC | CNGL | LT-Web | CSIS >> University of Limerick, Ireland >> telephone: +353-6120-2781 >> cellphone: +353-86-0222-158 >> facsimile: +353-6120-2734 >> mailto: david.filip@ul.ie >> >> --------------------------------------------------------------------- >> To unsubscribe from this mail list, you must leave the OASIS TC that generates this mail. Follow this link to all your TCs in OASIS at: >> https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php >> >
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]