Instructions for the sentence similarity annotation task

Introduction

During the course of this experiment, you will be presented with a series of sentence pairs near the top of your screen. Each sentence (in black) comes with one sentence of preceding and one sentence of succeeding context, in gray. It will also have one word underlined. An example is shown below:

Your task is to rate, for each pair of sentences, how similar in meaning the two underlined words are on a five-point scale.

The middle sentence (in black) contains the word to be annotated. In some cases, this one sentence may be all that is needed to understand the meaning of the underlined word, but extra context is provided (in gray) that may help you understand the middle sentence better.

The scale that you will be using for your judgements ranges from 1, meaning that the two instances of the word have completely different meanings, to 5, meaning that the two instances of the word have identical meanings.

Please ignore differences between the sentences that do not impact their meaning. For example, "eat" and "ate" express the same meaning, even though one is present tense, and the other one past tense. Another example of such an irrelevant distinction is singular vs. plural ("carrot" vs. "carrots").

Note that there are no right or wrong answers in this task, so please respond based upon your opinions alone. However, please try to be consistent in your judgments.

The following examples are meant to illustrate the different degrees of similarity or difference that you may find in the task. Note again that these are just examples.

In the following example, the two instances of "eat" can be viewed as being identical in meaning. Both refer to the physical act of consuming food.

In contrast, the two uses of "child" below are slightly different. The first use of "child" could be paraphrased as "young person", while the second instance has a meaning closer to "offspring".

A rating of 3 might be given to the two uses of "chicken" in the example below. The meanings differ in that one refers to the animal itself, and one to the meat of that animal. However, there is still a readily visible similarity between the two uses of the word:

In the following example, the two uses of the word "mouth" are very different. Unlike the "chicken" example above, where the two occurrences of the word share a literal similarity (chicken meat comes from chickens), the two occurrences of "mouth" below share only a figurative similarity, on the level that they are both talking about some type of opening.

The rating of 1 is used for two instances of a word that are completely different in their meaning, as is the case with "bank" below. Notice how this degree of difference is greater than the difference between the two uses of "mouth"; river banks and financial banks have no meaning in common at all, either literal or figurative.

Finally, there is also the option for you to leave a rating of "Cannot decide". Please use this rating only if absolutely necessary, when you are unable to make a decision as to the degree of similarity in meaning between the two underlined words. If there is a particular reason for your inability to decide, please leave a comment saying why.

The above examples of each point on the rating scale can serve as a guide for making your judgments during the task. For each pair of sentences you are presented, rate the similarity in meaning between the two underlined words they contain. Language is often ambiguous. Please read each sentence separately and decide upon the most plausible meaning of the underlined word in each sentence BEFORE comparing the two.

If you wish to leave a comment at any point during the task, just type it into the comment field that appears on the annotation screen. Once you have rated a given pair, click the "Submit" button near the bottom of the page.

As you proceed through the task, if you change your mind about a rating that you have previously supplied, you can navigate through the task using the Navigation panel located on the left side of your screen. You will see two options for navigating through the task: Current Pair and Completed Pairs. If you click on "Completed Pairs", you will be taken to a listing of all the words with sentence pairs for which you have provided annotations thus far. By clicking on a word you will be taken to a listing of all the sentences for that word for which you have provided annotations thus far, allowing you to go back and make the changes you wish.

Once have made your change, don't forget to click the "Submit" button again on that page to make sure your changes are recorded.

Once you are finished, clicking "Current Pair" will take you back to the point where your forward progress in the task stopped, and you can continue annotating.

As you proceed from one pair of sentences to the next, you will notice that often only one of the sentences will change. This is because we want to get your opinions about all the possible pairs of sentences. One sentence will stay the same, while you compare the underlined word in it to all the other instances of that word in the task. Once all those comparisons are made, the sentence which remained the same will be replaced with a new one, and the process continues in a like manner. You will only be shown any given pair of sentences one time during the course of the task.

The sentences included in this task were all gathered from the World Wide Web. They may be very long or short and some may be badly formed in some way. You may find that there are things that make a certain sentence hard to understand. Also, the sentence together with its one sentence of preceding and one sentence of succeeding context need not make a full, self-contained paragraph. Try to ignore these issues; focus only on the meaning of the underlined word in the context in which it occurs. If you find that a sentence is so flawed as to impair your ability to understand what the underlined word means, or that the meaning of the underlined word is ambiguous in the sentence, please be sure to leave a comment to this effect.

If you wish to stop working on the task and return to it later, simply click the link near the bottom of your page that says "logout". Your progress through the task will be saved, along with all your ratings and comments. When you return, simply log back in and you will be taken directly to where you left off.

Thank you again for your participation in this task. These instructions will be available during the course of the task; just click the link that says "Click for Task Instructions".

In total, there are 1170 sentence pairs.