The goal of the test is to teach an agent natural language from scratch. Estimated complexity of the test for humans – 8-10 years old.
The test consists of several steps. In each step the agent is getting a picture and set of phrases in an unknown language which describe what is going on in the picture.
An agent’s goal is to learn the language, associating language structures with visual inputs. Along the learning process the platform poses questions about the pictures which the agent is supposed to answer.
A simplified version of the test provides a structured semantic description of the picture along with the picture itself.
On each step the agent is presented with a picture (or its semantic representation) and a set of phrases. A phrase can contain a statement, a question or an answer to the previous question. If the platform answers its own question it finishes the phrase with an exclamation mark. For example:
It’s a ball statement
What is it? question
It’s a ball! self-answer to the previous question
After each set of phrases the platform is expecting an answer from the agent or an empty string if the agent has nothing to say. The agent’s answers are estimated using a special estimation system. For each answer that is similar to the expected one, the agent is awarded with a certain number of points, depending on the degree of relevance. The agent’s goal is to gain as many points as possible.
Here is a 6-step demo run of the test::
Step 1:
Semantic Image Representation:
{ "objects": [ { "id": "cat", "name": "cat", "objects": [ { "id": "ear1", "name": "ear", "feature": "pointy", "location": "upper-right" }, { "id": "ear2", "name": "ear", "feature": "pointy", "location": "upper-left" }, { "id": "eye1", "name": "eye", "location": "upper-left" } ....... ] } ] }
Dialog:
Platform: se on kisu
Platform: kisu on korvat
Platform: kisu on teravat korvat
Platform: kisu on silmat
Agent: <empty answer>
Step 2:
Semantic Image Representation:
{ "objects": [ { "id": "mouse", "name": "mouse", "objects": [ { "id": "ear1", "name": "ear", "feature": "round", "location": "upper-right" }, { "id": "ear2", "name": "ear", "feature": "round", "location": "upper-left" }, { "id": "eye1", "name": "eye", "location": "upper-left" } ....... ] } ] }
Dialog:
Platform: se on hiirella
Platform: hiirella on korvat
Platform: hiirella on pyoreat korvat
Platform: hiirella on silmat
Agent: <empty answer>
Step 3:
Semantic Image Representation:
{ "objects": [ { "id": "cat", "name": "cat", "objects": [ { "id": "ear1", "name": "ear", "feature": "pointy", "location": "upper-right" }, { "id": "ear2", "name": "ear", "feature": "pointy", "location": "upper-left" }, { "id": "eye1", "name": "eye", "location": "upper-left" } ....... ] } ] }
Dialog:
Platform: mika se on?
Platform: se on kisu!
Agent: <empty answer>
Step 4:
Semantic Image Representation:
{ "objects": [ { "id": "mouse", "name": "mouse", "objects": [ { "id": "ear1", "name": "ear", "feature": "round", "location": "upper-right" }, { "id": "ear2", "name": "ear", "feature": "round", "location": "upper-left" }, { "id": "eye1", "name": "eye", "location": "upper-left" } ....... ] } ] }
Dialog:
Platform: mika se on?
Agent: hiirella
Platform: <provides reward>
Step 5:
Semantic Image Representation:
{ "objects": [ { "id": "cat", "name": "cat", "objects": [ { "id": "ear1", "name": "ear", "feature": "pointy", "location": "upper-right" }, { "id": "ear2", "name": "ear", "feature": "pointy", "location": "upper-left" }, { "id": "eye1", "name": "eye", "location": "upper-left" } ....... ] } ] }
Dialog:
Platform: mita kisu on?
Platform: kisu on korvat ja silmat!
Agent: <empty answer>
Step 6:
Semantic Image Representation:
{ "objects": [ { "id": "mouse", "name": "mouse", "objects": [ { "id": "ear1", "name": "ear", "feature": "round", "location": "upper-right" }, { "id": "ear2", "name": "ear", "feature": "round", "location": "upper-left" }, { "id": "eye1", "name": "eye", "location": "upper-left" } ....... ] } ] }
Dialog:
Platform: mita hiirella on?
Agent: korvat
Platform: mita muuta hiirella on?
Agent: silmat
Platform: <provides reward>