Storytelling Workshop

Co-located with NAACL 2018

June 5th, 2018. New Orleans, Louisiana

2 Feb 2018:	Storytelling Challenge begins
2 March 2018:	Long, Short, Demo papers due
2 April 2018:	Notification of acceptance
16 April 2018:	Camera-ready papers due
16 May 2018:	Storytelling challenge ends
23 May 2018:	Human Evaluation Result Notification
28 May 2018:	Challenger Paper deadline (submit to arXiv, optional)
30 May 2018:	Challenger Paper deadline (submit to us, optional)
5 June 2018:	Workshop!

Storytelling

Human storytelling has existed for as far back as we can trace, predating writing. Humans have used stories for entertainment, education, cultural preservation; to convey experiences, history, lessons, morals; and to share the human experience.

Part of grounding artificial intelligence work in human experience can involve the generation, understanding, and sharing of stories. This workshop highlights the diverse work being done in storytelling and AI across different fields.

The program of Storytelling Workshop!

Proceedings of the First Storytelling Workshop

Invited Keynote Speaker:

Nasrin Mostafazadeh, Elemental Cognition

Event-centric Context Modeling: The Case of Story Comprehension and Story Generation

Building AI systems that can process natural language input, comprehend it, and generate an engaging and contextually relevant output in response, has been one of the longest-running goals in AI. In human-human communications, a major trigger to our meaningful communications are “events" and how they cause/enable future events. We often communicate through telling stories in the form of related series of events. In this talk, I present my work on language processing in terms of events and how they interact with each other in time. Mainly through the lens of storytelling, I will focus on story comprehension and collaborative story generation, with a major emphasis on commonsense reasoning and narrative knowledge as showcased in the Story Cloze Test framework. Through different use cases, I will highlight the importance of establishing a contentful context and modeling multimodal contexts (such as visual and textual) in various AI tasks.

Bio: Nasrin Mostafazadeh is a senior AI research scientist at Elemental Cognition where she works on the next generation of AI systems that not only comprehend language, but also explain their reasoning and answer 'why'. She has previously held research positions at BenevolentAI, Microsoft, and Google, working on various language comprehension tasks. Nasrin got her PhD at the University of Rochester at the conversational interaction and dialogue research group, during which she worked on language understanding in the context of stories, mainly through the lens of events and their causal and temporal relations. She has developed models for tackling various research tasks that push AI toward deeper language understanding with applications ranging from story generation to vision & language.

Cinque Terre

The Workshop

This one-day, multi-modal and interdisciplinary workshop will bring together researchers and practitioners in NLP, Computer Vision, and storytelling. The focus will be on human storytelling: What storytelling is, its structure and components, and how it’s expressed, connected to the state of the art in NLP and related ML/AI areas:

What we can understand from stories (natural language understanding)
What we can generate to create human-like stories (natural language generation)
What we can recognize multimodally for story understanding and generation (e.g., with computer vision)

The workshop will consist of:

Contributed talks and posters.
A visual storytelling challenge.
Invited talks given by researchers in NLP, Computer Vision, and Storytelling.

Call For Papers

We invite work involving human storytelling with respect to machine learning, natural language processing, computer vision, speech, and other ML/AI areas.
This spans a variety of research, including work on creating timelines, detecting content to be used in a story, generating long-form text, and related multimodal work.
Data input sources may include professional and social-media content.

We also encourage ideas about how to evaluate user experiences in terms of coherence, composition, story comprehensiveness, and other aspects related to the creation of stories.

Paper topics may include, but are not limited to:

The role of storytelling in artificial intelligence
Accessible and Assistive storytelling
Affect and emotion in stories
Augmenting human storytelling
Character relationships
Collaborative storytelling
Event summary diversity
Event-episodes detection and annotation
Multimodal grounding for storytelling
Multimodal event timelines
Narrative structure
Plot structure
Story concept detection and annotation
Story generation
Story understanding
Storytelling Applications/Demos
Temporal/Event structure
Temporal and semantic alignment
Writing stories
User studies

Papers should follow the NAACL 2018 style guidelines and are due on Softconf by 2 March, 2018.

Visual Storytelling Challenge

We're glad to announce that the "DG-DLMX" team is the winner of the fist VIST Challenge!

The DG-DLMX team is formed by Diana González-Rico and Gibran Fuentes-Pineda, who are from the Institute for Research in Applied Mathematics and Systems (IIMAS), Universidad Nacional Autónoma de México (UNAM).

Stay tuned for the papers from all four teams!

This challenge begins to scratch the surface on how well artificial intelligence can share in this cultural human experience. Participants are encouraged to work on creating AI systems that can generate stories for themselves, sharing the human experience that they see -- and begin to understand. (Click here to see more about the dataset.)

Participants may submit to two different tracks: The Internal track and the External track. Submissions are evaluated on how well they can generate human-like stories given a sequence of images as input.

Dates

2 Feb 2018: Data train set augmented with additional stories
16 May 2018: Submissions due on EvalAI. (You will need to create an account to view the challenge)
23 May 2018: Human Evaluation Result Notification to Challengers
28 May 2018: Challenger Paper deadline (submit to arXiv, optional)
30 May 2018: Challenger Paper deadline (submit to us, optional)
5 June 2018 (Workshop Day): Results announced

Submission Tracks

The VIST Challenge has the following two tracks: External Track and Internal Track.

We will evaluate each track separately. Namely, in VIST Challenge at NAACL 2018, we will have two winning teams, one for External Track and one for Internal Track.

Each team's *latest* submission before the submission deadline will be used for the final evaluation.

Each team can choose to submit to only one track, or to both tracks.

Internal Track

For apples-to-apples comparison, all participants should submit to the Internal track. In this track, the only allowable training data is:
- Any of the VIST storytelling data (SIS, DII, and/or the non-annotated album images) Data available here
Allowed pretraining data:
- Data from any version of the ImageNet ILSVRC Challenge (common in computer vision).
- Data from any version of the Penn Treebank (common in natural language processing).
If you wish to use any other sources of data/labels or pre-training, please submit to the External track.
External Track

Participants can use any data or method they wish during training (including humans-in-the-loop), but all data should be publicly available or made publicly available.

At test time, the systems must be stand-alone (no human intervention).

Possible datasets include data from ICCV/CVPR workshops, such as LSMDC, and other vision-language datasets, such as COCO, and VQA.

Prize

$500 (US dollars) will be awarded to the winning team of each track. The winning teams will be announced during the Storytelling Workshop at NAACL 2018.

Evaluation

Human Evaluation Result Notification: May 23, 2018

Evaluation will have two parts:

Automatic: On EvalAI, using the automatic metric of METEOR.
Human: Crowdsourced survey of the quality of the stories.

The winning team will be decided by a group of human judges. The organizers of the VIST Challenge will recruit a group of crowd workers from Amazon Mechanical Turk as the human judges to rate the following aspects of submitted stories (on a 5-point likert scale, from Disagree to Agree).

Focus ("This story is focused."):
Each story should have a focus, i.e., each sentence of the story should contain information that is "naturally" relevant to the rest of the story, nothing should be random or irrelevant.
- Good focus: "We went to the park. They had a great play ground. We also got to eat ice cream."
- Bad focus: "We went to the park. I grew up in Chicago. My neighbor is getting surgery. Tom Cruise is such a good actor."
Structure and Coherence ("The story is coherent."):
The story should be well-structured, grammatical and well-organized. The story shouldn't be a bunch of related information, but should build from sentence to sentence into a coherent body of readable text describing the set of images.
- Good coherence: "We went to the park. They had a great play ground. We also got to eat ice cream"
- Bad coherence: "We went to the park. We went to the store. We went outside."
"I Would Share": If these were my photos, I would like using a story like this to share my experience with my friends.
"Written by a Human": This story sounds like it was written by a human.
"Visually Grounded": This story directly reflects concrete entities in the photos.
Detailed: This story provides an appropriate level of detail.

Non-archival Publication and Presentation (Optional)

All the competition teams will be invited to submit a *non-archival* paper (in the NAACL 2018 short paper format) to describe their systems used in the competition. Submitting a paper is optional. Each team can decide if they want to submit a paper.

The submitted papers will be hosted on arXiv, and we will also list all the papers (with a PDF copy) on the VIST website, but *NOT be included in any NAACL proceedings NOR the ACL Anthology.*

For submitting a non-archival paper, each team needs to satisfy *both* of the following two criteria:

The competition team needs to automatically generate visual stories, submit the generated stories successfully to the VIST Challenge on EvalAI with a positive METEOR score at least once.
At least one of the co-authors will register and *show up in person* to the Storytelling Workshop at NAACL 2018 to present their work.

If a competition team develops two very different systems for External and Internal Tracks, respectively, the team will be allowed to submit two separate non-archival papers. However, we strongly encourage each team to come up with one single paper to describe all of their work.

In the Storytelling Workshop at NAACL 2018, we will arrange an oral or poster (depends on the number of submissions) "Challenger" session for all the competition teams to present their systems.