publications

2024

1. ACL 2024

   MovieSum: An Abstractive Summarization Dataset for Movie Screenplays

   Rohit Saxena, and Frank Keller

   In Findings of the Association for Computational Linguistics ACL 2024, Aug 2024

   Abs

   Movie screenplay summarization is challenging, as it requires an understanding of long input contexts and various elements unique to movies. Large language models have shown significant advancements in document summarization, but they often struggle with processing long input contexts. Furthermore, while television transcripts have received attention in recent studies, movie screenplay summarization remains underexplored. To stimulate research in this area, we present a new dataset, MovieSum, for abstractive summarization of movie screenplays. This dataset comprises 2200 movie screenplays accompanied by their Wikipedia plot summaries. We manually formatted the movie screenplays to represent their structural elements. Compared to existing datasets, MovieSum possesses several distinctive features: 1) It includes movie screenplays which are longer than scripts of TV episodes. 2) It is twice the size of previous movie screenplay datasets. 3) It provides metadata with IMDb IDs to facilitate access to additional external knowledge. We also show the results of recently released large language models applied to summarization on our dataset to provide a detailed baseline.

2. NAACL 2024

   Select and Summarize: Scene Saliency for Movie Script Summarization

   Rohit Saxena, and Frank Keller

   In Findings of the Association for Computational Linguistics: NAACL 2024, Jun 2024

   Abs

   Abstractive summarization for long-form narrative texts such as movie scripts is challenging due to the computational and memory constraints of current language models. A movie script typically comprises a large number of scenes; however, only a fraction of these scenes are salient, i.e., important for understanding the overall narrative. The salience of a scene can be operationalized by considering it as salient if it is mentioned in the summary. Automatically identifying salient scenes is difficult due to the lack of suitable datasets. In this work, we introduce a scene saliency dataset that consists of human-annotated salient scenes for 100 movies. We propose a two-stage abstractive summarization approach which first identifies the salient scenes in script and then generates a summary using only those scenes. Using QA-based evaluation, we show that our model outperforms previous state-of-the-art summarization methods and reflects the information content of a movie more accurately than a model that takes the whole movie script as input.

3. Under review

   Are We Done with MMLU?

   Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, and 13 more authors

   Jun 2024
4. Under review

   The Hallucinations Leaderboard – An Open Effort to Measure Hallucinations in Large Language Models

   Giwon Hong, Aryo Pradipta Gema, Rohit Saxena, and 8 more authors

   Jun 2024

2018

1. ACL 2018

   EmotionX-Area66: Predicting Emotions in Dialogues using Hierarchical Attention Network with Sequence Labeling

   Rohit Saxena, Savita Bhat, and Niranjan Pedanekar

   In Proceedings of the Sixth International Workshop on Natural Language Processing for Social Media, SocialNLP@ACL 2018, Jul 2018

   Abs

   This paper presents our system submitted to the EmotionX challenge. It is an emotion detection task on dialogues in the EmotionLines dataset. We formulate this as a hierarchical network where network learns data representation at both utterance level and dialogue level. Our model is inspired by Hierarchical Attention network (HAN) and uses pre-trained word embeddings as features. We formulate emotion detection in dialogues as a sequence labeling problem to capture the dependencies among labels. We report the performance accuracy for four emotions (anger, joy, neutral and sadness). The model achieved unweighted accuracy of 55.38% on Friends test dataset and 56.73% on EmotionPush test dataset. We report an improvement of 22.51% in Friends dataset and 36.04% in EmotionPush dataset over baseline results.

2017

1. ICDM 2017

   Live on TV, Alive on Twitter: Quantifying Continuous Partial Attention of Viewers During Live Television Telecasts

   Rohit Saxena, Savita Bhat, and Niranjan Pedanekar

   In 2017 IEEE International Conference on Data Mining Workshops (ICDMW), Jul 2017
2. CSCW 2017

   I Know What You Coded Last Summer: Mining Candidate Expertise from GitHub Repositories

   Rohit Saxena, and Niranjan Pedanekar

   In Companion of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing, Portland, Oregon, USA, Jul 2017

   Abs

   Resumes and social recommendations are often high-level indicators of a candidate’s technical skillset. In this paper, we present a method to create a more detailed technology skill profile of a candidate based on her code repository contributions. For this purpose, we annotate user contributions to GitHub code repositories with technology tags found in Stack Overflow questions and answers (Q&A) in an unsupervised manner. We also present SkillMap, a visual representation of candidate skill profile, for quick review and comparison with other candidate profiles. We create SkillMaps for 66 Java programmers and present a preliminary qualitative assessment though manual analysis and interviews of technical recruiters.

2015

1. CCS 2015

   POSTER: WinOver Enterprise Dark Data

   Manish Shukla, Sumesh Manjunath, Rohit Saxena, and 2 more authors

   In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, Denver, Colorado, USA, Jul 2015

   Abs

   Any persistent untagged, untapped and unclassified data can be termed as dark data. It has two common traits: first, it is not possible to determine its worth, and second, in most of the scenarios it is inadequately protected. Previous work and existing solutions are restricted to cater single node system. Moreover, they perform specialized processing of selected content, for example, logs. Further, there is total negligence of stakeholders and minimal focus on the data getting generated within the enterprise. From the perspective of an enterprise it is important to understand the distribution, nature and worth of dark data, as it helps in choosing right security controls, insurance or steps needed to pre-process a system before discarding it. In this paper we demonstrate a distributed system, called File WinOver, for File Lifecycle Management (FLM). The solution operates in a distributed environment where it identifies the dormant and active files on a system, filters them as per requirement and computes their fingerprint. Moreover, the content fingerprinting is utilized to detect closed user groups. After which, it classifies the content based on configured policies, and maps them with the stakeholders. This mapping is further used for valuating the risk exposure of the file. Thus, our system helps in identifying dark data and assigns quantitative risk value.