app-pinion

Who

John Sullivan, jsulli31
Troy Conklin, sconkli1
Nicholas Bottone, nbottone
Robert Koch, rkoch2

Introduction

We are going to be attempting to model twitter user’s stance/opinion label on various subject matter using historical tweet sequences from users and their “neighbors”.
We chose this paper because it seemed like an interesting/frightening piece of research, and it stood out to us the most out of any of the other papers we reviewed. The fact that it’s possible to model the opinion of a twitter user about subjects they haven’t yet tweeted about seems quite startling. The paper's objectives are to model a twitter user's future stance/opinion on their future tweets based on prior tweets from themselves and their neighbors, as well as predict a time interval in which they will post their next tweet.
This project uses natural language processing to address a sequence prediction problem.

Related Work

The paper we are implementing is “Neural Temporal Opinion Modelling for Opinion Prediction on Twitter” from the University of Warwick Department of Computer Science.
An additional article relevant to the paper we are re-implementing is Neural opinion dynamics model for the prediction of user-level stance dynamics, which is an extension of the original paper going into more depth on the model and the subject area. The paper highlights an emerging field of interest known as ‘opinion mining’ where a user’s actions on a web application may be used for opinion prediction.
https://github.com/cbaziotis/datastories-semeval2017-task4
https://github.com/somethingx01/TopicalAttentionBrexit

Data

We will be using the existing public Twitter datasets containing opinions about Brexit and opinions about the 2016 US Presidential Election that were used in the original implementation of this project. The datasets are both accessible from the GitHub repository linked above. The Brexit dataset contains a total of 363,961 tweets from 38,335 different users, while the US Election dataset contains a total of 452,128 tweets from 108,689 different users. Given that these datasets are relatively large, we will need to perform a fair amount of preprocessing and batching, but we can follow the preprocessing method of the existing research paper or reference the already-preprocessed versions of the datasets if needed.

Methodology

Input at timestep i is the user’s tweet, bag-of-word representation, user embedding, neighbors tweet queue, time interval between the i tweet and i-1 tweet
Bi-LSTM layer extracts features from input tweets
Neighbor tweets are processed by Bi-LSTM/LSTM stacked layer for context
Context is given to attention module
Attention is queried by user’s tweet and topic
Output of attention is concatenated with tweet representation, user representation, topic representation and time interval which is encoded with auto encoder.
Combined representation sent to GRU which computes intensity function and softmax, and predicts stance label of next tweet as well as the time interval
The hardest part will likely be figuring out the stacked Bi-LSTM layers and how those work together.

Metrics

We plan on using datasets of Twitter users’ opinions on Brexit and Trump. From each dataset, we will group tweets together by user, and categorize 90% of a user’s tweets into the training set and 10% of a user’s tweets into the testing set. We will use our trained model on the testing set to determine the stance prediction accuracy and evaluate the model’s overall effectiveness. We may introduce more datasets if time allows.
- We will be able to evaluate the accuracy of the stance prediction of our model, which should be the best metric for evaluation.
- The authors of the paper were hoping to model a user’s posting behavior by a temporal point process. When a user posts a tweet at a time, they need to decide on if they will be posting a new topic or post a topic influenced by past tweets by other users or themselves. So the model they propose is to jointly predict the time when the new post will be published and the associated “stance”. They quantified the results of their model with stance prediction accuracy and mean squared error of the time interval prediction.
Base, target, and stretch goals:
- Our base goal for the project is to implement the model given in our reference paper in Tensorflow to predict the stance of Twitter user’s opinions.
- Our target goal is to implement the model, achieve a relatively similar accuracy for the Brexit and 2016 Election datasets, and test the model on relevant datasets outside of the scope of those used in the paper.
- Our reach goal is to implement the specifications from base and target as well as use the network to predict the timeframe in which a user is next going to make a Twitter post.

Ethics

What broader societal issues are relevant to your chosen problem space?
- As our project is attempting to make predictions about the opinions of users on social media based on their public posts, there are some ethical issues which may arise as a result in the deployment of such programs. For example, such a program could be used by an individual / organization to commit hate crimes towards particular groups of people based on opinion. On the flip side, this project may be useful in gauging public overall public opinion by geographic region / social media space for important issues like elections. If a politician sees a lot of opposition from a particular region / platform, they could use this information to improve their campaign.
What is your dataset? Are there any concerns about how it was collected, or labeled? Is it representative? What kind of underlying historical or societal biases might it contain?
- The two datasets that we are using contain large amounts of opinionated tweet data relating to Brexit and the 2016 US Presidential election, so both datasets contain a fair amount of social and political concerns. These datasets are specific to particular major issues within the U.K. and the U.S., but there is not much representation of political issues in other parts of the world or non-political opinions. For the Brexit dataset, 31.6% of the tweets are labeled as “supporting” Brexit, 29.3% are labeled as “opposing” Brexit, and 39.1% of the tweets are neutral. This represents a relatively broad range of opinions, though it should be noted that these broad categories may not represent the nuances in different users’ opinions. For the US Election dataset, 74.2% of the tweets were from users who supported Donald Trump, 20.4% of the tweets were from users who opposed Donald Trump, and 5.4% of tweets were labeled as neutral. This represents a much less even distribution of opinions, and it should be noted that this can very likely impact the results of the prediction system. It should also be noted that, although the tweet data that is used in this project is available to the public, the users who created these tweets did not give explicit permission for their tweet data to be used by this prediction algorithm.
Who are the major “stakeholders” in this problem, and what are the consequences of mistakes made by your algorithm?
- Stakeholders include the Twitter platform, politicians, public figures that want to evaluate the public’s opinion of them, brands and corporations that want to evaluate their public image, and Twitter users. Mistakes made by the algorithm could cause public figures or corporations to misevaluate the public’s perception of particular issues or topics, and could lead to prioritizing issues that are not important to the public. For example, funding could be diverted to an issue that the public does not have a good opinion about, or a major issue could be left unaddressed by a politician.