This study involves the creation of a market sentiment index using data from the social network Twitter and Thomas Renault’s (2017) dictionary. An analysis of the link between market sentiment and its benchmark index (in this case the S&P500) is done first before giving way to a specific analysis of the link between uncertainty and market sentiment, the results of which confirm Baker’s intuitions.
The use of social networks has allowed us to enter into a new dimension in the estimation of investor sentiment. Indeed, they reach the same results as surveys’ while freeing themselves from the obvious constraints of the latter. Several studies have thus emerged, each with its own interesting conclusions. Siganos (2014) found by analysing Facebook data that the link between sentiment and the market was stronger for small caps, going in the direction of Baker’s hypothesis. Renault (2017), on the other hand, looked at intraday sentiment and finds that sentiment in the first half-hour of the session is useful for predicting performance in the last half hour of the session.
Like Sprenger (2010) or Ranco (2015), it was chosen to use the social network Twitter to build the database. It is an interesting social network in that all users can interact with each other, even if they do not follow each other. This allows people with no relationship to discuss a common topic. As noted by Sprenger (2010), the format of Twitter means that it makes it possible to have discussions just like one might have in a trading room.
All tweets are about S&P500 companies (identified by the $ticker cashtag) and are extracted by connecting to the Twitter API via Python. The database is composed of 3 million tweets spanning from October 4, 2019 to May 26, 2021 (600 days). That corresponds to a number of 150,000 tweets per month. For comparison, this is more than Sprenger (2010) or even Ranco (2015) who had each constituted databases of 250,000 tweets (42,000 per month) and 1.5M (100,000 per month). Each of the tweets contains the following information: date of creation of the tweet, author id, number of retweets and quotes, number of followers of the author, and the text of the tweet.
For scoring each tweet, it was first necessary to standardize the text in the same way as Thomas Renault (2017) because it will be used the same method as him to analyse them. Thus, each tweet was first put in lower case. Positive emojis (negative emojis) were replaced by the word «emojipos» (»emojineg»), numbers by the word «numbertag», cashtag by «cashtag», mentions by «usertag» and URL links by «linktag». The only stopwords that have been removed are the words «a», «an» and «the». Finally, negations with words such as «not», «no», «none», «neither», «never», «nobody» have been replaced by the attribute «negtag_» in front of the following word. Thus, «none of» becomes «negtag_of». Finally, the text is divided into bags of words of one or two words, thus considering the impact that two consecutive words can have on the meaning of a sentence.
For textual analysis, we chose to use the dictionary method. A dictionary is a lexicon containing words that are associated with a positive or negative score. For this method, we simply look for each word in the text if the word is present in the dictionary used and assign its score if necessary. This method seems very simple at first sight but it can be difficult to implement. Indeed, depending on the context of the study, words can have different meanings. Thus, in our context, a dictionary that is not specific to finance (Harvard IV) is not appropriate and neither is a dictionary that is not specific to the language of social networks. Indeed, it is very complicated to analyse social network data with dictionaries designed for other uses such as the analysis of press articles or corporate annual reports (Loughran and Macdonald), even if this one is designed for finance, as the sentence constructions are very different.
Therefore, Thomas Renault (2017) decided to create his own dictionary using data from the social network Stocktwits, which has the advantage of allowing users to label their own posts with bull, bear, or neutral. What makes it different from the other two mentioned above is that it was designed with texts from social networks. For example, it considers emojis but also some expressions like «lol» very used on this type of platform.
For the rating of a tweet, we simply take the sum of the scores of the words in it. The date of each tweet is calibrated so that tweets occurring after the closing time of the New York Stock Exchange are associated with the next day. In addition, tweets are weighted by the number of retweets and quotes as shown in formula 1.
The first observation that can be made is the clear link between the market sentiment index and its benchmark index. Indeed, the two indices track each other perfectly and this is confirmed by the correlation coefficient of 77%. All the lows of both indices occurred in March, at the very beginning of the pandemic. Logically, this period caused a psychological shock to investors who were faced with an unprecedented situation. At that moment, panic appeared in the markets, which is very well reflected by the market sentiment index created. We can see that the reaction occurred just after the announcement of the rate cut and the restart of quantitative easing by the FED, a reaction that occurred very quickly and was seen as reassuring by investors.
From July 2020 onwards, we gradually enter an upward trend in market sentiment. At that time, the European Union’s recovery plan is announced. A few months later, it was the turn of the first vaccine to appear and the uptrend continued (the 50 highest points in market sentiment occurred from that moment on), even though there was a decorrelation at the end of the period with this drop in sentiment while the market continued to rise.
Since the two variables are not stationary, they need to be transformed and their correlations recalculated. Thus, the correlation between the first difference of the market sentiment index and the S&P500 return is 27%. This is much higher than the correlation calculated by Sprenger (2010) who calculated a correlation between the SP100 return and his bullishness index of 15%. This result is encouraging as to whether or not there is a link between market returns and investor sentiment, and thus the importance of behavioural finance. One of the reasons for this much higher correlation than Sprenger (2010) is that the period analysed here is a very uncertain one, whereas Sprenger deliberately chose a calm period.
Once the first descriptive analysis of the market sentiment index and the link with its reference index is done, we can now proceed to the first regressions. Thus, we will proceed to 3 regressions: one where we will try to explain the evolution of the market with the sentiment, one where we will try to predict the market with the sentiment, and one where we will try to predict the sentiment with the market.
To work in stationary variables, we will take the first difference of the sentiment index. Although the first difference of the S&P500 is also sufficient to make it stationary, we will work on returns to facilitate the interpretation of the results and be in line with other studies on the subject. As control variables, we use the dependent variable at the previous value, the variation of the VIX (first differences) and the volatility of the S&P500 for the predictive regressions. For predictive regressions, the dependent variable is explained with the explanatory variable at the same period. For predictive regressions, the dependent variable is predicted with the previous period as well as the period before (see formula 2).
All our models when initially constructed had heteroscedasticity in the residuals as well as autocorrelation. The HAC estimator which corrects these anomalies was then used.
For the first model, all the coefficients are significant and the R² is even very high since it is 65%. It is surprising to see that the market return in the previous period is significant but this can be justified by the uncertainty of the COVID period and the negative coefficient (-0.11) is in line with this justification. It is not surprising to see the VIX being significant since the period analyzed represents a very turbulent and uncertain period. On the other hand, it is interesting to note that even when controlling with the VIX, the market sentiment variable is significant at the 1% level. This not only validates the existence of subjectivity in decision making that could be influenced by sentiment but also dissociates this notion of sentiment from the VIX representing the uncertainty surrounding the markets.
It should be noted, however, that even if it exists, the influence of sentiment on the market is limited. Indeed, by redoing the regression by removing the variables one by one, we can check the importance of each variable by calculating the new adjusted R² (necessarily lower) and thus the loss of R² associated with the removal of this variable. Even if the loss of R² is higher for market sentiment than for the SP500 return at the previous period (5% versus 1.5%), it is much lower than for the VIX (66%). This result is reassuring, however, as no one would want a market where most of the variations are explained by investor sentiment. Finally, the influence of sentiment on the market is positive, which is what one would expect at first sight.
For the predictive regression of the S&P500 (the second model), we are left with only two significant variables: the S&P500 return at the initial period and the market sentiment at the previous period (two periods before the return we want to predict). Thus, the sentiment of two days before affects the market but the sentiment of the previous day does not, contrary to what one might think. Now, the VIX is no longer significant and its share in the R² is minimal. The variable that has the biggest importance in the R² is the S&P500 return at the initial period (15%) followed by the market sentiment (6%). Thus, the latter can exert an influence on future market returns, which confirms the results of the first model and the existence of a link between sentiment and market. The adjusted R² (11.73%) is relatively high given the difficulty of explaining future market movements.
If the influence of market sentiment on index performance has been established over the observation period, it may also be interesting to look at the impact of the S&P 500’s performance on market sentiment to see if there is a mirror effect (Model 3). Several studies have concluded that the influence of the benchmark return on market sentiment is greater than the influence of sentiment on its benchmark. This phenomenon allows us to explain bubbles even more precisely. Indeed, in this case, a speculative bubble would be a period during which the influence of the sentiment on the index return as well as the influence of the index return on the sentiment would both be positive with coefficients larger than the average, thus leading to a snowball effect.
The significant variables in this predictive model are S&P500 return, market sentiment and VIX, all at the initial period. The coefficient on market is positive while those on sentiment and VIX are negative. The R² of the model (19.30%) is much higher than that of the predictive model 2 (11.07%), suggesting that sentiment is a more easily predicted variable than the market itself. This makes sense because the market is surely influenced by many more factors than sentiment itself. However, it is interesting to note that sentiment absorbs a very large amount of the R² (65% loss of R² by removing the variable). Thus, even if sentiment is more easily predictable, it is mainly predictable only by itself. The coefficient on sentiment (negative) implies a correction or indecision effect over the period.
After having built the market sentiment index and having been able to perform different statistical and econometric tests in order to verify its coherence and usefulness, it will be possible to move on to the next and final part of the project by proceeding to the construction of 3 different market sentiment indices. Indeed, the reference index (the S&P 500) will now be divided into three different groups (from the most volatile stocks to the least volatile ones) and for each of these groups, a market sentiment index will be created. The explanatory and predictive power of the index on its market will then be tested in the same way as it was done for the previous part. The objective is to study the influence of sentiment (the non-rational part of decision making) on assets as a function of uncertainty. It has already been identified that one possible explanation for the strength of the previous results compared to other studies could be the high uncertainty that characterizes the observation period. However, this section will formalize this fact.
As a proxy for uncertainty, it was chosen to simply take the volatility of the asset returns over the period. The assumption made is that sentiment plays a more important role in the decision as the uncertainty surrounding a company increases and the determination of its fair value becomes much more complicated to calculate.
We extract the price data of the 505 firms constituting the S&P500 over the period and calculate the returns and then the volatility of the returns over the period. 4 companies out of the 505 do not have data over the whole period so we remove them. We then divide the remaining 501 firms into 3 groups of 167 firms each according to their volatility. Group 1 represents the most volatile companies and group 3 the least volatile companies. As an example, Tesla is in group 1, Apple is in group 2 and Amazon is in group 3.
Once the groups are created, we can create a sentiment index for each of them by considering only the tweets with the concerned cashtags. The method of rating the tweets is not modified compared to the previous sections (use of Thomas Renault’s dictionary and logarithmic weighting in relation to the number of retweets and quotes). These three sentiment indices are obviously highly correlated, with correlations ranging from 91% to 95%.
Then, 3 market indices are created, each taking into account only the assets in the category. The indices are weighted by market capitalization. Due to the difficulty of finding market capitalization data for each company over the entire period, the market capitalization of the first day was taken. The evolution of the market capitalization is then done by multiplying the capitalization by the return on assets. The weakness of this method is that it does not consider capital increases. Once again, the correlations are strong, ranging from 82% between group 1 and group 3 to 98% between group 2 and group 3. Note that the S&P500 has a minimum correlation of 90% with each of the indices and that its highest correlation is logically with the market index number 2 (99.81%).
Finally, the same work as in the previous part will be done for each group, that is the construction of an explanatory model and a predictive model of the market index. Thus, there are 6 models to build, each one respecting the following equations (two per group) (see formula 3).
First, we make the same observations regarding the significance of the coefficients for each group as for the S&P500. Indeed, the three coefficients of the sentiment variable for the explanatory regressions are highly significant (only the coefficient of the third group has a p-value higher than 1%). For the predictive regressions, none of the coefficients on sentiment one period before are significant while all those two periods before are significant, exactly as with the S&P500 predictive regression (regression 2 in the table above).
It is interesting to compare the models between the groups themselves. First of all, we note that the same pattern of results is present for the explanatory models as for the predictive models. We observe that the coefficients associated with the sentiment variables of group 1 (the most volatile) are always higher than those of group 2, which themselves are higher than those of group 3 (the least volatile). The adjusted R² of the models follow the opposite trajectory, with those of the least volatile group always being the highest and those of the most volatile group being the lowest. Thus, even if the impact of sentiment seems higher (in explanatory and predictive regression), it is not enough to counterbalance the increase in volatility and does not necessarily lead to a more accurate explanation or prediction. Finally, the loss of R² obtained by removing the sentiment variables from the models follows the same trajectory as the coefficient: it is stronger for the most volatile group. These results show the relatively higher importance of sentiment when volatility is higher and are consistent with the central hypothesis that sentiment is more important in decision making as uncertainty increases.
This study focuses on the use of alternative data (in this case from social networks) to create a measure that is difficult to observe at first glance: investor sentiment. It first demonstrated the added value of alternative data by showing the link between the sentiment index and its market, the S&P500, through the construction of several explanatory and predictive linear models. By developing a hypothesis posed by Baker in 2007 according to which the greater the uncertainty, the greater the role of sentiment in decision making, it analyses more precisely the link between volatility and the impact of sentiment on the market itself. By dividing the S&P500 into three groups, from the most volatile to the least volatile, this study shows that the impact of sentiment (measured by its coefficient and its importance in the R² of the model) is indeed stronger when volatility is also stronger.
To continue this reflection and improve it, it would be possible to carry out a more rigorous analysis of the frequencies. Indeed, this research focuses mainly on the relationships between the variables over the entire observation period, i.e. 600 days. We are therefore interested in a relationship that is more similar to a medium-term relationship. On the other hand, it seems reasonable to assume that, like most observed relationships, the relationship between sentiment and the market is not stable over time. This could explain financial bubbles, which would be periods during which sentiment has a strong positive impact on the market and the market a strong positive impact on sentiment over a given period. Thus, a complementary study on the data could be done in this sense on the same model as the one done by Zhengke Ye (2020) where a use and extension of the Wavelet Coherence Transform is done. Such a study could identify periods during which the relationship between sentiment and the market is abnormally high. n
