When face-tracking meets social networks: a story of politics in news videos

Description of the data

Our video dataset consists in the daily-collected NII-TVRECS archive from (Katayama et al. 2005). The capture covers a period between March 17, 2001 and February 27, 2013; of the 4366-day long period, 4259 news programs have been collected cumulating about 2102 h (6.7 TB of video) from the Japanese public NHK channel’s daily News 7 broadcast. The few missing captures concerning mostly the beginning of the time period are due to the capture system setup. Most of the programs usually last 30 mn and only a few of them fall below or beyond this format (news programs may be shorter on Sundays, or longer during commemorative events, see Fig. 1(a)).

News segmentation

News are specific programs that can be segmented in different news topics. We thank the authors of (Ide et al. 2004) for providing us data in which news topics are derived from textual information (captions) synchronized with the news. In summary, a topic boundary corresponds to a point between sentences where the keyword distribution is distinct between preceding and following windows of sentences.

As a result, we have a segmentation of the news by topic, based on semantics analysis (segments in red in Fig. 2-1). To one topic then corresponds what we would commonly call a news segment (both terms may be used interchangeably in this paper). Although we do not have the semantic information of these topics (yet), this gives us time boundaries for news segments which will turn useful for analyzing people’s apparition on screen. Overall, taking into account the differences of lengths among programs, this summarizes in a distribution of an average 13.7 news topics per day (σ=4.3) as illustrated in Fig. 1(b).

However, this segmentation implies that topics are not consecutively segmented, and gaps may occur between two consecutive topic detections. So topic detections cover on average 72.6 % of the shows (σ=0.10), because the beginning (head news summary) and end of the news (weather reports) are ignored by the topic segmentation (Fig. 1(c)). In addition gaps between news segments average 5.1 s (σ=2.4) and can stretch up to 20,7 s (see Fig. 1(d)), in which faces may still be detected.

To help the face-tracking process, a standard shot segmentation derived from color histogram thresholding is provided (segments in blue in Fig. 2-1). This is a contiguous segmentation, without in-between gaps corresponding to video cut editing.

Face-track extraction

The whole process has detected over 30 million faces and 174,778 face-tracks were extracted. We now need to identify and recognize groups of face-tracks, so clustering naturally appears as the following step. However, clustering implies many new issues that we have not yet addressed this work (but we include this goal as part of our future work in “Discussion and future work” Section). Yet, we can still use a different approach to construct our networks, that is of face retrieval.

The faces of 139 characters have been annotated during the evaluation campaign of (Ngo et al. 2013) giving a ground truth for retrieving matching face-tracks. These faces are the faces of well known people among the Japanese media scene, including celebrities and politicians (Japanese and international), for which we had the highest precision of retrieval and identification. In total, over 5 thousands face-tracks were annotated, and 16,714 face-tracks of the 139 different characters were retrieved. This corresponds to 2984 days of news program over the whole archive having matching people, covering a total of 36 h of face tracks.

The coverage of the face-tracks averages 2.4 % of a program (σ=2.6), reaching the maximum of 38.9 % of a program. The ground truth has been provided during the 2010 period, for which the face-tracks appear slightly denser (3.3 % on average). The average screen apparition per person is 15.7 min (σ=32.5) but there is a lot of variation between people (actually it fits a lognormal distribution Fig. 3(a)), and a few people seem to hold most of the screen time (Table 1).

Person	Screentime	#Days	#News segments
Junichiro KOIZUMI	215	516	523
Yukio HATOYAMA	179	330	368
Ichiro OZAWA	138	294	304
Naoto KAN	118	243	249
Shinichi TAKEDA	110	337	491
Shinzo ABE	103	281	298
Yoshihiko NODA	97	195	245
Taro ASO	87	181	187
Yasuo FUKUDA	60	144	139
Seiji MAEHARA	55	116	126
Takafumi HORIE	52	116	121
George BUSH	45	86	93
Kazuo SHII	42	116	98
Sae NAKARAI	40	317	190
Katsuya OKADA	37	115	109
Hideki MATSUI	26	126	73
Wataru ABE	22	84	104

As a result, the tracks work as follows: everyday, we have a news broadcast, and every broadcast contains news segments (topics) and face-tracks of different people. We then observe an average of 23.8 sec of cumulated detections per topics (σ=28.9, with a maximum of 383.0 - Fig. 3(b)), with in average 1.28 people detected per topic (σ=0.66, with a maximum of 7 - Fig. 3(c)). Thankfully, this shows that we can reasonably expect people to overlap across topics, although 80 % of our news segments do not show more than one person detected (see Fig. 3(c)).

Looking closer at the distribution of inter-day occurrences of people in topics, we can see that most of them appear on screen on a daily basis, with bigger gaps then (Fig. 3(d) is an example). This is sometimes referred as a characteristic of “bursty” data (Wang et al. 2007), meaning that, over the whole period of time, there is a lower probability for two people to be detected together than random, making these links especially interesting. We can also notice that most of the people we are tracking seem to take part in similar topics during the 2008–2011 period.

About the 139 people

Some background information is necessary for a good understanding of this news data. With a little domain knowledge, we have classified the 139 characters identified into 9 categories depending on what brought them under the light of news: Politics (71), Sports (27), Culture (11), Business (7), Imperial family (5), Journalism (4), Religion (3), Law (3), and Other (5). Additionally, we have enriched them with their country: 96 individuals are from Japan, among which 39 political figures and 22 athletes.

Incidentally, the Politics class includes 23 international leaders (presidents, prime ministers...). The Japanese prime ministers – hereafter referred as PM – governing during the whole period of capture are of course represented, allowing us to create time frames covering their cabinet(s). Yoshiro Mori was the first PM in the timeline, but his mandate only covers a few weeks from the beginning of the capture, so himself is not included in the people subset. Finally, we obtain 11 time periods (Fig. 9 details them in chronological order, note that the numbers following a PM’s name represent different cabinets formed by the same PM). Figures 4 and 5 reflect these different periods as colored section in their background.

Although Pearson’s correlations between the three measures (Screentime S, #days D, and #news segments T – per person) are very high (S−D=0.95, S−T=0.96, and D−T=0.97), we can use the ranking of the top 10 percentile to extract people of interest (as presented in Table 1).

A background checking gives us supplemental information explaining their occupation of screen space. We can already discard of our analysis the journalists of NHK (S. Takeda and S. Nakarai) who have an obvious aggregating force due to their role on TV. Out of the obvious known figures and the aforementioned Prime Ministers, I. Ozawa, S. Maehara, K. Shii, and K. Okada are famous politicians, and we will talk about them in greater details in “A tool for social analysis” Section. T. Horie is a businessman, H. Matsui is a baseball player, and W. Abe is active on the music scene.

Network of people overlapping on screen

Our first network connects two people when two face-tracks overlap in time. This means that we create a link between two people when they have been detected simultaneously on screen. These links are enriched with the screen duration of the overlapping of tracks as weights.

This network presents 35 nodes and 44 edges, with a main connected component of 29/41 (Fig. 6, left). This connected component is only composed of politicians, with one business person (M. Shirakawa, connected to Y. Hatoyama). It is worth noting that J. Koizumi, the top individual among all other metrics, only presents here a degree of 2. Four nodes stand out in terms of betweenness centrality (S. Abe:0.16, I. Ozawa:0.14, Y. Hatoyama:0.18, and Y. Noda:0.15, with the rest of the dataset below 0.09), and 2 nodes in terms of degree (Y. Hatoyama:10 and Y. Noda:7), however no clear convincing cut of communities is shown by Louvain’s algorithm (Blondel et al. 2008).

A few links stand out in terms of screen duration (over 1000), connecting: Y. Noda and S. Tanigaki, in 2012, I. Ozawa and N. Kan in 2003, 2006, and 2010, Y. Hatoyama and I. Ozawa in 2006, 2010, and 2012, Y. Hatoyama and B. Obama in 2009, J. Koizumi and Kim Jong Il 2002, V. Putin and S. Abe in 2012, T. Aso and Y. Fukuda in 2009. When looking at the number of days in which two different people appear together, we can notice stronger links between: S. Tanigaki and T. Aso in 2006, J. Koizumi and S. Abe in 2002, and H. Clinton and B. Obama in 2008.

Network of people appearing in a same shot

This second family of ties defines links between people appearing in a same shot (i.e. an uncut segment of video). This network roughly extends the previous network, with the difference that people do not need to appear on screen together. Because shot duration greatly varies depending on the cut of the video, we cannot use it as a meaningful metric to weigh edges, instead, we will consider the number of different days that include these shots.

The network (Fig. 6, center) presents 49 nodes for 75 edges with a main component of 41/71. The maximum k-core (k=3) (Seidman 1983) presents a very intricate subnetwork of 18 nodes (Fig. 6, right). It includes the PM, and the main anchorman (S. Takeda), later referred as the ‘main actors’. All the other nodes are politicians, including I. Ozawa. Getting their full list and description may go beyond the scope of this paper, but it is interesting to notice that N. Yamaguchi stands out as the only politician not directly connected to any of the PM. The main component presents a wider range of types of people, including 3 athletes, 3 business people, and O. Bin Laden. A Louvain segmentation does not present a clear cut of denser subgroups in this network. If we remove the ‘main actors’, we can interestingly observe two communities of politicians (the colored areas in Fig. 6, center), one centered on M. Fukushima and N. Yamaguchi, and the other one on K. Okada. However one should carefully interpret the meaning of these links given the low amount of common shots (at most three).

Three edges stand out with links displaying between 5 and 8 days of connections, T. Aso and S. Tanigaki, N. Kan and Y. Hatoyama, J. Koizumi and Kim Jong Il. If we consider links connecting two people over one day only as ‘casual’ and discard them, we can reveal a network of stronger ties of people with ‘recurrent’ interactions (23/26). In this network, I. Ozawa displays the highest betweenness centrality, followed then by the different PM.

Networks of people appearing during a same news segment

The following network connects individuals when they have been detected during a same news segment, based on the topic segmentation described in Section 4. This means that two people are connected when they took part of a same media event. The graph connects 107 people over 507 links with a main connected component of 96/499 (Fig. 7, left). This graph presents characteristics closer to complex networks with a long tail distribution of node degrees (actually fitting a lognormal distribution).

Knowing that co-detection during a news segment is the reason linking nodes, we should first remove the journalists – occurring a lot in the dataset, in order to focus on other people’s interactions. The resulting graph presents a maximal k-core (k=12) gathering 15 Japanese politicians and the 7 PM in a subgraph \(G^{\prime }_{k=12}\) with a density \(D_{G'_{k=12}}=0.79\) (Fig. 7, right).

A degree and centrality analysis will bring focus to the same people identified in the previous networks. To go beyond, we will look at the graph without the ‘main actors’, leaving 67 nodes for 221 edges. This graph clearly presents community structures, and by running a Louvain algorithm, we obtain a very interesting clustering result. The two main partitions (in light greenand orange in Fig. 8) clearly present international politicians and national politicians (respectively). We are now able to spot the non-PM Japanese politicians who played an active role in international matters by highlighting them (circled in purple in the Fig. 8, mostly at the right frontier of the orange community). We do so by counting the number of their ties with international representatives and threshold them based on their cumulative probability distribution (Herman et al. 2000). As a result, we find Y. Edano, S. Tanigaki, S. Maehara, M. Fukushima, Y. Sengoku, I. Ozawa, T. Kanzaki, M. Khomura. With the same process on the other side, we can identify (circled in red) Yu Jiang, Jiabao Wen, and Lee Myung-bak as having redundant apparition on topics with national politicians. The case of Lee Myung-Bak seems to have particularly raised a great interest among national politicians, totalizing 5 connections.

Time slicing the common segment network

The topic network has a dynamic multiplex characteristic – although we do not fully exploit this characteristic in this paper. A link is established between two people when matching in different points in time, which means we have virtually an individual link for each matching at different moment in time. Looking at the network in different timeframes will result in different arrangements of the links between nodes and different weights if these multiple links were to be collapsed in one single interaction. Thus, thanks to well defined periods of time corresponding to PM cabinets, we can use topic segmentation as a support to observe not the overall network but each slice involving the people’s interactions over the different cabinets (Fig. 9).

Before we tackle the political analysis in Section 4, we can quickly compare the political landscapes of each cabinet. To do so, we pick out the top 3 Japanese politicians in ranking of centrality and number of news segments, who are neither a PM nor have been detected during the preceding cabinets. In total we have collected 21 prominent politicians, which will be used to compare cabinets one to another. This creates a vector of all politicians per cabinet.

Based on these vectors of 21 (+ 7 PM) politicians, we can finally estimate a rough (Jaccard) proximity between cabinets as shown in Fig. 10. The periods from Abe 1 to Noda known for the series of resigning PM, shows the highest proximity one to another, and interestingly to Koizumi 1. However, Koizumi’s two following cabinets appear very different, suggesting that he set a very different media/politics scene during this time.

Preliminary observations

Before we solely focus on a political analysis of the news landscape, the exploration of these networks led us to some understanding of the media/politics scene presented by NHK News 7. Based on this data, together with the knowledge of people, we can confirm that the different PM stand out like no one else in the NHK news. They can be directly identified in all aspects of the data: first, purely quantitatively speaking, they occupy most of the media scene during their own cabinets; then, in the different networks, they also occupy a very central place; the different time-related analysis makes it especially obvious during their cabinets.

We also learn by looking at individual PM: most of them show some level of activity before their mandate and we can observe two opposite cases. On one side, Abe is actually more central than Koizumi himself during Koizumi 3 (Fig. 9(d)). On the other side, Noda came ‘out of nowhere’ before becoming PM (Fig. 4). Additionally, despite of Hatoyama and Aso appearing quite strong nodes in the different networks, they have never been detected on screen together (Fig. 6) even if they were heading two consecutive cabinets in period of time where the media/politics scene of consecutive cabinets is very similar – maybe because they are the leaders of two opposite parties.

Strikingly, one very particular politician comes out all along this study, I. Ozawa, who is (in)famously known as the “Shadow Shogun”. Getting into the details of Ozawa’s role in the Japanese politics is a fascinating work on its own (Meyer 2014), but put in short, after being leader of the opposition, he is known for all the connections and roles he has played behind the scene, building alliances and often changing side – although never he became PM.

Another very interesting point which is worth noting concerns the Imperial family. The Japanese Constitution forbids the Imperial family to take any part in politics, and observing the links surrounding the members of the family are of high interest to survey their actions. Our system finds very little connections (purple nodes in Fig. 9(c), (d), and (h)): they mostly concern the revision of the Imperial Household Law because of the issue concerning the succession to the Imperial Throne.

Political affiliations

We first need to subset the data: because the Mori and Abe2 cabinets have only partial data, we will discard those from our analysis. We will also focus only on the national and international politically involved actors, reducing to 75 tracked individuals (including 40 Japanese politicians).

For each of the 40 Japanese politicians, we have manually collected their political affiliations over time. We collected in total 18 different parties (including independent politicians and the many minor parties) represented among the 40 politicians. As illustrated in Figs. 10 and 11, the two main parties of the Japanese politics are the Liberal Democratic Party (LDP) – ruling the cabinets from Koizumi1 to Aso – and its main opposition the Democratic Party of Japan (DPJ) – ruling from Hatoyama to Noda.

As we can see from Fig. 11, the LDP and DPJ accumulate most of the affiliations and many small parties only have one or two members tracked in our dataset. A few politicians have created independent parties at different moments in time. Different coalitions of opposing politicians appeared especially in the end of Noda’s cabinet when it was foreseen that DPJ would not win the next election. This explains why we obtain in total more affiliations than we have politicians.

We are interested in studying how the balance of power reflects in the NHK news. In modern democracies, there are often two main parties: a party in power and its main opposition. In the case of Japan, the leader of the party in power is the Prime Minister, and the leader of the main opposition (hereafter OL) is the president of the main party not in power (which is here either DPJ or LDP). However, this leader changes over time such as more than one politician may lead the opposition during a cabinet. To study the opposition, we created an artificial node entitled “Opposition leader” (visible in Fig. 13). We extracted from the different persons representing the OL all their detections during their leading mandate and unified them altogether in this abstract entity. This allows us to question the place of the Opposition leader in different the political networks.

Political networks

Given this information, we can color the nodes of the previous networks using their political affiliations Fig. 12 – the category foreigners and others includes Japanese business people and religious figures who could be involved in political matters. The coverage (number of news segments in which a politician has been mentioned) encodes the size of a node and of a link (for which the co-occurrence is measured). A longitudinal analysis is only permitted using the time sliced topic networks. We can notice that the editorial choice of the NHK seems to well follow the governmental trend, i.e. we detect more LDP members when LDP is ruling, and DPJ members when DPJ is ruling.

A small analysis of the political networks

With all uncertainties taken into account (which will be discussed in “Discussion and future work” Section), we want to use the generated networks as support for social analysis. Inspired by recent works in social science (Jou and Endo 2015), general questions drive our study: How is the opposition represented during the different cabinets? Can we find traces of “presidentialization” of the politics in the media? These questions can of course be tackled quantitatively by studying the coverage of the different actors, but can we learn something else from a social network analysis perspective? To investigate these questions, we propose to observe two phenomena: firstly we can study the place of political leaders in their social networks, secondly we can have a thorough look at the links between the two political parties.

Although we cannot measure a clear trend such as a growing importance of the OL over time, we can notice a couple of facts. Most often, PM and OL get the top structural positions in the different networks. Except maybe for the PM, more coverage for a politician does not mean higher position in terms of structural measures. We also have two remarkable cases. First is the case of Koizumi3 in which the OL reaches higher centrality than the PM himself. This is due to the inclusion of a very broadcasted and connected S. Abe who tops all metrics. Then comes the cabinet of Kan: the main opposition does not seems very represented so the OL does not hold an important role. As we mentioned earlier, during this cabinet, NHK was probably more focused on reacting to the tragedy, probably displaying alignment of ideas rather than a diversity of point of views. Figure 14(b) reports the timeline of the news segments attributed to the PM and the OL during Kan’s cabinet, and the last detection of the OL in our dataset only happened a few days before the tragedy.

To study the interactions between the two parties, we can plot the share of news segments that involves each of the two main parties, alongside with the ones that involves both of them together (Fig. 15(a)). It is expected to see most of the cabinets having the interaction of their own members put forward in the media during their own cabinet. This is especially true during all DPJ cabinets, Koizumi and Abe’s first cabinets. Aso’s cabinet displays the opposite behavior. Indeed, it became clear at some point that DPJ will win the following election. At this moment, the media increasingly reported the activity of the opposition (as we can see from the timeline in Fig. 14(a)). This same behavior is confirmed by looking only at the activity of the main figures of both parties (Fig. 15(b)) suggesting that most activities in the political networks are supported by these strong characters.