Points allocation in the Eurovision Song Contest
Introduction
Background
This project will use data from the Eurovision Song Contest. The Eurovision Song Contest is a yearly competition where different countries submit an artist and song to compete in a live televised event against other countries. Each participating country awards points to their favourite songs and the country with the most points is allowed to host the contest the following year. The first contest took place in 1956 and included 14 songs from 7 countries. Since then, the contest has grown dramatically, with the 2023 competition hosted in Liverpool including songs from 37 countries. In recent years, the competition has gained popularity in countries outside Europe, with Australia joining the contest in 2015, the competition being televised in China and America, and America staging their own version of the competition in 2022 called the “American Song Contest”.
Over the 67 years that the competition has been held, the voting system used to determine the winning song has changed drastically. Changes in the voting system have affected the number of points awarded to each country and therefore, the points total needed to win the competition.
Project Aim
This project will seek to visualize how changes in the voting system have affected the number of points allocated in each year. Specifically, this project will seek to visualise differences in the number of points accumulated by the winning song in each year of the competition, and how this relates to the voting system used in each year.
Project organization
The /raw folder contains the raw data used for this project and /figs contains the figures produced from this project.
A codebook can be found in the project folder, which describes the variables and the functions used in the project.
Loading Packages
The renv package was used to store the packages used in this project, and to keep an account of the package versions that were used. These can be found in the file /renv.lock.
#Load packages with renv
if (!require('renv'))
{
install.packages('renv');
}
library(renv)
renv::restore()
#Load packages
library(dplyr)
library(ggplot2)
library(here)
library(janitor)
library(plotly)
library(tidyverse)
library(toOrdinal)
library(rvest)
library(htmlwidgets)
library(orca)Data Origins
The raw data was scraped from the Eurovision Website https://eurovision.tv/history. The data for each year of the competition is located in nested links from this page. You can only access the data from one year at a time, by selecting the year that you would like to view on this page. Selecting the year would take you to an overview page that would appear in the following format https://eurovision.tv/event/turin-2022. From this link, you have to select whether to view data from the semi finals or the final. Semi-finals were introduced in the 2004 competition as a way of allowing more countries to compete. This project is only interested in gathering data from the final shows since this will contain the data relating to the winning songs. The following link contains a data table showing the results from the 2023 competition “https://eurovision.tv/event/liverpool-2023/final”. The data is presented in the same format for each year of the competition. For this project, I needed to scrape the tables from the URLs for each year and combine them into a large dataset.
Data scraping
The approach used to scrape data
I first had to scrape the URL links that contained the data tables for each year. These links were scraped from the page https://eurovision.tv/history using the web browser extension SelectorGadget. A description of SelectorGadget is provided at the following link: https://rvest.tidyverse.org/articles/selectorgadget.html.
A problem I encountered at this stage was that the URL links varied in an unexpected way. For the competitions before 2004, the URL link containing the data tables finished with “/final”. However, for the competitions that included a semi final (from 2004 onwards), the URL link finished with “/grand-final”. This inconsistency meant that I was unable to scrape all the URL links using a single piece of code. To resolve this issue, I had to scrape the data using four sets of code. Each piece of code was relatively similar, except that they created a list of URL links that ended in “/final” or “/grand-final”. One set of code was used to scrape data from page 1, containing the year links for competitions between 2008 and 2023. Another set of code was used to scrape the year links from pages 3, 4 and 5, which contained year links for the competitions between 1956 and 1992. Two sets of code had to be used to scrape data from page 2, which contained data from competitions that had “/final” at the end of their URL (1993-2003) and “/grand-final” at the end of their URL (2004-2007).
Once I had collected the URL links, I scraped the data tables from each URL link using the lapply() function. The scraped tables were added to four data frames in R, one for each piece of code. Below is an example of the code used to scrape data from pages 3, 4 and 5. The rest of the code used to scrape data from pages 1 and 2 is provided in the R Markdown file.
#-----------SCRAPING DATA FROM PAGES 3, 4 & 5 -------------
scrape56 <- data.frame() # declaring scrape56 will be a data frame
# using for loop to download the URL's for from multiple pages
for(page_result in 2:4) {
link = paste0("https://eurovision.tv/history?page=", page_result)
page = read_html(link)
# getting nested links for each year of the contest
year_links = page %>%
html_nodes(".views-field-field-event-year a") %>% # Asking R to find this data on the webpage
html_attr("href") %>% # Extract href (e.g., turin-2022) from scraped data
paste0("https://eurovision.tv", ., "/final") # Pasting the href into the URL
# retrieving the tables and the years from each nested link
get_data <- lapply(year_links, function(i) {
year_page = read_html(i) # Reading each URL link into R
year_data = year_page %>%
html_nodes("table.cols-7") %>%
html_table() %>%
.[[1]] # Scraping the data tables from each URL link
year <- str_extract(i, "\\d{4}") # Extracting the year from the URL links
year_data$year <- year # Adding the year data to the data tables
return(year_data) # Asking R to return the datatable
})
# Adding all data to a new datatable called scrape56
scrape56 = rbind(scrape56, data.table::rbindlist(get_data))
}Combining the scraped data
Once all the data was scraped from the Eurovision website, the four dataframes were combined using the rbind() function. The final dataframe containing all the data was saved as a CSV file.
# Combining the four datasets into a dataframe and saving it as a csv
df_esc <- rbind(scrape08, scrape04, scrape93, scrape56)
write.csv(df_esc, here("raw", "esc_raw_data.csv"))A limitation of the data scraping method
The Eurovision webpage used to scrape the data for this project changes every year around early May, when a new competition is staged and the data from this competition is added to the webpage. The data from the 2023 competition was added on the 14th May 2023. The data scraping method used in this project is not flexible to these changes. Four pieces of code were used to scrape data from specific years of the competition. Each piece of code creates a list of URL’s to scrape data from, which either end in “/final” or “/grand-final”. Any changes to the website may affect the data that each piece of code attempts to scrape. For example, the code used to scrape data from the competitions between 1993 and 2003 may attempt to scrape data from the 2004 competition. Since the data for the 2004 competition is located in a URL that ends with “/grand-final” rather than “/final”, the R code would be attempting to locate a URL that does not exist. Therefore, once the data from the 2024 competition is added to the Eurovision website, this R code may fail to scrape the data fully.
By saving the scraped data as a CSV file, I have tried to minimise the impact that changes to the Eurovision website have on the rest of this R project. The data management and visualisation sections of this project will use a new data frame, produced by reading the CSV file into R.
df <- read.csv(here("raw", "esc_raw_data.csv"))
head(df)## X R.O..Sort.descending Half Country Participant
## 1 1 1 1st half Austria Teya & Salena
## 2 2 2 1st half Portugal Mimicat
## 3 3 3 1st half Switzerland Remo Forrer
## 4 4 4 1st half Poland Blanka
## 5 5 5 1st half Serbia Luke Black
## 6 6 6 1st half France La Zarra
## Song Points Rank year
## 1 Who The Hell Is Edgar? 120 15th 2023
## 2 Ai Coração 59 23rd 2023
## 3 Watergun 92 20th 2023
## 4 Solo 93 19th 2023
## 5 Samo Mi Se Spava 30 24th 2023
## 6 Évidemment 104 16th 2023Description of the data
The data contains 8 variables, which are: R/O Sort descending is the order in which the songs performed in the contest, Half is whether the country performed in the first or second half of the contest, Country is the name of the participating country, Participant is the name of the artist, Song is the name of the song, Points is the total amount of points each country received, Rank is the position which the song achieved in the final, and year is the year of the competition.
Data Preparation
Cleaning the column names
The column names contain capitals and special characters, which may cause errors in the R code and will not look appealing in a visualisation. For this reason, I used the clean_names() function from the janitor package to tidy these names.
#--------------TIDY DATA------------------
# cleaning the column names
df <- clean_names(df)Removing the duplicate rows for 1969 competition
Since there were multiple winning songs in 1969, the 1969 competition has been repeated four times on the page “https://eurovision.tv/history?page=3”. This has resulted in the 1969 data being scraped multiple times. The below code was used to remove the duplicate rows for the 1969 competition.
The same set of code has also been used to drop columns that contain irrelevant data. The following columns were dropped because they contained data that would not be used for the visualisation: half, r_o_sort_descending and x. The x column was added when the data was saved as a CSV and then read into R. The half column only contains data for the 2021 competition, so will not be useful when trying to compare data across the years of the competition. The r_o_sort_descending may have been interesting to visualise, but will not be useful for this visualisation.
df1 <- df %>%
select(year, country,
participant, song,
points, rank) %>% # dropping the unnecessary columns
unique() #remove duplicate rows Replacing missing data in the dataset
There are 51 pieces of missing data in the dataframe, under the points column. The Eurovision Website is inconsistent in how it represents zero points in the datatables, with some years using the number zero and others using a blank space. When the data was scraped from the Eurovision website, these blank spaces were converted to NA’s. Since the NA’s may lead to errors when visualising the data, I converted them back into zeros.
A large proportion of the missing data is from the 1956 competition. The reason for this is that the points allocation process was not made public for the 1956 competition. The lack of any useful points data for the 1956 competition meant that this data could be removed from the dataframe.
# To count the NA's in each column: colSums(is.na(df))
# Dealing with missing data
df2 <- df1 %>%
replace_na(list(points = 0)) %>% # replace missing data with 0's
filter(year != 1956) # filter out data from 1956Voting system data added to the dataframe
Voting system data was taken manually from the website Eurovision.World.com, a fan website dedicated to the Eurovision Song Contest (https://eurovisionworld.com/esc/voting-systems-in-eurovision-history). The reason this website was chosen was because the information is not presented on the Eurovision website. Other information on the Eurovision.World.com has been checked with the Eurovision website for accuracy such as the points totals in specific years of the competition. The voting system data was not scraped directly from Eurovision.World.com because the data was not presented in a format that would complement a visualisation. The descriptions were too long and may have been difficult for a non-Eurovision fan to understand. Instead, the data was summarised and added to seven voting systems variables in R. The number at the end of each variable name represents the year that the voting system was introduced. Using the mutate function, this data was added to the data frame as new column called voting system.
# Adding voting system data to variables in R
voting_system16 <- "Two sets of points (12, 10, 8, 7, ....1) awarded to ten songs"
voting_system75 <- "One set of points (12, 10, 8, 7, ....1) awarded to ten songs"
voting_system57 <- "10 points split between one to ten songs"
voting_system62 <- "One set of points (3, 2, 1) awarded to three songs"
voting_system64 <- "One set of points (5, 3, 1) awarded to three songs"
voting_system63 <- "One set of points (5, 4, 3, 2, 1) awarded to five songs"
voting_system71 <- "2-10 points awarded to each song"
# adding a new column for voting system to the dataframe
df3 <- df2 %>%
mutate(voting_system = case_when(year > 2015 ~ voting_system16,
year > 1974 & year < 2016 ~ voting_system75,
year == 1957 | year == 1958 |
year == 1959 | year == 1960 |
year == 1961 | year == 1967 |
year == 1968 | year == 1969 |
year == 1970 | year == 1974 ~ voting_system57,
year == 1962 ~ voting_system62,
year == 1963 ~ voting_system63,
year == 1971 | year == 1972 |
year == 1973 ~ voting_system71,
year == 1964 | year == 1965 |
year == 1966 ~ voting_system64))
# Changing the order of the voting systems in the dataset
df3$voting_system <-
factor(df3$voting_system,
levels = c("10 points split between one to ten songs",
"One set of points (3, 2, 1) awarded to three songs",
"One set of points (5, 4, 3, 2, 1) awarded to five songs",
"One set of points (5, 3, 1) awarded to three songs",
"2-10 points awarded to each song",
"One set of points (12, 10, 8, 7, ....1) awarded to ten songs",
"Two sets of points (12, 10, 8, 7, ....1) awarded to ten songs"))Extracting the winning songs
Since the visualisation will only present data for the winning songs, I needed to filter this data out of the dataset. I felt that it would be a more intuitive and interesting to visualise the points totals for the winning songs than to visualise the total or average number of points.
# Filter data that was ranked 1st
df4 <- df3 %>%
filter(rank == "1st")
head(df4)## year country participant song points rank
## 1 2023 Sweden Loreen Tattoo 583 1st
## 2 2022 Ukraine Kalush Orchestra Stefania 631 1st
## 3 2021 Italy Måneskin Zitti E Buoni 524 1st
## 4 2019 Netherlands Duncan Laurence Arcade 498 1st
## 5 2018 Israel Netta TOY 529 1st
## 6 2017 Portugal Salvador Sobral Amar Pelos Dois 758 1st
## voting_system
## 1 Two sets of points (12, 10, 8, 7, ....1) awarded to ten songs
## 2 Two sets of points (12, 10, 8, 7, ....1) awarded to ten songs
## 3 Two sets of points (12, 10, 8, 7, ....1) awarded to ten songs
## 4 Two sets of points (12, 10, 8, 7, ....1) awarded to ten songs
## 5 Two sets of points (12, 10, 8, 7, ....1) awarded to ten songs
## 6 Two sets of points (12, 10, 8, 7, ....1) awarded to ten songsCollapsing the rows for the 1969 competition
The new dataframe contains four rows for the 1969 competition, because there were four winners in 1969. I wanted to collapse these four rows so that the information for all the winning songs, participants and countries appears in a single row. This would be beneficial when adding the information to a scatter plot as it meant that I could add a label to the data containing all the information for the winners in the 1969 competition. Collapsing the four rows was made easier by the fact that the winning songs in 1969 had the same points total, rank and year.
# Adding the four country names to each row for 1969, separated by a comma
df4$country <- ifelse(df4$year == 1969,
paste(unique(df4$country[df4$year == 1969]), collapse = ", "),
df4$country)
# Adding the four song names to each row for 1969, separated by a comma
df4$song <- ifelse(df4$year == 1969,
paste(unique(df4$song[df4$year == 1969]), collapse = ", "),
df4$song)
# Adding the four participant names to each row for 1969, separated by a comma
df4$participant <- ifelse(df4$year == 1969,
paste(unique(df4$participant[df4$year == 1969]), collapse = ", "),
df4$participant)
# Removing the duplicate columns for 1969. One row for 1969 left
df5 <- df4 %>%
unique()Data Visualisation
Visualisation 1 - Reasoning
A scatter plot was most appropriate to visualise how the data changed over time. The year was used as the x axis and points as the y axis, with voting system as the grouping variable. The scale of the x axis was manually set to increase in sets of 5 to make the year data easier to read. The size of each point was increased so that more of the graph was filled and the graph would look less empty and more appealing. However, the size was not increased further to avoid too much overlap between the points.
The plot was converted into a plotly interactive graph using the function ggplotly. This meant that the graph contained labels for each point which would appear when they were hovered over with the cursor. The text contained within each label was the year, the winning country, the song, the participant and the number of points they received. This information was important to include as it would provide further context as to why a winning song may have received more or less points. The datapoint for the 1969 competition will contain the data for the four winning songs in the year. Another benefit of using a Plotly graph is that you can double click on a voting system that you would like to display data for and the plot will isolate the datapoints from this voting system. Plotly can make it difficult to display titles, subtitles, and captions on a ggplot graph. These had to be added in once the graph was converted into a plotly graph using the layout() function. The choice was made to display the legend in a horizontal fashion below the graph. This was appropriate as the legend was long and would have reduced the space available for the graph if presented vertically alongside the graph.
Visualisation 1 - Labels
The labels for visualisation 1 were assigned to variables to enhance the readability of the R code used to create the plot.
#--------------PLOT DATA------------------
# labels that will be added to the plot
plottitle <- "Points allocation in the Eurovision Song Contest"
grouplab <- "Voting system for each country"
caplab <- "Source: eurovision.tv/history"
subtitle <- "Points total of the winning songs 1957-2023"Visualisation 1 - R code
### Creating visualisation 1
# constructing the plot using ggplot
# Year as the x axis and share of points on the Y axis
# Voting system used as colour and text added for labels
p1 <- ggplot(df5, mapping = aes(x = year,
y = points,
col = voting_system,
text = paste("<br>Year:",year,
"<br>Winning country:",country,
"<br>Artist:",participant,
"<br>Song:", song,
"<br>Points:",points)))
# Adding additional elements to the plot e.g., points, scale, title and legends
p1 <- p1 + geom_point(size = 2.5) +
scale_x_continuous(limits = c(1955, 2023),
breaks = seq(1955, 2023, 5)) +
labs(title = plottitle,
x = "Year",
y = "Points",
colour = grouplab)
# Converting the ggplot into a ggplotly animation with hover labels
p1 <- ggplotly(p1, tooltip = c("text"), width = 800,
height = 600) %>% # Converting to a plotly graph with specified size
layout(margin = list(l = 30, r = 30, b=60, t = 60, pad = 4), # Changing the margins of the plot
title = list(text = paste0(plottitle,
'<br>',
'<sup>',
subtitle,'</sup>')), # Adding a subtitle to the plot
annotations = list(x = 1, y = -0.2,
text = caplab,
showarrow = FALSE,
xref = "paper",
yref = "paper",
xanchor="right", yanchor="auto",
xshift = 0, yshift=0,
font=list(size=13))) # Adding a annotation to the plot
p1 <- p1 %>% layout(legend = list(orientation = 'h',
xanchor = "center",
x = 0.5,
y =-0.2)) %>% # Changing the legend to be horizontal
style(legendgroup = NULL)
p1saveWidget(p1, file = here("figs", "points_total_esc.html")) # Save the plotVisualisation 1 - Summary
Visualisation 1 shows that the points totals of the winning songs in the Eurovision Song Contest have increased exponentially over the years. Changes in the points totals have coincided with changes in the voting system, particularly in the early years of the competition. In the first few years, where fewer points were awarded to a smaller number of countries, the points totals for the winning songs were as low as 18 points in 1969, though another explanation for this low points total is that four countries won in that year. Changing the voting system in 1971 led to a rapid increase in the number of points given to the winning song, likely because each country awarded points to each of the songs. Between 1975 and 2003, the same voting system was used, which resulted in the points totals remaining consistent. From the 2004 competition, there is a slight increase in the point totals, which can be explained by the introduction of the semifinals leading to more countries voting in the final. Once the choice was made for each country to allocate two sets of points in 2016, the points totals of the winning songs increased again, leading to the highest points total that has been achieved in the Eurovision Song Contest, Portugal in 2017 with 758 points.
The aim of this project is partially achieved in visualisation 1. This interactive plot shows how changes in the voting system have affected the points totals of the winning songs. However, this plot does not isolate the effect of the voting system changes from the effect of the increasing number of panticipating countries on points allocation. Therefore, it is unclear whether the changes in the points totals of the winning songs are a result of the changes in the voting system. To accurately visualise how voting system has affected points allocation, a second visualisation was produced controlling for the number of participating countries.
Visualisation 2 - Controlling for the number of countries
I calculated the share of points that the winning songs received in each year by dividing the total number of points awarded in each year by the number of points allocated to the winning song. The aim was to isolate the effect of voting system and control for the number of participating countries in each year.
A new column was added to the data frame called “share_points”. To calculate the share of points, I needed to use an earlier version of the data frame that was not filtered by rank. As a result, the additional changes that were made to df4 and df5 had to be repeated here.
# Create a new column to show share of points
df6 <- df3 %>%
group_by(year) %>%
mutate(share_points = (points/(sum(points))*100))
# Change the share_point column to two decimal places
df6 $share_points <- round(df6 $ share_points, 2)
df7 <- df6 %>%
filter(rank == "1st") # filter data to only include winning songs
# collapsing the four rows for 1969 in the new dataframe
df7$country <- ifelse(df7$year == 1969,
paste(unique(df7$country[df7$year == 1969]), collapse = ", "),
df7$country)
df7$song <- ifelse(df7$year == 1969,
paste(unique(df7$song[df7$year == 1969]), collapse = ", "),
df7$song)
df7$participant <- ifelse(df7$year == 1969,
paste(unique(df7$participant[df7$year == 1969]), collapse = ", "),
df7$participant)
df8 <- df7 %>%
unique()Visualisation 2 - Additional label
A new subtitle will be needed for this visualisation:
subtitle2 <- "Share of points of the winning songs 1957-2023"Visualisation 2 - R code
A similar code is used to construct visualisation 2 as visualisation 1. The main difference is that share of points will be used for the y axis instead of points.
# constructing the plot using ggplot
# Year as the x axis and share of points on the Y axis
# Voting system used as colour and text added for labels
p2 <- ggplot(df8, mapping = aes(x = year,
y = share_points,
col = voting_system,
text = paste("<br>Year:",year,
"<br>Winning country:",country,
"<br>Artist:",participant,
"<br>Song:", song,
"<br>Share of Points:",share_points, "%")))
# Adding additional elements to the plot e.g., points, scale, title and legends
p2 <- p2 + geom_point(size = 2.5) +
scale_x_continuous(limits = c(1955, 2023),
breaks = seq(1955, 2023, 5)) +
labs(title = plottitle,
x = "Year",
y = "Share of points (%)",
colour = grouplab)
# Converting the ggplot into a ggplotly animation with hover labels
p2 <- ggplotly(p2, tooltip = c("text"), width = 700,
height = 600) %>% # Converting to a plotly graph with specified size
layout(margin = list(l = 30, r = 30, b=60, t = 60, pad = 4), # Changing the margins of the plot
title = list(text = paste0(plottitle,
'<br>',
'<sup>',
subtitle,'</sup>')), # Adding a subtitle to the plot
annotations = list(x = 1, y = -0.2,
text = caplab,
showarrow = FALSE,
xref = "paper",
yref = "paper",
xanchor="right", yanchor="auto",
xshift = 0, yshift=0,
font=list(size=13))) # Adding a annotation to the plot
p2 <- p2 %>% layout(legend = list(orientation = 'h',
xanchor = "center",
x = 0.5,
y =-0.2)) %>% # Changing the legend to be horizontal
style(legendgroup = NULL)
p2saveWidget(p2, file = here("figs", "points_share_esc.html")) # Save the plotVisualisation 2 - Summary
Visualisation 2 is more successful at demonstrating the impact of the voting system changes have had on the points received by the winning songs than visualisation 1.
In particular, visualisation 2 is effective at showing the impact that the voting system had on points allocation for the early years of the competition (1957-1974). The frequent changes in the voting system in the first 20 years of the competition led to large variability in the share of points achieved by the winning song. In 1964, Italy received the highest share of points in the Eurovision Song Contest so far with 34.03%. In contrast, Luxembourg received the lowest shared of points of any winning song in the contest in 1972 with 8.3%. The reason for this is that the voting system in 1972 meant that each country awarded more points to their favourite song and awarded more countries points, in comparison with the 1964 voting system.
Visualisation 2 also shows that the share of points has remained consistent between 1975 and 2022 because a similar voting system has been used since 1975. It is interesting that the share of points awarded to the winning song did not change from 2016, when the number of points awarded by each country was doubled. It would be interesting to see whether this continues for future competitions.
Future Direction
There was some additional information about the voting system that was not added to this plot. Different years of the Eurovision Song Contest have used a Jury voting system, a televote/public voting system or a mixture of the two. It was decided that this information would not be important for this visualisation. However, it would be interesting if a future project visualised differences in the points allocation for these different types of voting systems.
Another interesting way to investigate the Eurovision data would be to visualise the effect of running order position on the points totals of each song in the contest. There is an assumption among Eurovision fans that the 2nd position is the worst position to perform from in either the semi-finals or the grand-final. It would be interesting to investigate whether this is true using a data visualisation.
Finally, this Eurovision data could also be used to determine whether certain countries are more likely to give points to one another. Many people view the points allocation process in Eurovision to be politically motivated, where certain countries are more likely to award points to one another. There may be some truth in this, given that Cyprus and Greece are notorious for giving one another maximum points each year. It would be interesting to find a way to visualise this in the future.
References
The references provided are not permanent citations. The Eurovision website and Eurovision.World website did not contain a DOI that could be referenced in this project. These webpages may change or be removed in the future, which may impact the reproducibility of certain elements of this project such as the data scraping section.
Repo: https://github.com/paulgering/PSY6422_project Eurovision website: https://eurovision.tv Eurovision.World website: https://eurovisionworld.com