Estevão Vilela

Estevão Vilela

Working with school violence in Minas Gerais, Brazil, and studying computational mathematics at UFMG

Exploring Covid 19 data - Part 1

3 minute read

We are facing a hurtful pandemic at a global scale, and in Brazil it’s not different. The numbers of death and confirmed cases grow each day and at the same time we see opinions - from the government - trying to minimize the problem.

In this tweet from Professor Bernardo Queiroz, he states the lack of detailed data provided by the Ministry of Health of Brazil in order to calculate mortality ratios. This institution only provides aggregate data in a special website dedicated to Covid-19.

Other source of data is the administrative records of deaths, through death certificates. This data is provided by Brazilian Civil Registry Offices at ‘Portal da Transparência - Registro Civil’. This data is desaggregated by gender, age groups and brazilian states. So, we decided to collect it.

We need to point out the differences between the date:

  1. As pointed out by Brazilian Civil Registry Offices, data about more recent dates could suffer some changes due to entry of the new records - the procedure in order to get the data into the system could take some days - and corrections made in those already recorded, although this is a rare case (1);
  2. In the administrative records, Covid-19 represents a cause of death or a suspected cause of death. This happens, because the Ministry of Health and the National Justice Council stated that burials and cremation can be realized without the death certificates (2). So, when there is a death by respiratory cause suspected to be caused by Covid-19 and not confirmed, it must be written in the death certificate as a “suspect” or “likely” death by Covid-19 (3).

Webscraping

So, how to collect the data if the website doesn’t provide it directly via a .csv download, for exemple? We are stating here only examples of programmatic ways of getting data, because you can always copy and paste, manually.

During the Summer School hosted by the Computer Science Department at UFMG, Bernardo Abreu and Vinícius Silva - from the company ‘Big Data’ - lectured a course on ‘Collecting data from WEB’. They talked about getting data from an API and by ‘Web scraping’.

The presentation has been shared and so the code - using an API or Scraping HTML.

The code and the examples shared by them helped to get what we needed to do. The data that we want to collect it’s in this graph:

Portal da Transparência

A web page is constructed using the HTML language (and other programming languages, like CSS and JavaScript). Using the Developer tools in a web browser (Ctrl+Shift+I, in Google Chrome) we can look of what a web page looks like ‘behind the scenes’, in a toy example:

Quotes to scrap

Quotes to scrap - Devtools

What we have to know is that a web page is constructed by blocks and in the language the blocks have an hierarchy structure that helps us to find the information we want to collect. So, in the example above, to get a quote from Einstein we need to get the content of the tag ‘span’ located after several ‘div’ (another type of tag).

In this post, we won’t go into the details of HTML or XPath, a language we can use to parse elements in XML document, and we do this because the website we’re looking at, the data it’s not in the HTML page, as you can see in the image below, so we can’t scrap it right away:

Portal da Transparência - Devtools

Another way of getting the data it’s look at the ‘browser console’ and inspect the requests made by the web page in the ‘Network’ section (once you go there, you need to refresh the page). There are several request made, but we can search for the one we are interested in: deaths of Covid-19 grouped by gender. Double-clicking in we see that the data we want it’s in a JSON format with the following link. We can see that the we need to iterate through dates and states to get daily data disaggregated by this attributes.

We’ll see in the next post how he got the data using R and created a DataViz using ggplot2. Feel free to leave a comment below, this is my first data science post so your feedback is more than welcome!

  1. https://agenciabrasil.ebc.com.br/saude/noticia/2020-04/plataforma-dos-cartorios-reune-informacoes-de-mortes-por-covid-19 ↩

  2. https://noticias.uol.com.br/saude/ultimas-noticias/redacao/2020/04/09/covid-19-declaracoes-de-obito-apontam-48-mais-mortes-do-que-dado-oficial.htm ↩

  3. https://www.conjur.com.br/2020-mar-31/portaria-permite-sepultamento-cremacao-certidao-obito ↩

You may also enjoy