Data Clean Up and Analysis

-

Tools used: OpenRefine, Python, Python Libraries (Pandas and Matplotlib)

I decided to refer back to my data science class from freshman year and highschool and brush off in using python script and python libraries to clean, reorganize and render the data


Initial Goal: To find some trend on the frequency of heat.

Steps taken:

  • Uploaded the raw CSV file into OpenRefine to delete columns that restart the process and to appropriately split and align columns 
  • Used python script to read csv file
  • Deleted the rows of new process data 
  • Converted the heat binary to float to see the mean and the percentage of how often heat is turned on 
  • Got a percentage of the amount of time heater was on (~40%) minority of the time
  • Exported the cleaned csv data
  • Tried to plot the trends of heat with other variables (this is still a work in progress)
Reflection: 

I feel good about the output. I think with more studying and time, I could focus on a wider array of insights and provide a more accurate analysis. Unfortunately, when dealing with data this large, data processing tools are fundamental, and I’m afraid that being rusty might have affected the quality of the analysis. I was reminded of how interesting and frustrating data analysis is, but it was a great opportunity to sit down and write code for data after such a long time.

The biggest struggle I had was the amount of data, and the cleaning process took a long time. I couldn’t do it through Python alone, so I had to get my hands dirty and edit using OpenRefine. One piece of feedback I have is that if we had totals and averages of the variables, it would have been easier to catch trends. If the data collector had an automated process that added up sums and averages of the collected data, it would have helped immensely.

(To be Edited: I will embed a source code of my process once I figure out the problems with my github account)

Comments

Post a Comment

Popular posts from this blog

Bad Data

-
Image
Data whose originator had good intentions: Eugenics: Eugenics is a set of beliefs and practices aimed at improving the genetic quality of a human population, often through selective breeding or sterilization. The concept has its origins in the late 19th and early 20th centuries, drawing on advancements in genetics and evolutionary theory. Okay labeling this as "good intentions" is a stretch but for the sake of the argument given by the leaders of the eugenics movement, let us take their word for it and assume this was an attempt at a scientific breakthrough Reflection: Eugenics is an immoral and pseudoscientific theory that claims it is possible to perfect people and groups through genetics and the scientific laws of inheritance. Eugenicists used an incorrect and prejudiced understanding of the work of Charles Darwin and Gregor Mendel to support the idea of “racial improvement.” In their quest for a perfect society, eugenicists labelled many people as “unfit,” including eth...
Read more

Digital Humanities: Week 1 Reflection

-
The overall goal set for this class states: “ …By the end of this class, each student should be able to engage in their craft in new and digital ways…” It is a broad and bold goal, open to multiple interpretations. The interdisciplinary nature of this course allows it to flow and interact with various facets of life and innovation. However, one word stands out in this objective: craft. Merriam-Webster defines craft as “to make or produce with care, skill, or ingenuity.” In a broader interpretation, crafting involves planning, plotting, intentionality, and demonstrating cleverness and aptitude in how we approach things. The main buzzword people often associate with crafting is creativity. So naturally, the next important question is: what does it mean to merge crafting with the digital world? The same dictionary defines digital as “something characterized by electronic and especially computerized technology.” When defining digital, terms like electronics, computation, data, and patter...
Read more

Good Data

-
Image
Good Data:  Data whose originator had good intentions:  The FEWS Net (Ethiopia): FEWS NET monitors and provides early warning analysis of ongoing, imminent, or emerging threats to food security and maintains the flexibility to analyze unanticipated crises around the world. Between 1983 and 1985 there was a heavy drought and famine in Ethiopia particularly in the northern region.  FEWS NET has significantly helped Ethiopia by providing early warnings and detailed food security analyses, enabling timely responses to famine threats. By monitoring factors like rainfall, crop yields, market prices, and nutrition data, it predicts food shortages and identifies at-risk populations. These forecasts have allowed the Ethiopian government and humanitarian organizations to prepare in advance, mobilize resources, and implement targeted interventions, such as food distribution and cash transfers. The FEWS NET is data being used for a good purpose that has a positive life saving impac...
Read more