Teilaufgaben

Untersuchen Sie die Daten auf offensichtliche Fehler. Wenn Fehler in den Daten vorhanden sind, wie können diese bseitigt werden?

• Manchmal “N/A” wird as -1 räpresentiert
• Das ist kein Fehler, aber die Gewichte (‘weight’) von die Packungen können verschieden sein.

Geben Sie drei “Einsichten”, Erkenntnisse oder auch weitergehende Fragen an, die Sie bei der Untersuchung der Daten erhalten haben. Mit “Einsicht” ist dabei ein Verständnis der Daten und ihrer Eigenschaften sein, das nicht relativ offensichtlich oder intuitiv ist.

1. Es gibt eine SEHR verschiedene Anzahl von Cereals von verschiedene Hersteller

• General Mills und Keloggs - the most (22, 23)
• American Home Food Products - the least (1)

2. People (say they) care about their health: ratings negatively correlate with sugar and calories

Stereotypically ‘bad’ things like sugar and calories are the strongest factors in making ratings decrease. Fiber, on the other hand, makes ratings increase.

Correlation with rating: - more sugar -0.8 - calories -0.6 - fiber +0.6

BUT:

• calories VS fiber: -0.7 - might be confounding the ratings above!
• Maybe no one cares about fiber, but there’s some food technology reason why more fiber means fewer calories.
• E.g.: Everyone (says they hate) calories, and fiber is negatively corelated with calories, giving the illusion.

3. If we remove two outliers, not much changes

• “Nabisco” (highest mean rating and lowest mean calories) and “All Bran Extra Fiber” (all-time highest-rating and highest-fiber cereal) are outliers and might be special.

• Maybe “All Bran Extra Fiber” is just a REALLY good cereal

  • It’s an outlier both inside Keloggs cereals and overall

• Maybe Nabisco (since it’s never on the third/highest shelf) is too a really good cereal by itself, not because few fibers

• Just for fun, we remove both Nabisco-brand cereals and the “All Bran Extra Fiber” one.

• 

• Now, calories (-0.5) still matter less than before and much sugars (-0.8)

• But overall no dramatic differences

Beschreiben Sie den Prozess, mit dem Sie bei der explorativen Analyse der Daten vorgegangen sind.

• Did everything in a Jupyter notebook
• Did a DataFrame.describe(), found the minimum -1 values. Looked into them and replaced them with np.nans so that semantically they make sense.
• First I looked at the definitions of the columns, found special ones that don’t really mean numbers (e.g. shelf is an ordinal variable)
• Found that weight is different, and selected the subset of columns that makes sense to divide (Verhältnissskala ones) to get comparable per-ounce values
• Then I started looking for correlations etc, using a function to highlight the most interesting ones
• Then I wanted to look for outliers in general and started doing PCA-like things
  • But then I got distracted and started doing 3D plots of them and learning to plot in seaborn etc., but the main insights were found by that point

Beschreiben Sie Herausforderungen oder Probleme, auf die Sie bei der Analyse gestoßen sind. Wurden Sie durch irgendetwas eingeschränkt? Wenn Sie auf keine Probleme oder Herausforderungen getroffen sind, denken Sie an potenzielle Herausforderungen, die jemand haben könnte.

• Visually parsing correlation tables was hard for me, so I wrote a helper function to highlight the important bits (seen on the screenshots)
  • Before:
  • After:
• One could have missed the -1s or especially the different cereal weights!
• Not being able to do explorative plots was very unusual for me and I felt blind; hypothetically the same info could be gotten from tables with the quantiles etc., but again this is something I can parse MUCH better with a box plot.