Importing Data

Ha Khanh Nguyen (hknguyen)

• 1. Importing Modules/Libraries
  • 1.1 What is pandas?
  • 1.2 What is a csv file?
• 2. Importing Data Using pandas
  • 2.1 Importing using a URL
  • 2.2 Importing a local file
  • Exercise
• 3. DataFrame
• 4. Computing Summary Statistics

---

1. Importing Modules/Libraries

• From now on, we will be using data structures and functions provided in a module/library.
• These data structures and functions are NOT built-in to Python.
  • Strings, list, etc. are data structures that are built-in to Python.
  • pandas DataFrame (will be covered later) is NOT a built-in data structure.
  • len() is a built-in function returning the length of a string or a list, etc.
  • read_csv() is NOT built-in and can only be called after importing Pandas library.
• You only need to import the module once every time you start/open a Jupyter Notebook.

# import a library/module
import pandas

# call read_csv() function
pandas.read_csv('https://stat107.hknguyen.org/files/datasets/rivers.csv')

	length
0	735
1	320
2	325
3	392
4	524
...	...
136	720
137	270
138	430
139	671
140	1770

141 rows × 1 columns

# import pandas but now call it pd
import pandas as pd

# call read_csv() function
pd.read_csv('https://stat107.hknguyen.org/files/datasets/rivers.csv')

	length
0	735
1	320
2	325
3	392
4	524
...	...
136	720
137	270
138	430
139	671
140	1770

141 rows × 1 columns

• We can also just import the function we need from a module.
  • Above, we were importing the WHOLE module.

# not recommended for pandas!
from pandas import read_csv

read_csv('https://stat107.hknguyen.org/files/datasets/rivers.csv')

	length
0	735
1	320
2	325
3	392
4	524
...	...
136	720
137	270
138	430
139	671
140	1770

141 rows × 1 columns

1.1 What is pandas?

"pandas is a fast, powerful, flexible and easy to use open source data analysis and manipulation tool, built on top of the Python programming language." -pandas.pydata.org

• pandas provides the data structures and functions needed to conduct clean data and conduct data analysis.

1.2 What is a csv file?

• csv stands for comma separated value. This means that the values of a row are comma separated.
• We will be dealing with mainly only csv files in this class.

---

2. Importing Data Using pandas

• pandas features a number of functions for reading tabular data (data in table format) as a DataFrame object.

Function	Description
read_csv	Load delimited data from a file, URL, or file-like object; use comma as default delimiter
read_table	Load delimited data from a file, URL, or file-like object; use tab ('\t') as default delimiter
read_excel	Read tabular data from an Excel XLS or XLSX file
read_json	Read data from a JSON (JavaScript Object Notation) string representation
read_sql	Read the results of a SQL query (using SQLAlchemy) as a pandas DataFrame

• For our class, we will mainly be using read_csv(), but we never know, we might practice with read_json() and read_excel() at the end of the course.

2.1 Importing using a URL

• Dataset is sometimes provided as an URL. For example, the rivers.csv we used in part 1 was loaded to Python using a URL.

rivers = pd.read_csv('https://stat107.hknguyen.org/files/datasets/rivers.csv')

• The rivers variable is now referencing an object of type DataFrame (people also say pandas DataFrame). What is a DataFrame? Well, it's a table with named columns!
  • In this example, rivers only has 1 column which is named length.

coffee_sleep = pd.read_csv('https://stat107.hknguyen.org/files/datasets/coffee-sleep.csv')
coffee_sleep

	coffee	sleep	level
0	9.2	6.2	g
1	6.4	8.1	u
2	9.0	5.8	u
3	6.7	10.8	g
4	4.7	8.8	u
...	...	...	...
95	0.0	9.0	u
96	0.0	9.4	u
97	0.0	10.7	g
98	0.0	7.6	u
99	0.0	10.7	u

100 rows × 3 columns

2.2 Importing a local file

• Now, what's if you want to load a local dataset (a file that is locally available on your computer)?
• First, you need to know where it is? Is it in Downloads folder? Is it in Documents?
• For our class, you will want the dataset to be in the same folder as your Jupyter Notebook.

pd.read_csv("path to the file")

• If the file is in the SAME folder as your Jupyter Notebook:

pd.read_csv("filename")

Exercise

• Download coffee-sleep.csv.
• Then move it to your NetID folder on Desktop.
• Open a command prompt, navigate to your NetID folder.
• Then type jupyter notebook and press Enter.
• Load the dataset to Python.

---

3. DataFrame

coffee_sleep

	coffee	sleep	level
0	9.2	6.2	g
1	6.4	8.1	u
2	9.0	5.8	u
3	6.7	10.8	g
4	4.7	8.8	u
...	...	...	...
95	0.0	9.0	u
96	0.0	9.4	u
97	0.0	10.7	g
98	0.0	7.6	u
99	0.0	10.7	u

100 rows × 3 columns

• There are 3 columns in this DataFrame:
  • 'coffee': daily average coffee consumption of a student in STAT 430 (in oz)
  • 'sleep': daily average number of hours of sleep (in hours)
  • 'level': whether the student is a graduate ('g') or undergradte ('u') student

• To select a single column:

coffee_sleep['coffee']

0     9.2
1     6.4
2     9.0
3     6.7
4     4.7
     ... 
95    0.0
96    0.0
97    0.0
98    0.0
99    0.0
Name: coffee, Length: 100, dtype: float64

---

4. Computing Summary Statistics

• To compute summary statistics of a variable in a DataFrame, you first need to select the column, then select the needed function!
• The details and examples of available functions are here.

# compute mean
coffee_sleep['sleep'].mean()

7.942

# compute median
coffee_sleep['sleep'].median()

8.0

# get all summary statistics
coffee_sleep['sleep'].describe()

count    100.000000
mean       7.942000
std        1.766648
min        3.300000
25%        6.675000
50%        8.000000
75%        9.325000
max       11.500000
Name: sleep, dtype: float64

# compute median
coffee_sleep['level'].value_counts()

u    59
g    41
Name: level, dtype: int64

• There are many different ways to do the same thing in Python. You will soon find that there are other ways to calculate the basic statistics!