Mastering Data Analysis with Pandas: Surviving the Titanic Dataset

Author: Piyush | Topic: Python, Data Science, Pandas

Have you ever looked at a messy spreadsheet and thought, "There has to be a better way to make sense of this?" Well, there is, and its name is Pandas 🐼.

In this hands-on guide, we're going to dive deep into the world of Data Analysis using Python. Instead of boring, abstract examples, we'll be exploring the legendary Titanic Dataset. We'll learn how to load data, uncover hidden trends, handle missing values, and answer the ultimate question: What kinds of people were most likely to survive?

Buckle up! It's time to get our hands dirty with some code. 💻

🔍 Phase 1: Understanding Our Dataset

Before we can analyze anything, we need to know what we are dealing with. Let's load up our data and take a peek under the hood!

import pandas as pd # Here, pd is a alias used to call pandas library.

df = pd.read_csv("C:/Users/hp/Downloads/Titanic-Dataset.csv");

Head is used to display the first rows of dataframe.

df.head(5)

👀 Click to view output

   PassengerId  Survived  Pclass  \
0            1         0       3   
1            2         1       1   
2            3         1       3   
3            4         1       1   
4            5         0       3   

                                                Name     Sex   Age  SibSp  \
... [Data Truncated for Readability]

Tail is used to display the last rows of a DataFrame.

df.tail(5)

👀 Click to view output

     PassengerId  Survived  Pclass                                      Name  \
886          887         0       2                     Montvila, Rev. Juozas   
887          888         1       1              Graham, Miss. Margaret Edith   
888          889         0       3  Johnston, Miss. Catherine Helen "Carrie"   
889          890         1       1                     Behr, Mr. Karl Howell   
890          891         0       3                       Dooley, Mr. Patrick   

        Sex   Age  SibSp  Parch      Ticket   Fare Cabin Embarked  
... [Data Truncated for Readability]

It is an attribute which gives the dimensions of the DataFrame. Here, the output is given in the form of tuple.

df.shape

👀 Click to view output

(891, 12)

Here, it is also an attribute representing the names of columns in the tuple form. OR df.columns.tolist() # Here, it will convert those columns names into Python list.

df.columns

👀 Click to view output

Index(['PassengerId', 'Survived', 'Pclass', 'Name', 'Sex', 'Age', 'SibSp',
       'Parch', 'Ticket', 'Fare', 'Cabin', 'Embarked'],
      dtype='object')

It is used to tell the data type of each column in the form of pandas series(One-dimensional array capable of holding any data type).

df.dtypes

👀 Click to view output

PassengerId      int64
Survived         int64
Pclass           int64
Name            object
Sex             object
Age            float64
SibSp            int64
Parch            int64
... [Data Truncated for Readability]

It is used to display the compact overview of the whole dataframe. Remember: If the non-null count is less than total no.of rows, the column contain missing values.

df.info()

👀 Click to view output


RangeIndex: 891 entries, 0 to 890
Data columns (total 12 columns):
 #   Column       Non-Null Count  Dtype  
---  ------       --------------  -----  
 0   PassengerId  891 non-null    int64  
 1   Survived     891 non-null    int64  
 2   Pclass       891 non-null    int64  
... [Data Truncated for Readability]

Generate descriptive statistics for numerical columns. Where 50% represent the median. statistics -> count, mean, std, min.

df.describe()

👀 Click to view output

       PassengerId    Survived      Pclass         Age       SibSp  \
count   891.000000  891.000000  891.000000  714.000000  891.000000   
mean    446.000000    0.383838    2.308642   29.699118    0.523008   
std     257.353842    0.486592    0.836071   14.526497    1.102743   
min       1.000000    0.000000    1.000000    0.420000    0.000000   
25%     223.500000    0.000000    2.000000   20.125000    0.000000   
50%     446.000000    0.000000    3.000000   28.000000    0.000000   
75%     668.500000    1.000000    3.000000   38.000000    1.000000   
... [Data Truncated for Readability]

To display both the numerical and categorical columns we use include="all". Statistics -> count, unique, top, freq

df.describe(include="all")

👀 Click to view output

        PassengerId    Survived      Pclass                 Name   Sex  \
count    891.000000  891.000000  891.000000                  891   891   
unique          NaN         NaN         NaN                  891     2   
top             NaN         NaN         NaN  Dooley, Mr. Patrick  male   
freq            NaN         NaN         NaN                    1   577   
mean     446.000000    0.383838    2.308642                  NaN   NaN   
std      257.353842    0.486592    0.836071                  NaN   NaN   
min        1.000000    0.000000    1.000000                  NaN   NaN   
... [Data Truncated for Readability]

🎯 Phase 2: Selecting and Filtering Data

We rarely need an entire dataset at once. Usually, we want to ask specific questions, like finding passengers above a certain age, or filtering by gender. Let's see how Pandas makes this incredibly easy.

df.Age For selecting a single column data. This method is generally preferred because it is more reliable with column names containing spaces or special characters. Syntax -> df["column_name"]

df["Age"]

👀 Click to view output

0      22.0
1      38.0
2      26.0
3      35.0
4      35.0
       ... 
886    27.0
887    19.0
... [Data Truncated for Readability]

Syntax -> df[["column1", "column2", "column3"]] Remember, here outer bracket is used to select DataFrame and inner bracket is used to select columnes.

df[["Survived", "Pclass", "Sex", "Age", "Fare"]]

👀 Click to view output

     Survived  Pclass     Sex   Age     Fare
0           0       3    male  22.0   7.2500
1           1       1  female  38.0  71.2833
2           1       3  female  26.0   7.9250
3           1       1  female  35.0  53.1000
4           0       3    male  35.0   8.0500
..        ...     ...     ...   ...      ...
886         0       2    male  27.0  13.0000
... [Data Truncated for Readability]

iloc-> It is used to select data using integer positions/ location. Syntax -> df.iloc[row_positions, column_positions] df.iloc[0] # Shows first row. df.iloc[:5] # or [0:5] First 5 rows. df.iloc[:, 0:4] # First 4 columns.

df.iloc[0:5,0:4]

👀 Click to view output

   PassengerId  Survived  Pclass  \
0            1         0       3   
1            2         1       1   
2            3         1       3   
3            4         1       1   
4            5         0       3   

                                                Name  
... [Data Truncated for Readability]

Here, we are using boolean filtering to select rows based on a condition. Syntax -> df[df["column"] > value] df["Age"] > 30 It will display true/false values which satisfy this condition. It will match rows according to the given condition.

df[df["Age"] > 30]

👀 Click to view output

     PassengerId  Survived  Pclass  \
1              2         1       1   
3              4         1       1   
4              5         0       3   
6              7         0       1   
11            12         1       1   
..           ...       ...     ...   
873          874         0       3   
... [Data Truncated for Readability]

Now, we are performing boolean filtering with categorical/text columns. Syntax -> df[df["column"] == "value"]

df[df["Sex"] == "female"]

👀 Click to view output

     PassengerId  Survived  Pclass  \
1              2         1       1   
2              3         1       3   
3              4         1       1   
8              9         1       3   
9             10         1       2   
..           ...       ...     ...   
880          881         1       2   
... [Data Truncated for Readability]

Now, we will combine multiple conditions using &(and), |(or) operators. Syntax -> df[df[(condition1) & (condition2)]]

df[(df["Sex"] == "female") & (df["Age"] > 30)]

👀 Click to view output

     PassengerId  Survived  Pclass  \
1              2         1       1   
3              4         1       1   
11            12         1       1   
15            16         1       2   
18            19         0       3   
..           ...       ...     ...   
862          863         1       1   
... [Data Truncated for Readability]

df[(df["Pclass"] == 1) | (df["Pclass"] == 2)] without using isin() isin() function is used to check whether values belong to a set/list of specific values. Syntax -> df[df["column"].isin([value1, value2, ...])] using isin()

df[df["Pclass"].isin([1,2])]

👀 Click to view output

     PassengerId  Survived  Pclass  \
1              2         1       1   
3              4         1       1   
6              7         0       1   
9             10         1       2   
11            12         1       1   
..           ...       ...     ...   
880          881         1       2   
... [Data Truncated for Readability]

Between() is used to check whether values fall between a specific range. Syntax -> df[df["column"].between(lower, upper)]

df[df["Age"].between(20, 30)]

👀 Click to view output

     PassengerId  Survived  Pclass  \
0              1         0       3   
2              3         1       3   
8              9         1       3   
12            13         0       3   
23            24         1       1   
..           ...       ...     ...   
882          883         0       3   
... [Data Truncated for Readability]

query() help us to filter DataFrame using a string expression. df.query("condition");

df.query("Age > 40")

👀 Click to view output

     PassengerId  Survived  Pclass  \
6              7         0       1   
11            12         1       1   
15            16         1       2   
33            34         0       2   
35            36         0       1   
..           ...       ...     ...   
862          863         1       1   
... [Data Truncated for Readability]

🧹 Phase 3: Cleaning the Mess (Missing Values & Duplicates)

Real-world data is almost never clean. It has missing values, duplicates, and errors. A good data scientist knows how to clean it up before doing any serious analysis.

To check the missing values (NaN) we use isna() [R] and isnull() [Numpy] function. df.isna() function gives the output in the form of true or false. Here, we are counting the missing values.

df.isna().sum()

👀 Click to view output

PassengerId      0
Survived         0
Pclass           0
Name             0
Sex              0
Age            177
SibSp            0
Parch            0
... [Data Truncated for Readability]

Here, we will perform the sum() function again to get the total missing values in a whole dataset.

df.isna().sum().sum()

👀 Click to view output

np.int64(689)

Now, we are going to calculate the percentage of missing values Or Alternative df.isna().mean() * 100 because mean of boolean values treats [True-> 1, False-> 0]. So, it gives the proportion of missing values.

df.isna().sum() / len(df) * 100

👀 Click to view output

PassengerId     0.000000
Survived        0.000000
Pclass          0.000000
Name            0.000000
Sex             0.000000
Age            19.865320
SibSp           0.000000
Parch           0.000000
... [Data Truncated for Readability]

We will use fillna() function to replace missing values with the desired inputs. Syntax-> df["column"].fillna(value) or df["Age"] = df["Age"].fillna(df["Ag"].median()) Remember, fillna() will modify the series/dataframe but to change the original data we have to use inplace=True Here, it is permanently updating the values.

median_age = df["Age"].median()
df["Age"] = df["Age"].fillna(median_age)

Now, we will drop the rows containing missing values using dropna() Remember, this creates a new DataFrame by default and does not modify the original df. It will remove rows containing missing values and to remove columns use dropna(axis=1) df_clean.shape Used to check the dimensions (rows, columns)

df_clean = df.dropna()
df_clean

👀 Click to view output

     PassengerId  Survived  Pclass  \
1              2         1       1   
3              4         1       1   
6              7         0       1   
10            11         1       3   
11            12         1       1   
..           ...       ...     ...   
871          872         1       1   
... [Data Truncated for Readability]

We use duplicated() function which gives boolean values. It will give the number of duplicate records.

df.duplicated().sum()

👀 Click to view output

np.int64(0)

To drop the duplicate rows we may use the function drop_duplicates() df.drop_duplicates().shape

df_no_duplicates = df.drop_duplicates()
df_no_duplicates.shape

👀 Click to view output

(891, 12)

📊 Phase 4: Sorting and Frequency Analysis

Who paid the highest fare? What was the most common age group? Let's sort our data and find out the frequencies of different categories.

unique() is used to return all distinct values present in a column. Syntax-> df["column"].unique()

df["Sex"].unique()

👀 Click to view output

array(['male', 'female'], dtype=object)

Here, we are using the value_counts() to count how many time each unique value has been repeated. Syntax-> df["column"].value_counts()

df["Sex"].value_counts()

👀 Click to view output

Sex
male      577
female    314
Name: count, dtype: int64

Here, we are calculating the percentage of male and female passengers. df["Sex"].value_counts(normalize=True) # It will return the output in the form of proportion for every unique value [Means dividing the count with the total no. of items]

df["Sex"].value_counts(normalize=True) * 100

👀 Click to view output

Sex
male      64.758698
female    35.241302
Name: proportion, dtype: float64

Now, we are sorting according to age in ascending order. Syntax-> df.sort_values("column") by default ascending = True. Or df.sort_values("Age", ascending = True)

df.sort_values("Age")

👀 Click to view output

     PassengerId  Survived  Pclass                                  Name  \
803          804         1       3       Thomas, Master. Assad Alexander   
755          756         1       2             Hamalainen, Master. Viljo   
469          470         1       3         Baclini, Miss. Helene Barbara   
644          645         1       3                Baclini, Miss. Eugenie   
78            79         1       2         Caldwell, Master. Alden Gates   
..           ...       ...     ...                                   ...   
116          117         0       3                  Connors, Mr. Patrick   
... [Data Truncated for Readability]

Now, we are sorting fare price in descending order. Syntax-> df.sort_values("column", ascending=False)

df.sort_values("Fare", ascending=False)

👀 Click to view output

     PassengerId  Survived  Pclass                                Name  \
679          680         1       1  Cardeza, Mr. Thomas Drake Martinez   
258          259         1       1                    Ward, Miss. Anna   
737          738         1       1              Lesurer, Mr. Gustave J   
88            89         1       1          Fortune, Miss. Mabel Helen   
438          439         0       1                   Fortune, Mr. Mark   
..           ...       ...     ...                                 ...   
806          807         0       1              Andrews, Mr. Thomas Jr   
... [Data Truncated for Readability]

Now, we are calculating 5 highest fares. Syntax-> df.sort_values("column", ascending = False).head(n) or can use df.nlargest(5, "Fare") [nlargest()-> sort_values() + head()]

df.sort_values("Fare", ascending=False).head(5)

👀 Click to view output

     PassengerId  Survived  Pclass                                Name  \
679          680         1       1  Cardeza, Mr. Thomas Drake Martinez   
258          259         1       1                    Ward, Miss. Anna   
737          738         1       1              Lesurer, Mr. Gustave J   
88            89         1       1          Fortune, Miss. Mabel Helen   
438          439         0       1                   Fortune, Mr. Mark   

        Sex   Age  SibSp  Parch    Ticket      Fare        Cabin Embarked  
... [Data Truncated for Readability]

Here, we are displaying the 5 youngest passengers, or use df.nsmallest(5, "Age")

df.sort_values("Age").head(5)

👀 Click to view output

     PassengerId  Survived  Pclass                             Name     Sex  \
803          804         1       3  Thomas, Master. Assad Alexander    male   
755          756         1       2        Hamalainen, Master. Viljo    male   
469          470         1       3    Baclini, Miss. Helene Barbara  female   
644          645         1       3           Baclini, Miss. Eugenie  female   
78            79         1       2    Caldwell, Master. Alden Gates    male   

      Age  SibSp  Parch  Ticket     Fare Cabin Embarked  
... [Data Truncated for Readability]

🧮 Phase 5: Aggregate Functions (The Big Picture)

Finally, let's crunch some numbers. Averages, minimums, maximums, and totals. This is where we extract high-level insights from our dataset.

Here, we are calculating the average age. Syntax-> df["column"].mean()

df["Age"].mean()

👀 Click to view output

np.float64(29.36158249158249)

Now, we are calculating the median age. Here, median is the middle value when the data is arranged in order. It useful when data contains outliers.

df["Age"].median()

👀 Click to view output

np.float64(28.0)

Now, we calculating min and max passanger age. Or df["Age"].agg(["min", "max"]) we can aggregate both the operations.

df["Age"].min()
df["Age"].max()

👀 Click to view output

np.float64(80.0)

Now are calculating the total fare payed by each passanger. Adding all column values.

df["Fare"].sum()

👀 Click to view output

np.float64(28693.9493)

df["Fare"].mean()

👀 Click to view output

np.float64(32.204207968574636)

Standard deviation-> Tell how spread out the values are around the mean. Moreover, larger std-> greater var. variance = square root of std. or df["Age"].agg(["std", "var"])

df["Age"].std()
df["Age"].var()

👀 Click to view output

np.float64(169.51249827942345)

Mode-> It is the value that occurs most frequently and its output is given in series (1D labelled array) unlike mean and median.

df["Age"].mode()

👀 Click to view output

0    28.0
Name: Age, dtype: float64

Aggregate function-> It allows multiple functions to the same column at once. Using agg() on single column.

df["Age"].agg(["count", "mean", "median", "min", "max", "std"])

👀 Click to view output

count     891.000000
mean       29.361582
median     28.000000
min         0.420000
max        80.000000
std        13.019697
Name: Age, dtype: float64

Now, we are using agg() on multiple columns. Syntax -> df.agg({ "column1": ["function1", "function2"], "column2": ["function3", "function4"] })

df.agg({
    "Age": ["mean", "median", "min", "max"],
    "Fare": ["mean", "sum", "min", "max"] 
})

👀 Click to view output

              Age          Fare
mean    29.361582     32.204208
median  28.000000           NaN
min      0.420000      0.000000
max     80.000000    512.329200
sum           NaN  28693.949300

GroupBy Operations

groupby() divide the DataFrame into groups based on a column. It helps to ans questions such as "What is the average age for each passenger clas?" Now, we are finding the average age of male and female passengers using groupby()

df.groupby("Sex")["Age"].mean()

👀 Click to view output

Sex
female    27.929936
male      30.140676
Name: Age, dtype: float64

Now, we are calculating the survived rate for male and female passengers.

df.groupby("Sex")["Survived"].mean() * 100

👀 Click to view output

Sex
female    74.203822
male      18.890815
Name: Survived, dtype: float64

Now, we are calculating the number of passengers in each passenger class. size() counts the number of rows in each group. class_counts = df['Pclass'].value_counts() print(class_counts)

df.groupby("Pclass").size()

👀 Click to view output

Pclass
1    216
2    184
3    491
dtype: int64

Here, we are calculating the average fare for each passenger class. Syntax-> df.groupby("group_column")["value_column"].mean() Remember, we can identify the group_column and the value_column based on the sentence structure.

df.groupby("Pclass")["Fare"].mean()

👀 Click to view output

Pclass
1    84.154687
2    20.662183
3    13.675550
Name: Fare, dtype: float64

Now, we are calculating the average age for each passenger class.

df.groupby("Pclass")["Age"].mean()

👀 Click to view output

Pclass
1    36.812130
2    29.765380
3    25.932627
Name: Age, dtype: float64

Now, we are calculating the survival rate for each passenger class.

df.groupby("Pclass")["Survived"].mean() * 100

👀 Click to view output

Pclass
1    62.962963
2    47.282609
3    24.236253
Name: Survived, dtype: float64

Now, we are calculating the no. of survivors in each passenger class. Remember, the difference between count() and size(). count() -> Only counts rows that contain valid data, meaning avoiding missing values. size() -> It counts every single row in a dataset. Here, we can also use sum() because survived -> 1 and unsurvived -> 0.

df.groupby("Pclass")["Survived"].size()

👀 Click to view output

Pclass
1    216
2    184
3    491
Name: Survived, dtype: int64

Now, we have to group the passengers by both Pclass and Sex to calculate their survival rate. Now, we have to group two columns.

df.groupby(["Pclass", "Sex"])["Survived"].mean() * 100

👀 Click to view output

Pclass  Sex   
1       female    96.808511
        male      36.885246
2       female    92.105263
        male      15.740741
3       female    50.000000
        male      13.544669
Name: Survived, dtype: float64

Now, we are calculating mean, minimum and maximum age for each passenger class. Here, we are grouping using multiple statistics.

df.groupby("Pclass")["Age"].agg({
    "mean",
    "min",
    "max"
})

👀 Click to view output

         min       mean   max
Pclass                       
1       0.92  36.812130  80.0
2       0.67  29.765380  70.0
3       0.42  25.932627  74.0

Now, we are calculating the mean, median and maximum fare for each passenger class.

df.groupby("Pclass")["Age"].agg({
    "mean",
    "median",
    "max"
})

👀 Click to view output

             mean   max  median
Pclass                         
1       36.812130  80.0    35.0
2       29.765380  70.0    28.0
3       25.932627  74.0    28.0

Part G-> Creating and Transforming columns

transform() is used when you want a group-level calculation but want the result returned for every original row. Syntax-> df["new_column"] = df.groupby("group_column")["value_column"].transform("mean")

df["Class_Mean_Age"] = df.groupby("Pclass")["Age"].transform("mean")
df[["Pclass", "Age", "Class_Mean_Age"]].head()

👀 Click to view output

   Pclass   Age  Class_Mean_Age
0       3  22.0       25.932627
1       1  38.0       36.812130
2       3  26.0       25.932627
3       1  35.0       36.812130
4       3  35.0       25.932627

Now, we have to create a new age column with categories such as child, teenager, young adult, adult and senior. Here, we can use pd.cut() to divide the Age values into meaningful categories. Syntax-> df["Age_Group"] = pd.cut( df["Age"], bin=[...], # It is used to define the boundaries. labels=[...] # It is used to define the category names. )

df["Age_Group"] = pd.cut(
    df["Age"],
    bins=[0, 12, 19, 29, 50, float("inf")],
    labels=["Child", "Teenager", "Young Adult", "Adult", "Senior"]
)
df[["Age", "Age_Group"]].head(10)

👀 Click to view output

    Age    Age_Group
0  22.0  Young Adult
1  38.0        Adult
2  26.0  Young Adult
3  35.0        Adult
4  35.0        Adult
5  28.0  Young Adult
6  54.0       Senior
... [Data Truncated for Readability]

Now, we have to count the number of passengers in each age group. Here, we have used value_count() to check how many passengers fall into each category.

df["Age_Group"].value_counts().sort_index()

👀 Click to view output

Age_Group
Child           69
Teenager        95
Young Adult    397
Adult          266
Senior          64
Name: count, dtype: int64

Now, we are going to use apply(), classify passengers as Minor or Adult. apply() allows you to apply a function to each value in a series. Syntax-> df["new_column"] = df["column"].apply( lambda x: ... ) Here, we can also use pd.cut alongside groupby which is more effective and fast.

df["Age_Status"] = df["Age"].apply(
    lambda age: "Minor" if age < 18 else "Adult"
)
df[["Age", "Age_Status"]].head(10)

👀 Click to view output

    Age Age_Status
0  22.0      Adult
1  38.0      Adult
2  26.0      Adult
3  35.0      Adult
4  35.0      Adult
5  28.0      Adult
6  54.0      Adult
... [Data Truncated for Readability]

Now, we will use map() to convert the survival value. map() is particularly convenient when you have a dictionary mapping. There is a difference between map() and apply(), map() only modify one column and want to use a dictionary lookup, whereas apply() write a custom function that depends on multiple columns at once.

survival_map = {
    0: "Did Not Survive",
    1: "Survived"
}

df["Survival_Status"] = df["Survived"].map(survival_map)
df[["Survived", "Survival_Status"]].head()

👀 Click to view output

   Survived  Survival_Status
0         0  Did Not Survive
1         1         Survived
2         1         Survived
3         1         Survived
4         0  Did Not Survive

Now, we are going to see the use of replace() function. Syntax-> df["column"].replace({ "old": "new" })

df["Sex"] = df["Sex"].replace({
    "male": "Male",
    "female": "Female"
})
df["Sex"].unique()

👀 Click to view output

array(['Male', 'Female'], dtype=object)

Part H -> Correlation and Crosstab

Correlation-> It measures how strongly two numerical variables are related. Here, we have used the numeric_only which will select only numeric values, we need it because our DataFrame contain categorical columns such as Sex.

df.corr(numeric_only=True)

👀 Click to view output

                PassengerId  Survived    Pclass       Age     SibSp     Parch  \
PassengerId        1.000000 -0.005007 -0.035144  0.034212 -0.057527 -0.001652   
Survived          -0.005007  1.000000 -0.338481 -0.064910 -0.035322  0.081629   
Pclass            -0.035144 -0.338481  1.000000 -0.339898  0.083081  0.018443   
Age                0.034212 -0.064910 -0.339898  1.000000 -0.233296 -0.172482   
SibSp             -0.057527 -0.035322  0.083081 -0.233296  1.000000  0.414838   
Parch             -0.001652  0.081629  0.018443 -0.172482  0.414838  1.000000   
Fare               0.012658  0.257307 -0.549500  0.096688  0.159651  0.216225   
... [Data Truncated for Readability]

Now, we have to find the relationship between Pclass, Fare, Age and Survived Syntax-> df[["col1", "col2", "col3"]].corr() It will create a matrix where each variable is compared with every other variable.

df[["Pclass", "Fare", "Age", "Survived"]].corr()

👀 Click to view output

            Pclass      Fare       Age  Survived
Pclass    1.000000 -0.549500 -0.339898 -0.338481
Fare     -0.549500  1.000000  0.096688  0.257307
Age      -0.339898  0.096688  1.000000 -0.064910
Survived -0.338481  0.257307 -0.064910  1.000000

pd.crosstab() creates a frequency table showing the relationship between categorical variables. The core difference is that correlation measures the mathematical direction and strength of a relationship between numbers, while a crosstab counts frequencies and overlaps between categories. Syntax-> pd.crosstab(df["row_column"], df["column_column"])

pd.crosstab(df["Sex"], df["Survived"])

👀 Click to view output

Survived    0    1
Sex               
Female     81  233
Male      468  109

Now, we are going to create a percentage crosstab using normalize = "index" which convert raw counts to row-wise percentages (proportions). Syntax-> pd.crosstab( df["row_column"], df["column_column"], normalize="index" ) * 100

pd.crosstab(
    df["Sex"],
    df["Survived"],
    normalize="index"
) * 100

👀 Click to view output

Survived          0          1
Sex                           
Female    25.796178  74.203822
Male      81.109185  18.890815

Now, we are going to create the crosstable for passenger class vs survival status. OR pd.crosstab( df["Pclass"], df["Survived"] )

pd.crosstab(
    df["Pclass"],
    df["Survival_Status"]
)

👀 Click to view output

Survival_Status  Did Not Survive  Survived
Pclass                                    
1                             80       136
2                             97        87
3                            372       119

Part I-> Pivot Tables

Pivot table-> It is basically a summarized table that groups data acros multiple dimensions. pd.pivot_table() creates a summarized table by specifying index, columns, values and aggfunc. Syntax -> pd.pivot_table( df, # Define the dataset first to use for create table. index="row_group", columns="column_group", values="value_column", aggfunc="mean" )

pd.pivot_table(
    df,
    index="Pclass",
    columns="Sex",
    values="Survived",
    aggfunc="mean"
)

👀 Click to view output

Sex       Female      Male
Pclass                    
1       0.968085  0.368852
2       0.921053  0.157407
3       0.500000  0.135447

Now, we have to create a pivot table showing the average fare by passenger class and sex.

pd.pivot_table(
    df,
    index="Pclass",
    columns="Sex",
    values="Fare",
    aggfunc="mean"
)

👀 Click to view output

Sex         Female       Male
Pclass                       
1       106.125798  67.226127
2        21.970121  19.741782
3        16.118810  12.661633

Now, we are creating a pivot table showing the average age by passenger class and sex.

pd.pivot_table(
    df,
    index="Pclass",
    columns="Sex",
    values="Age",
    aggfunc="mean"
)

👀 Click to view output

Sex        Female       Male
Pclass                      
1       33.978723  38.995246
2       28.703947  30.512315
3       23.572917  26.911873

Now, we are going to create a pivot table showing the number of passengers by class and sex.

pd.pivot_table(
    df, 
    index="Pclass",
    columns="Sex",
    aggfunc="size"
)

👀 Click to view output

Sex     Female  Male
Pclass              
1           94   122
2           76   108
3          144   347

Now, we are creating the pivot table showing the mean, median, and maximum fare according to the passenger class and sex. Syntax-> pd.pivot_table( df, index="group1", columns="group2", values="column", aggfunc=["function1", "function2", "function3"] )

pd.pivot_table(
    df,
    index="Pclass",
    columns="Sex",
    values="Fare",
    aggfunc=["mean", "median", "max"]
)

👀 Click to view output

              mean               median                max          
Sex         Female       Male    Female     Male    Female      Male
Pclass                                                              
1       106.125798  67.226127  82.66455  41.2625  512.3292  512.3292
2        21.970121  19.741782  22.00000  13.0000   65.0000   73.5000
3        16.118810  12.661633  12.47500   7.9250   69.5500   69.5500

margins=True adds on All rows and/or column containing the overall aggregate. Basically, margins shows the mean(average) of those entire groups or rows because aggfunc = mean.

pd.pivot_table(
    df,
    index="Pclass",
    columns="Sex",
    values="Survived",
    aggfunc="mean",
    margins=True
)

👀 Click to view output

Sex       Female      Male       All
Pclass                              
1       0.968085  0.368852  0.629630
2       0.921053  0.157407  0.472826
3       0.500000  0.135447  0.242363
All     0.742038  0.188908  0.383838

Now, we are going to compare the survival rates of male and female passengers across the three passenger classes.

survival_pivot = pd.pivot_table(
    df, 
    index="Pclass",
    columns="Sex",
    values="Survived",
    aggfunc="mean"
)
survival_pivot

👀 Click to view output

Sex       Female      Male
Pclass                    
1       0.968085  0.368852
2       0.921053  0.157407
3       0.500000  0.135447

groupby() vs pivot_table() groupby-> df.groupby(["Pclass", "Sex"])["Survived"].mean(), produces grouped results, often with a MultiIndex. pivot_table-> pd.pivot_table( df, index="Pclass", columns="Sex", values="Survived", aggfunc="mean" ) Produces a more table/matrix-like layout. Both can answer similar analytical questions, but pivot tables are particularly convenient when you want one grouping variable as rows and another as columns.

Three Final Observations

Survival by Sex Survival by Pclass Average Fare by Pclass

df.groupby("Sex")["Survived"].mean() * 100

df.groupby("Pclass")["Survived"].mean() * 100

df.groupby("Pclass")["Fare"].mean()

👀 Click to view output

Pclass
1    84.154687
2    20.662183
3    13.675550
Name: Fare, dtype: float64

🚀 Wrapping Up

And there you have it! We've taken a raw CSV file and turned it into actionable insights using nothing but Python and Pandas. We learned how to:

Load & Inspect data effortlessly.
Slice & Filter to find exactly what we need.
Clean the inevitable mess of real-world datasets.
Aggregate & Sort to reveal hidden trends.

[!TIP]
Keep Practicing! The best way to master Pandas is to find a dataset you are passionate about (sports, gaming, finance) and start exploring. The sky is the limit!

Happy Coding! 🎈