DAT223 - Gathering Requirements and Collecting Data

Preface

• Breakdown of the Course
• How to Read This Course
• Setup

Pandas DataFrame Basics

• Introduction
• Load Your First Data Set
• Look at Columns, Rows, and Cells
• Grouped and Aggregated Calculations
• Basic Plot
• Conclusion

Pandas Data Structures Basics

• Create Your Own Data
• The Series
• The DataFrame
• Making Changes to Series and DataFrames
• Exporting and Importing Data
• Conclusion

Plotting Basics

• Why Visualize Data?
• Matplotlib Basics
• Statistical Graphics Using matplotlib
• Seaborn
• Pandas Plotting Method
• Conclusion

Tidy Data

• Columns Contain Values, Not Variables
• Columns Contain Multiple Variables
• Variables in Both Rows and Columns
• Conclusion

Apply Functions

• Primer on Functions
• Apply (Basics)
• Vectorized Functions
• Lambda Functions (Anonymous Functions)
• Conclusion

Data Assembly

• Combine Data Sets
• Concatenation
• Observational Units Across Multiple Tables
• Merge Multiple Data Sets
• Conclusion

Data Normalization

• Multiple Observational Units in a Table (Normalization)
• Conclusion

Groupby Operations: Split-Apply-Combine

• Aggregate
• Transform
• Filter
• The pandas.core.groupby. DataFrameGroupBy object
• Working With a MultiIndex
• Conclusion

Missing Data

• What Is a NaN Value?
• Where Do Missing Values Come From?
• Working With Missing Data
• Pandas Built-In NA Missing
• Conclusion

Data Types

• Data Types
• Converting Types
• Categorical Data
• Conclusion

Strings and Text Data

• Introduction
• Strings
• String Methods
• More String Methods
• String Formatting (F-Strings)
• Regular Expressions (RegEx)
• The regex Library
• Conclusion

Dates and Times

• Python's datetime Object
• Converting to datetime
• Loading Data That Include Dates
• Extracting Date Components
• Date Calculations and Timedeltas
• Datetime Methods
• Getting Stock Data
• Subsetting Data Based on Dates
• Date Ranges
• Shifting Values
• Resampling
• Time Zones
• Arrow for Better Dates and Times
• Conclusion

Linear Regression (Continuous Outcome Variable)

• Simple Linear Regression
• Multiple Regression
• Models with Categorical Variables
• One-Hot Encoding in scikit-learn with Transformer Pipelines
• Conclusion

Generalized Linear Models

• About This Lesson
• Logistic Regression (Binary Outcome Variable)
• Poisson Regression (Count Outcome Variable)
• More Generalized Linear Models
• Conclusion

Survival Analysis

• Survival Data
• Kaplan Meier Curves
• Cox Proportional Hazard Model
• Conclusion

Model Diagnostics

• Residuals
• Comparing Multiple Models
• k-Fold Cross-Validation
• Conclusion

Regularization

• Why Regularize?
• LASSO Regression
• Ridge Regression
• Elastic Net
• Cross-Validation
• Conclusion

Clustering

• k-Means
• Hierarchical Clustering
• Conclusion

Life Outside of Pandas

• The (Scientific) Computing Stack
• Performance
• Dask
• Siuba
• Ibis
• Polars
• PyJanitor
• Pandera
• Machine Learning
• Publishing
• Dashboards
• Conclusion

It’s Dangerous To Go Alone!

• Local Meetups
• Conferences
• The Carpentries
• Podcasts
• Other Resources
• Conclusion

Appendix A: Concept Maps

Appendix B: Installation and Setup

• B.1 Install Python
• B.2 Install Python Packages
• B.3 Download Book Data

Appendix C: Command Line

• C.1 Installation
• C.2 Basics

Appendix D: Project Templates

Appendix E: Using Python

• E.1 Command Line and Text Editor
• E.2 Python and IPython
• E.3 Jupyter
• E.4 Integrated Development Environments (IDEs)

Appendix F: Working Directories

Appendix G: Environments

• G.1 Conda Environments
• G.2 Pyenv + Pipenv

Appendix H: Install Packages

• H.1 Updating Packages

Appendix I: Importing Libraries

Appendix J: Code Style

• J.1 Line Breaks in Code

Appendix K: Containers: Lists, Tuples, and Dictionaries

• K.1 Lists
• K.2 Tuples
• K.3 Dictionaries

Appendix L: Slice Values

Appendix M: Loops

Appendix N: Comprehensions

Appendix O: Functions

• O.1 Default Parameters
• O.2 Arbitrary Parameters

Appendix P: Ranges and Generators

Appendix Q: Multiple Assignment

Appendix R: Numpy ndarray

Appendix S: Classes

Appendix T: SettingWithCopyWarning

• T.1 Modifying a Subset of Data
• T.2 Replacing a Value
• T.3 More Resources

Appendix U: Method Chaining

Appendix V: Timing Code

Appendix W: String Formatting

• W.1 C-Style
• W.2 String Formatting: .format() Method
• W.3 Formatting Numbers

Appendix X: Conditionals (if-elif-else)

Appendix Y: New York ACS Logistic Regression Example

Appendix Z: Replicating Results in R

• Z.1 Linear Regression
• Z.2 Logistic Regression
• Z.3 Poisson Regression