I will be taking deeplearning.ai’s 5-course Deep Learning Specialization on Coursera. Here are my notes from week 1 (Introduction to deep learning) of the first course (Neural Networks and Deep Learning).
- Standard Neural nets: Real estate, online advertising (selecting best ad to show user based on prediction regarding whether they will click on ad or not).
- Convolutional Neural Nets (CNNs): Image recognition.
- Recurrent Neural Nets (RNNs): Audio processing (speech recognition) and text translation.
- Hybrid: Autonomous driving.
- Structured data example: Data arranged neatly such as in a Pandas DataFrame.
- Unstructured data examples: Audio, image, text.
- Unstructured data has traditionally been harder for computers to understand which is why the NN improvements in this area have been so ground-breaking.
- At small scale, the most important factor is manipulating the data (with feature engineering, etc.). At small scale, traditional machine learning algorithms perform comparably well to large neural nets.
- At large scale, large neural nets with large amounts of data will perform the best.
- Increasing the amount of training data usually does not harm the performance of the algorithm.
The code below generates a replica of the graph which Andrew Ng drew to illustrate neural net performance as it compares to data size and neural net size.
# Import libraries import seaborn as sns import numpy as np import matplotlib.pyplot as plt %matplotlib inline #Create x linspace x = np.linspace(1,50,990) #Create dummy functions to represent performance trad_alg = np.log(x) small_nn = np.log(x*np.linspace(1,3,990)) medium_nn = np.log(x*np.linspace(1,6,990)) large_nn = np.log(x*np.linspace(1,10,990))*np.linspace(1,1.3,990) #Plot dummy functions fig = plt.figure(figsize=(10,6)) g = sns.regplot(x, trad_alg, fit_reg=False,label='Traditional Algorithms') sns.regplot(x, small_nn, fit_reg=False,label='Small NN') sns.regplot(x, medium_nn, fit_reg=False,label='Medium NN') sns.regplot(x, large_nn, fit_reg=False,label='Large NN') #Set labels and remove numbers for axes plt.xlabel('Amount of labeled data (m)') plt.ylabel('Performance') plt.legend() g.set(yticks=) g.set(xticks=); #Add annotation g.text(0, -2, "|- - With small amounts of data - -|\n NN complexity does not \n greatly corresepond to \n improved performance", fontsize = 12) #Create title plt.title('Large Neural Nets with Large Training Sets have the Best Performance');
- Neural networks are developed in an iterative fashion. This fact, combined with the need for large neural nets trained on massive data sets mean that improvements in training speed allows for faster innovation with NNs.
- Algorithmic improvments have helped improve training speed. Developments such as using the ReLU function for activation instead of the sigmoid function have improved training speed which allow for faster iterations of implementing NNs.
- Depending on the scale of the application, training can take a wide range of time from 10 minutes to an entire month.