defdropout_layer(one_layer: tf.tensor, dropout_probability: float): assert0 <= dropout_probability <= 1 if dropout_probability == 1: # in this case, all elements are dropped out return tf.zeros_like(one_layer) # tf.zeros_like: create a tensor with all elements set to zero
if dropout_probability == 0: # in this case, all elements are kept return one_layer
# tf.random.uniform: outputs random values from a uniform distribution. # with (1-p)'s probability that value will become x / (1-p) mask = tf.random.uniform(shape=tf.shape(one_layer), minval=0, maxval=1) < (1 - dropout_probability)
# 在框架中使用快速调用drop out net = tf.keras.models.Sequential([tf.keras.layers.Dense(256, activation=tf.nn.relu, kernel_regularizer=tf.keras.regularizers.l2(0.3)), tf.keras.layers.Dropout(0.5), tf.keras.layers.Dense(256, activation=tf.nn.relu, kernel_regularizer=tf.keras.regularizers.l2(0.3)), tf.keras.layers.Dropout(0.2), tf.keras.layers.Dense(10)])
Dropout in Practice
Typically, we disable dropout at test time. Given a trained model and a new example, we do not drop out any nodes and thus do not need to normalize. However, there are some exceptions: some researchers use dropout at test time as a heuristic for estimating the *uncertainty* of neural network predictions: if the predictions agree across many different dropout outputs, then we might say that the network is more confident.
Dropout通常只运用正在隐藏层，在激活函数之后执行
Posted onEdited onInmachine learning Symbols count in article: 953Reading time ≈3 mins.
Limitation of Linear Models
linearity implies the weaker assumption of monotonicity, i.e., that any increase in our feature must either always cause an increase in our model’s output (if the corresponding weight is positive), or always cause a decrease in our model’s output (if the corresponding weight is negative).
However, the sigmoid has largely been replaced by the simpler and more easily trainable ReLU for most use in hidden layers. Much of this has to do with the fact that the sigmoid poses challenges for optimization since its gradient vanishes for large positive and negative arguments.
Related results suggest that even with a single-hidden-layer network, given enough nodes (possibly absurdly many), and the right set of weights, we can model any function. Actually learning that function is the hard part, though. You might think of your neural network as being a bit like the C programming language. The language, like any other modern language, is capable of expressing any computable program. But actually coming up with a program that meets your specifications is the hard part.
Moreover, just because a single-hidden-layer network can learn any function does not mean that you should try to solve all of your problems with one. In fact, we can approximate many functions much more compactly by using deeper (rather than wider) networks
//other class a car builder that need classCarEngine; classGPS;
//a pure abstract class to use as an interface classCarBuilder { public: virtualvoidreset()= 0; virtualvoidsetSeats(int number)= 0; virtualvoidsetEngine(const CarEngine &engine)= 0; virtualvoidsetGPS(const GPS &gps)= 0; };
classCar; //a car builder that build car itself classCarCarBuilder:public CarBuilder { private: Car *_car; public: virtualvoidreset()override{ if (_car) { delete _car; } _car = new Car(); }
virtualvoidsetSeats(int number)override{ //set seat for car itself }
virtualvoidsetEngine(const CarEngine &engine)override{ //set engine for car itself }
virtualvoidsetGPS(const GPS &gps)override{ //set gps for car itself }
virtual Car* getResult(){ return _car; }; };
classCarManual; //a car builder that build car's manual classCarManualBuilder:public CarBuilder { private: CarManual *_manual;
public: virtualvoidreset()override{ if (_manual) { delete _manual; } _manual = new CarManual() }
virtualvoid setSeats() override { //add description for how to use seats in manual }
virtualvoid setEngine(const CarEngine &engine) override { //add car engine description in manual }
virtualvoid setGPS(const GPS &pgs) override { //add gps use guide in manual }
//a manager class that use builder to build car classCarCarBuilderDirector { public: Car* buildSUV(CarBuilder &builder){ builder.setSeats(4); NormalEngine normalEngine; builder.setEngine(normalEngine); NormalGPS normalGps; builder.setGPS(normalGps); }
