{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Image Classification with MNIST Using a Petastorm Dataset\n", "\n", "In this notebook we will read a training dataset saved in the Petastorm format in the project's feature store and use that to train a Deep CNN defined in Keras/Tensorflow to classify images of digits in the MNIST dataset.\n", "\n", "This notebook assumes that you have already created the training datasets in the feature store, which you can do by running this notebook: \n", "\n", "[Create Petastorm MNIST Dataset Notebook](PetastormMNIST_CreateDataset.ipynb)\n", "\n", "![Petastorm 6](./../images/petastorm6.png \"Petastorm 6\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Imports" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Starting Spark application\n" ] }, { "data": { "text/html": [ "\n", "
IDYARN Application IDKindStateSpark UIDriver logCurrent session?
4application_1559565096638_0006pysparkidleLinkLink✔
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "SparkSession available as 'spark'.\n" ] } ], "source": [ "from hops import hdfs, featurestore, tensorboard, experiment\n", "\n", "# IMPORTANT: must import tensorflow before petastorm.tf_utils due to a bug in petastorm\n", "import tensorflow as tf\n", "from tensorflow import keras\n", "from tensorflow.python.keras.callbacks import TensorBoard\n", "import json\n", "import numpy as np\n", "import pydoop\n", "from petastorm import make_reader\n", "from petastorm.tf_utils import tf_tensors, make_petastorm_dataset" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Constants\n", "\n", "In this tutorial we will just use static hyperparameters, you can potentially achieve better accuracy by optimizing the hyperparameters using hyperparameter search ([docs](https://hopsworks.readthedocs.io/en/latest/hopsml/hopsML.html))" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "TRAIN_DATASET_NAME = \"MNIST_train_petastorm\"\n", "TEST_DATASET_NAME = \"MNIST_test_petastorm\"\n", "BATCH_SIZE = 64\n", "SHUFFLE_BUFFER_SIZE = 3*BATCH_SIZE\n", "NUM_EPOCHS = 5\n", "STEPS_PER_EPOCH = 80" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Step 1: Define The Model" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "def create_model():\n", " \"\"\"\n", " Defines a three-layer CNN with batch normalization, dropout and max pooling, relu activation, \n", " and softmax output\n", " \"\"\"\n", " model = keras.Sequential()\n", " model.add(keras.layers.Conv2D(filters=32, kernel_size=3, padding='same', activation='relu', input_shape=(28,28,1))) \n", " model.add(keras.layers.BatchNormalization())\n", " model.add(keras.layers.MaxPooling2D(pool_size=2))\n", " model.add(keras.layers.Dropout(0.3))\n", "\n", " model.add(keras.layers.Conv2D(filters=64, kernel_size=3, padding='same', activation='relu'))\n", " model.add(keras.layers.BatchNormalization())\n", " model.add(keras.layers.MaxPooling2D(pool_size=2))\n", " model.add(keras.layers.Dropout(0.3))\n", "\n", " model.add(keras.layers.Flatten())\n", " model.add(keras.layers.Dense(128, activation='relu'))\n", " model.add(keras.layers.Dropout(0.5))\n", " model.add(keras.layers.Dense(10, activation='softmax'))\n", " return model" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "_________________________________________________________________\n", "Layer (type) Output Shape Param # \n", "=================================================================\n", "conv2d (Conv2D) (None, 28, 28, 32) 320 \n", "_________________________________________________________________\n", "batch_normalization (BatchNo (None, 28, 28, 32) 128 \n", "_________________________________________________________________\n", "max_pooling2d (MaxPooling2D) (None, 14, 14, 32) 0 \n", "_________________________________________________________________\n", "dropout (Dropout) (None, 14, 14, 32) 0 \n", "_________________________________________________________________\n", "conv2d_1 (Conv2D) (None, 14, 14, 64) 18496 \n", "_________________________________________________________________\n", "batch_normalization_1 (Batch (None, 14, 14, 64) 256 \n", "_________________________________________________________________\n", "max_pooling2d_1 (MaxPooling2 (None, 7, 7, 64) 0 \n", "_________________________________________________________________\n", "dropout_1 (Dropout) (None, 7, 7, 64) 0 \n", "_________________________________________________________________\n", "flatten (Flatten) (None, 3136) 0 \n", "_________________________________________________________________\n", "dense (Dense) (None, 128) 401536 \n", "_________________________________________________________________\n", "dropout_2 (Dropout) (None, 128) 0 \n", "_________________________________________________________________\n", "dense_1 (Dense) (None, 10) 1290 \n", "=================================================================\n", "Total params: 422,026\n", "Trainable params: 421,834\n", "Non-trainable params: 192\n", "_________________________________________________________________" ] } ], "source": [ "create_model().summary()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Step 2: Define Tensorflow Dataset\n", "\n", "Petastorm datasets can be read directly with tensorflow by using `make_reader` and `make_petastorm_dataset` from the Petastorm library" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "def create_tf_dataset(dataset_url, shuffle_buffer_size, batch_size, num_epochs):\n", " \"\"\"\n", " Defines the Tensorflow Dataset Abstraction from the Petastorm Dataset.\n", " One-hot encodes the labels.\n", " \"\"\"\n", " with make_reader(dataset_url, num_epochs=None, hdfs_driver='libhdfs',\n", " workers_count=1, shuffle_row_groups=False) as train_reader:\n", " train_dataset = make_petastorm_dataset(train_reader)\n", " def preprocess(sample):\n", " return sample.image, tf.one_hot(sample.digit, 10)\n", " return train_dataset.map(preprocess).shuffle(shuffle_buffer_size).batch(batch_size).repeat(num_epochs)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Step 3: Put it All Together in a Training Function\n" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [], "source": [ "def train_fn():\n", " # get dataset path from the featurestore\n", " train_dataset_path = featurestore.get_training_dataset_path(TRAIN_DATASET_NAME)\n", " # get dataset\n", " dataset = create_tf_dataset(train_dataset_path, SHUFFLE_BUFFER_SIZE, BATCH_SIZE, NUM_EPOCHS)\n", " # define model\n", " model = create_model()\n", " # define optimizer\n", " model.compile(loss=keras.losses.categorical_crossentropy,optimizer=keras.optimizers.Adam(),metrics=['accuracy'])\n", " # setup tensorboard\n", " tb_callback = TensorBoard(log_dir=tensorboard.logdir(), histogram_freq=0,write_graph=True, write_images=True)\n", " # setup model checkpointing\n", " model_ckpt_callback = keras.callbacks.ModelCheckpoint(tensorboard.logdir() + '/checkpoint-{epoch}.h5',monitor='acc', verbose=0, save_best_only=True)\n", " callbacks = [tb_callback, model_ckpt_callback]\n", " # train model\n", " history = model.fit(dataset, epochs=NUM_EPOCHS, steps_per_epoch=STEPS_PER_EPOCH, callbacks=callbacks)\n", " # save training history to HDFS\n", " results_path = hdfs.project_path() + \"mnist/mnist_train_results.txt\"\n", " hdfs.dump(json.dumps(history.history), results_path)\n", " # save trained model\n", " model.save(\"mnist_tf_ps.h5\") #Keras can't save to HDFS in the current version so save to local fs first\n", " hdfs.copy_to_hdfs(\"mnist_tf_ps.h5\", hdfs.project_path() + \"mnist\", overwrite=True) # copy from local fs to hdfs\n", " # return latest accuracy\n", " return history.history[\"acc\"][-1]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Step 4: Training Experiments \n", "\n", "We can use the experiments service to run our train_fn function and handle things like Tensorboard, logging, versioning etc. While the experiment is running you can monitor it using Tensorboard and the logs, it is explained in the README [here](https://github.com/logicalclocks/hops-util-py)." ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Finished Experiment \n", "\n", "'hdfs://10.0.2.15:8020/Projects/demo_featurestore_admin000/Experiments/application_1559565096638_0006/launcher/run.2'" ] } ], "source": [ "experiment.launch(train_fn, name=\"mnist_tf_ps\", \n", " description=\"Petastorm MNIST Tensorflow Example\", local_logdir=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Step 5: Plot Training Results\n", "\n", "Inside the `train_fn` function we saved the training history to HDFS, which means we can later read it in %%local mode for plotting." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Load Training Results From HDFS" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [], "source": [ "%%local\n", "import json\n", "from hops import hdfs\n", "import matplotlib.pyplot as plt\n", "from pylab import rcParams\n", "results_path = hdfs.project_path() + \"mnist/mnist_train_results.txt\"\n", "results = json.loads(hdfs.load(results_path))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Plot Loss/Epoch During Training" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[]" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3deZhcdZ3v8fenu9PZk+6kO1tnhyTsIRABBUXZQYeEqxdhVARlGJhRdFRwxufecR1n1HEZldGLirjiTkBlF5BRWUxIWBIIhJCts+8hayf9vX+c00ml02vSVae66/N6nnqoqnOqz7dP6PrU+f3Oqa8iAjMzK11lWRdgZmbZchCYmZU4B4GZWYlzEJiZlTgHgZlZiXMQmJmVOAeBWSsk3S7pc1nX0Z1J+q6kT3T1uta1HAQ9jKQlks7Luo6uJulTkhokvZZz25x1XZ2R/tvskVTT7Pm5kkLS+PTx7enj03LWOVpS5Dx+VNK1OY8/IenVdL+skPTz9Pn5Oftrn6RdOY8PetOV9O2cZXua7e97D+d3johrI+LzXb2udS0HgRUdSRWtLPp5RAzIuVUVtLCu8SpwZdMDSScC/VpYbyPQoaMRSe8F3gOcFxEDgOnAHwAi4vim/QX8D/CBnP130JtuRFyfs+7nOXh/X9zCdlv7d7JuxkFQQiT9naRFkjZKulvSqPR5SfqqpLWStkp6TtIJ6bJLJC2QtE1SvaSPtfKzr5b0Z0nflLRF0ouSzs1ZPljS9yStSn/O5ySVN3vtVyVtAD51GL9bSLpR0mJJ6yV9SVJZuqxM0v+RtDT9HX8oaXDOa8+S9BdJmyUtl3R1zo+ulvT79Pd/UtJRna2tmR8BV+U8fi/wwxbW+wFwkqSzO/AzXwfcHxGvAETE6oi49QjrPETTUYmkayQtAx5I9+2vJK1O99+jko7Nec2PJX0qvX9eelR0s6R1klZKuuow161N/122SnpK0uclPdrVv3OpcBCUCEnnAP8OXA6MBJYCP0sXXwC8CZgMDE7X2ZAu+x7w9xExEDgBeLiNzZwOvALUAJ8EfiNpSLrsdmAvcDQwLd3mtc1euxgYDvzbYf6al5F8Gj4FmAG8L33+6vT2FmAiMAD4JoCkccC9wDeAWuBkYF7Oz7wC+DRQDSw6gtqaPAEMknRsGoRXAD9uYb0dJJ/KO7K9J4CrJN0kaXpTwObRm4BjgLemj38HTAJGAM+ThF1rRgN9gVHA9cC3JA06jHW/BWwm+f/lfSSBaofJQVA63gXcFhFPR8Ru4F+A1ysZl24ABpL8cSsiXoiIVenrGoDjJA2KiE0R8XQb21gLfC0iGiLi58BC4K2ShgOXAB+OiO0RsRb4KsmbYJOVEfGNiNgbETtb+fmXp586m26PNFv+hYjYGBHLgK9xYAjmXcBXImJxRLyW/u5XpEMbfws8FBF3pHVviIjcILgzIp6KiL3AT0iC4kg1HRWcD7wA1Ley3v8Dxko6ZFgmV0T8GPggcCHwR2CtpI93QZ2t+WRE7IiInRHRGBG3R8S2iNhFcjR3qqT+rbx2F/C5dF/fDewm+QDS4XUl9QJmAv+a1tBe+Fg7HASlYxTJUQAA6RviBqAuIh4m+YR8C8mbyK05n7zeTvImvlTSHyW9vo1t1MfB32K4NN3uOKAXsKrpTZzkTW5YzrrLO/A7/CIiqnJub2m2PPdnNG0bmv3u6f0Kkk+TY0iOYlqzOuf+DpKjiUPo4InW9s58+RFJAF1Ny8NCAKSB/dn01qaI+ElEnAdUkXx6/qykC9t73WHav58llUv6Yjokt5XkqAmSo8KWrI+IfTmPW92nbaw7HCjn4H/vjvz/Y61wEJSOlSRvyACkn9iGkn4ajYivR8SpwHEkn9BuSp//a0TMIHnTngX8oo1t1ElSzuOx6XaXk3yaq8l5Ex8UEcfnrNsVX4M7poVtQ7PfPV22F1iT1nak4/4HTbS2d+ZLRCwlmTS+BPhNOz/6+yRv7v+rg3U0RMQvgWdJhvK6XLOwv4rk9ziHZFjx6PR5NX9dF1oDNJIMHTUZ08q61gEOgp6pl6Q+ObcK4A7gGkknS+pNMv78ZEQskfQ6Saenh9zbSQ7JGyVVSnqXpMER0QBsJfkDbM0w4EZJvST9b+BY4J50mOkB4MuSBqUTjEd1cCK0M26SVC1pDPAh4Ofp83cA/yRpgqTcM2KahnvOk3S5pApJQyV1xfBPe94PnBMR29taKa3xk0CrQz1KJtvfKmlgum8vBo4HnuzSils2kCTkN5Cc/XSkcyjtSv9fnAV8WlJfSccD7873dnsyB0HPdA+wM+f2qYh4CPi/wK+BVSSfgpvG6AcB3wE2kQybbAC+lC57D7AkPey/nmS8vTVPkkwarid5Q3hHRDRNOl8FVAIL0u38imTSujPeqYOvI3hNUu7w0l3AHJLJ3t+TTHQD3EYyHPMYySfxXSRj6qTzCZcAHyU5ZXMeMLWTdXVaRLwSEbM7uPodJP9mrdkKfAJYRjKB+kXghoj405FV2SHfJzniWgnMB/5SgG0C3EByRLsmreEOkkCywyA3prGukJ5yeW1EnJXR9gOYFBGL2l3ZehxJXwaqIuL9WdfSHfmIwMy6HUnHSTpRiTOAa4A7s66ru/KVgWbWHQ0imd8ZSTI89B8R8btsS+q+PDRkZlbiPDRkZlbiut3QUE1NTYwfPz7rMszMupU5c+asj4jalpZ1uyAYP348s2d39Kw7MzMDkLS0tWUeGjIzK3EOAjOzEucgMDMrcQ4CM7MS5yAwMytxDgIzsxLnIDAzK3ElEwTLNuzg07+dT8O+tr5O38ys9JRMELy8dhvf//MSfjHbHe3MzHKVTBCcc8wwpo+r5ut/eJmde/a1/wIzsxJRMkEgiZsvOoY1W3fzg8eXZF2OmVnRKJkgADhtwhDeMqWWbz36Clt2NmRdjplZUSipIAD42IVT2LKzgVsfeyXrUszMikLJBcHxowZz6dRR3PanJazdtivrcszMMldyQQDwkfMn07CvkW8+7D7nZmZ5CwJJt0laK+n5Vpa/S9Kzkp6T9BdJU/NVS3Pja/rzzteN4Y6nlrFsw45CbdbMrCjl84jgduCiNpa/CpwdEScCnwVuzWMth7jx3EmUl4mvPvRSITdrZlZ08hYEEfEYsLGN5X+JiE3pwyeA0fmqpSXDB/Xh6jdMYNa8el5cvbWQmzYzKyrFMkfwfuDe1hZKuk7SbEmz161b12UbveHsoxjQu4L/vH9hl/1MM7PuJvMgkPQWkiD4eGvrRMStETE9IqbX1rbYe/mwDO7Xi+vPPoqHXljLnKWtHryYmfVomQaBpJOA7wIzImJDFjVcc+Z4agb05gv3LSQisijBzCxTmQWBpLHAb4D3RERmM7b9Kiv40LlH89SrG/njS1037GRm1l3k8/TRO4DHgSmSVkh6v6TrJV2frvKvwFDgvyXNkzQ7X7W0552vG8uYIX354n0LaWz0UYGZlZaKfP3giLiyneXXAtfma/udUVlRxkfPn8KHfz6P3z+3ir+ZOirrkszMCibzyeJicenUURwzYiBffmChm9eYWUlxEKTKysRNF05hyYYd/HL2iqzLMTMrGAdBjnOOGcap46r5rz+8xK4GN68xs9LgIMghiY83Na/5y5KsyzEzKwgHQTOnTRjCm6fU8t9uXmNmJcJB0IKb0uY133lscdalmJnlnYOgBcePGszfTB3F9/70qpvXmFmP5yBoxUfT5jW3uHmNmfVwDoJWjK/pz+WvG8NP3bzGzHo4B0EbPnTuJMrk5jVm1rM5CNowfFAfrj5zvJvXmFmP5iBoh5vXmFlP5yBoR1W/SjevMbMezUHQAW5eY2Y9mYOgA/pVVnCjm9eYWQ/lIOigK9y8xsx6KAdBB1VWlPGR8yezYNVWfv/cqqzLMTPrMg6CTrh0ap2b15hZj+Mg6ITyMvGxC9y8xsx6FgdBJ517rJvXmFnP4iDoJEncfOEUN68xsx7DQXAYTp841M1rzKzHcBAcpo9d4OY1ZtYzOAgO0wl1bl5jZj2Dg+AIfOT8yexx8xoz6+YcBEdgQk1/3pk2r1m+0c1rzKx7chAcoRvPSZvXPOjmNWbWPTkIjtCIwUnzmjvdvMbMuikHQRc40LzGRwVm1v04CLrAgeY1a9y8xsy6HQdBF3HzGjPrrhwEXcTNa8ysu3IQdKGm5jVfut/Na8ys+8hbEEi6TdJaSc+3slySvi5pkaRnJZ2Sr1oKpal5zfyVbl5jZt1HPo8IbgcuamP5xcCk9HYd8K081lIwTc1rvvLgS25eY2bdQt6CICIeA9o6hWYG8MNIPAFUSRqZr3oKpal5zavrt7t5jZl1C1nOEdQBy3Mer0ifO4Sk6yTNljR73brin4h18xoz6066xWRxRNwaEdMjYnptbW3W5bTLzWvMrDvJMgjqgTE5j0enz/UIbl5jZt1FlkFwN3BVevbQGcCWiOhRp9q4eY2ZdQf5PH30DuBxYIqkFZLeL+l6Sdenq9wDLAYWAd8B/iFftWQlt3nNum27sy7HzKxFFfn6wRFxZTvLA/jHfG2/WHzk/Mnc89wqvvnwy3x6xglZl2NmdohuMVncnbl5jZkVOwdBAbh5jZkVMwdBAeQ2r1m4elvW5ZiZHcRBUCBNzWu+dP/CrEsxMzuIg6BADm5esynrcszM9nMQFFBT85ov3veim9eYWdFwEBRQU/OaJ1/dyGMvr8+6HDMzwEFQcE3Na75434tuXmNmRcFBUGC5zWvueb5HfaOGmXVTDoIMXDq1jinDB/LlB9y8xsyy5yDIQHmZuOnCpHnNr+a4eY2ZZctBkJFzjx3GKWOr+NpDbl5jZtlyEGREEh+/6BjWbN3NDx9fknU5ZlbCHAQZOn3iUM6enDSv2brLzWvMLBsOgozddOEUNu9w8xozy46DIGMn1A3mbSeNdPMaM8uMg6AIfPSCKeze28gtjyzKuhQzK0EOgiIwoaY/l08fw0+eXOrmNWZWcA6CIvGhc9PmNQ+5eY2ZFZaDoEiMGNyHq98wnjvnunmNmRWWg6CI3PBmN68xs8JzEBSRqn6V/P2bJrp5jZkVlIOgyFxz5gQ3rzGzgnIQFJn+vSv44DluXmNmheMgKEJXnjaW0dVuXmNmheEgKEJuXmNmheQgKFIzTnbzGjMrDAdBkSovEx9z8xozKwAHQRE7z81rzKwAHARFTBI3u3mNmeWZg6DIneHmNWaWZw6CbsDNa8wsn/IaBJIukrRQ0iJJ/9zC8rGSHpE0V9Kzki7JZz3dlZvXmFk+dSgIJB0lqXd6/82SbpRU1c5ryoFbgIuB44ArJR3XbLX/A/wiIqYBVwD/3dlfoFS4eY2Z5UtHjwh+DeyTdDRwKzAG+Gk7rzkNWBQRiyNiD/AzYEazdQIYlN4fDKzsYD0lx81rzCxfOhoEjRGxF7gM+EZE3ASMbOc1dcDynMcr0udyfQp4t6QVwD3AB1v6QZKukzRb0ux169Z1sOSex81rzCwfOhoEDZKuBN4L/C59rlcXbP9K4PaIGA1cAvxI0iE1RcStETE9IqbX1tZ2wWa7JzevMbN86GgQXAO8Hvi3iHhV0gTgR+28pp5kCKnJ6PS5XO8HfgEQEY8DfYCaDtZUkq4/+ygGVFbwnw+4eY2ZdY0OBUFELIiIGyPiDknVwMCI+EI7L/srMEnSBEmVJJPBdzdbZxlwLoCkY0mCoHTHfjqgun8lf3/2RB5c4OY1ZtY1OnrW0KOSBkkaAjwNfEfSV9p6TTqn8AHgfuAFkrOD5kv6jKRL09U+CvydpGeAO4Crw91Y2pU0r6l08xoz6xIVHVxvcERslXQt8MOI+KSkZ9t7UUTcQzIJnPvcv+bcXwCc2ZmCral5zSQ+efd8Hnt5PWdPLt15EzM7ch2dI6iQNBK4nAOTxZahpuY1X7rfzWvM7Mh0NAg+QzLE80pE/FXSRODl/JVl7WlqXvN8vZvXmNmR6ehk8S8j4qSIuCF9vDgi3p7f0qw9bl5jZl2ho5PFoyXdKWltevu1pNH5Ls7a5uY1ZtYVOjo09H2SUz9Hpbffps9Zxpqa1/zXQy+7eY2ZHZaOBkFtRHw/Ivamt9sBn6pSBJqa16zeusvNa8zssHQ0CDZIerek8vT2bmBDPguzjnPzGjM7Eh0NgveRnDq6GlgFvAO4Ok812WFw8xozO1wdPWtoaURcGhG1ETEsImYCPmuoiLh5jZkdriPpUPaRLqvCuoSb15jZ4TiSIFCXVWFdws1rzOxwHEkQ+HsNipCb15hZZ7UZBJK2Sdrawm0byfUEVmTcvMbMOqvNIIiIgRExqIXbwIjo6DeXWoG5eY2ZdcaRDA1ZkcptXvP0MjevMbO2OQh6KDevMbOOchD0UE3Na55YvJH/eXl91uWYWRFzEPRgTc1rvujmNWbWBgdBD5bbvObe51dnXY6ZFSkHQQ834+Q6Jg8fwJcfWMheN68xsxY4CHq48jJx04XHsNjNa8ysFQ6CEtDUvOZrbl5jZi1wEJSA3OY1P3p8adblmFmRcRCUiDMmDuVNk2u55dFFbl5jZgdxEJSQm9PmNd918xozy+EgKCEn1A3mrSeN5LtuXmNmORwEJeaj50928xozO4iDoMRMrB3A5dNHu3mNme3nIChBN6bNa7720MtZl2JmRcBBUIJGDu7Le98wnt/MXcFLa9y8xqzUOQhK1A1NzWvud/Mas1LnIChR1f0rue5NE3nAzWvMSl5eg0DSRZIWSlok6Z9bWedySQskzZf003zWYwd731luXmNmeQwCSeXALcDFwHHAlZKOa7bOJOBfgDMj4njgw/mqxw7Vv3cFH3jL0W5eY1bi8nlEcBqwKCIWR8Qe4GfAjGbr/B1wS0RsAoiItXmsx1pw5eluXmNW6vIZBHXA8pzHK9Lnck0GJkv6s6QnJF3U0g+SdJ2k2ZJmr1u3Lk/llqbeFeX803luXmNWyrKeLK4AJgFvBq4EviOpqvlKEXFrREyPiOm1tbUFLrHnmznNzWvMSlk+g6AeGJPzeHT6XK4VwN0R0RARrwIvkQSDFVB5mfjYBVPcvMasROUzCP4KTJI0QVIlcAVwd7N1ZpEcDSCphmSoyF+NmYHzjxvONDevMStJeQuCiNgLfAC4H3gB+EVEzJf0GUmXpqvdD2yQtAB4BLgpIjbkqyZrnSRuvtDNa8xKkbrb+ePTp0+P2bNnZ11Gj3XVbU/x7IrNPHbzWxjUp1fW5ZhZF5E0JyKmt7Qs68liKzJuXmNWehwEdhA3rzErPQ4CO4Sb15iVFgeBHSK3ec2tj73C6i27si7JzPKoIusCrDj90/mTeXnNa3z+nhf593tf5PUThzJzWh0XnTDCk8hmPYzPGrI2vbp+O3fNq2fW3HqWbNhB74oyzjt2ODOn1XH25FoqK3xQadYdtHXWkIPAOiQimLd8M3fNW8lvn1nJhu17qOrXi7eeOJLLptVx6rhqJGVdppm1wkFgXaphXyN/enk9s+bVc//81exqaGTMkL7MmFrHzGl1HD1sQNYlmlkzDgLLm9d27+WB+auZNW8lf3p5HY0BJ9YNZsbJo7h06iiGDeqTdYlmhoPACmTttl389plV3DWvnmdXbKFMcObRNcw8uY4LTxjBgN4+N8EsKw4CK7hX1r3GXXPruXNePcs37qRPrzIuOG4EM6eN4o2TaulV7klms0JyEFhmIoKnl21i1tyV/O7ZlWza0cCQ/pX8zUkjmTGtjmljqjzJbFYADgIrCnv2NvLYS+uYNa+eBxesYffeRsYN7cfMk5NJ5gk1/bMu0azHchBY0dm2q4H7nl/NXfNW8udX1hMBU8dUMfPkUbztpFHUDuyddYlmPYqDwIra6i27+O0zK5k1r575K7dSXibOOrqGy6bVccHxw+lX6UlmsyPlILBu46U125g1t5675q2kfvNO+lWWc8FxyZXMZx1dQ4Unmc0Oi4PAup3GxmD20k3cObeee55bxZadDdQMqORtJ43isml1nDR6sCeZzTrBQWDd2u69+3h04Tpmza3nDy+uZc/eRibW9GfGyXXMnDaKcUM9yWzWHgeB9RhbdjZw3/OruHNuPU++upEIOGVsFTOn1fHWE0cydIAnmc1a4iCwHmnl5p3c/cxKZs2t58XV26goE2dPrmXGtDrOP3Y4fSvLsy7RrGg4CKzHe2HVVmbNq+fueStZtWUX/SvLueiEkcycNoo3HFVDeZnnE6y0OQisZDQ2Bk++upFZc+u55/lVbNu1l2EDe3Pp1FHMnFbH8aMGeZLZSpKDwErSroZ9PPLiWu6cW88jC9fSsC84etgALptWx6VTRzFmSL+sSzQrGAeBlbzNO/Zwz3OrmTW3nqeWbATgdeOr908yV/WrzLhCs/xyEJjlWLFpB3fNSyaZX177Gr3KxZunDOOyaXWcc8ww+vTyJLP1PA4CsxZEBAtWbd1/JfPabbsZ2LuCi08cwcxpdZwxYShlnmS2HsJBYNaOfY3BE4s3cOfceu57fjWv7d7LiEF9mHFyMsl87MhBWZdodkQcBGadsKthHw+9sIZZc+t5dOE69jYGU4YPZOa0OmacPIpRVX2zLtGs0xwEZodp4/Y9/P65VcyaW8+cpZsAOH3CEC6bVsfFJ45kcN9eGVdo1jEOArMusGzDDu6al7TfXLxuO5XlZZxzzDBmThvFW44ZRu8KTzJb8XIQmHWhiOC5+i3MmruSu59ZyfrXdjOoTwWXnDiSsybVMH3cEEYM7pN1mWYHcRCY5cnefY38+ZUN3DW3nvvmr2bHnn0A1FX15dRx1UwfX80pY6s5ZsRA91KwTDkIzAqgYV8jC1ZuZfbSTTy9dBOzl25kzdbdAPSvLOfksVWcOm4Ip46rZtrYKgb18fyCFU5mQSDpIuC/gHLguxHxH62s93bgV8DrIqLNd3kHgXUXEcGKTTt5etkmZi/ZxJylm3hx9VYaAySYMnzg/qOGU8cOYcyQvv4eJMubTIJAUjnwEnA+sAL4K3BlRCxott5A4PdAJfABB4H1ZNt2NfDM8i3MXrqROUs3MXfZZl7bvReA2oG9OXVsOpw0rpoTRg2mssLDSdY12gqCfHYFPw1YFBGL0yJ+BswAFjRb77PAF4Cb8liLWVEY2KcXZ02q4axJNUByIdtLa7YdNJx03/zVAFRWlDF19OD9w0mnjqtmSH9/J5J1vXwGQR2wPOfxCuD03BUknQKMiYjfS2o1CCRdB1wHMHbs2DyUapaN8jJx7MhBHDtyEO85YxwAa7fuYs7SZChp9tJNfO9Pi/n2H5Mj94k1/Q8MJ42rZmLNAH8Nhh2xfAZBmySVAV8Brm5v3Yi4FbgVkqGh/FZmlq1hg/pw8YkjufjEkUBypfOzK5LhpKeXbuKhF9bwyzkrAKjq14tTxlbvP2KYOrrKndms0/IZBPXAmJzHo9PnmgwETgAeTSfIRgB3S7q0vXkCs1LSp1c5p00YwmkThgDJJPTi9duZs6TpqGEjD7+4FoCKMnH8qEH7h5Omj69m+CBf02Bty+dkcQXJZPG5JAHwV+BvI2J+K+s/CnzMk8Vmnbdp+x6eXnZgOOmZ5ZvZvbcRSK5paBpKOnVcNceMGOTWnSUok8niiNgr6QPA/SSnj94WEfMlfQaYHRF352vbZqWmun8l5x47nHOPHQ7Anr2NLFi1ldlLNvL0sk08/soG7pq3EkiuaZg2NjkzaXp6TcNAX9NQ0nxBmVkJaLqmIXcSemGzaxqajhqmjxvC6Gpf09DT+MpiMzvEtl0NzFu+eX84NL+mYfq4A8NJx/uahm4vq+sIzKyIDezTizdOquWNk2qB5JqGhau3MSe92G320k3c+3xyTUPvijKmjq7aP5x06rhqqn1NQ4/hIwIza9WaZtc0zK/fwt7G9JqG2v45Rw1DOKq2v4eTipiHhsysS+xq2MczyzczZ9mm5PTVZZvYvKMBSK5pODVnEvokX9NQVDw0ZGZdok+vck6fOJTTJw4FkknoV9ZtP2g46Q+51zTUDd7//UnTx1UzzNc0FCUfEZhZl9q4fQ9PL920/6jhmRUHrmkYXd13/3DSKeOqOap2AH16+aihEDw0ZGaZ2bO3kfkrtxw017Bu2+79y2sG9Kauui+jq/om/63uS116v66qr69x6CIOAjMrGrl9GpZt2EH95p3Ub97Jik3Jf/ekRw9NBvftdVAwjN4fFv2oq+5Ldb9enqTuAM8RmFnRkMSYIf0YM6TfIcsaG4P123dTv+lAMNSn/126YTt/WbSe7Wk70Cb9KssPCorkqKLf/tCoHdDb39DaDgeBmRWNsjIxbGAfhg3sw7Sx1Ycsjwi27GxgxSFBkRxZzFu+ef9ZTE0qy8sYVdXnQFBU9UuOKNLHIwf3Kfl+0g4CM+s2JFHVr5KqfpWcUDe4xXW27967PyBWbNrBipyjikcWrjtofgKgTDBiUJ/kKOKgo4rk/qiqvj1+QttBYGY9Sv/eFUwePpDJwwe2uHxXwz5Wbdm1Pyj2h8bmnTz16kZWb93FvsaD505rBvTefxRx8KR2Eh4Denfvt9LuXb2ZWSf16VXOhJr+TKjp3+LyvfsaWb111/6jiBWbDhxRLFi5lQfnr2HPvkMntJuf7TQ6Z66iqsgntB0EZmY5KsrLGF3dj9HVh05mQzqh/dru/UNOK5rmKDbtZMmG7fxp0Xp2tDKhfWBuot/+o4rRVX2pyXhC20FgZtYJZWVi2KA+DBvUh1NamdDevKNh/9FE7vBT/eadzG1jQrvpCCL3qKKuui8jBuV3QttBYGbWhSRR3b+S6v6tT2i/tnvvgbOd0qOKpiOMhxeuPWRCu7xMjBjUh2vOHM+1b5zY5TU7CMzMCmxA7wqmjBjIlBGtT2iv3Lwz5+yn5H7twN55qcdBYGZWZPr0Kmdi7QAm1g4oyPZK+yoKMzNzEJiZlToHgZlZiXMQmJmVOAeBmVmJcxCYmZU4B4GZWYlzEJiZlbhu16pS0jpg6WG+vAZY34XldJVirQuKtzbX1Tmuq3N6Yl3jIqK2pQXdLgiOhKTZrfXszFKx1gXFW5vr6hzX1TmlVpeHhszMSpyDwMysxJVaENyadQGtKNa6oHhrc12d47o6p6TqKqk5AjMzO1SpHRGYmVkzDgIzsxLXI4NA0m2S1kp6vpXlkvR1SYskPSvplCs8tQEAAAUzSURBVCKp682Stkial97+tQA1jZH0iKQFkuZL+lAL6xR8f3Wwriz2Vx9JT0l6Jq3r0y2s01vSz9P99aSk8UVS19WS1uXsr2vzXVfOtsslzZX0uxaWFXx/dbCuLPfXEknPpdud3cLyrv2bjIgedwPeBJwCPN/K8kuAewEBZwBPFkldbwZ+V+B9NRI4Jb0/EHgJOC7r/dXBurLYXwIGpPd7AU8CZzRb5x+Ab6f3rwB+XiR1XQ18s5D7K2fbHwF+2tK/Vxb7q4N1Zbm/lgA1bSzv0r/JHnlEEBGPARvbWGUG8MNIPAFUSRpZBHUVXESsioin0/vbgBeAumarFXx/dbCugkv3wWvpw17prfkZFzOAH6T3fwWcK0lFUFcmJI0G3gp8t5VVCr6/OlhXMevSv8keGQQdUAcsz3m8giJ4k0m9Pj28v1fS8YXccHpIPo3k02SuTPdXG3VBBvsrHU6YB6wFHoyIVvdXROwFtgBDi6AugLenQwm/kjQm3zWlvgbcDDS2sjyT/dWBuiCb/QVJiD8gaY6k61pY3qV/k6UaBMXqaZLvA5kKfAOYVagNSxoA/Br4cERsLdR229NOXZnsr4jYFxEnA6OB0ySdUIjttqcDdf0WGB8RJwEPcuBTeN5IehuwNiLm5HtbndHBugq+v3KcFRGnABcD/yjpTfncWKkGQT2Qm+6j0+cyFRFbmw7vI+IeoJekmnxvV1Ivkjfbn0TEb1pYJZP91V5dWe2vnO1vBh4BLmq2aP/+klQBDAY2ZF1XRGyIiN3pw+8CpxagnDOBSyUtAX4GnCPpx83WyWJ/tVtXRvuradv16X/XAncCpzVbpUv/Jks1CO4Grkpn3s8AtkTEqqyLkjSiaWxU0mkk/z55/YNIt/c94IWI+EorqxV8f3Wkroz2V62kqvR+X+B84MVmq90NvDe9/w7g4Uhn+LKsq9kY8qUk8y55FRH/EhGjI2I8yUTwwxHx7marFXx/daSuLPZXut3+kgY23QcuAJqfadilf5MVh11tEZN0B8kZJTWSVgCfJJk8IyK+DdxDMuu+CNgBXFMkdb0DuEHSXmAncEW+/yBIPhm9B3guHV8G+AQwNqeuLPZXR+rKYn+NBH4gqZwkeH4REb+T9BlgdkTcTRJgP5K0iOTkgCvyXFNH67pR0qXA3rSuqwtQV4uKYH91pK6s9tdw4M70M04F8NOIuE/S9ZCfv0l/xYSZWYkr1aEhMzNLOQjMzEqcg8DMrMQ5CMzMSpyDwMysxDkIzJqRtC/nGyfnSfrnLvzZ49XKt8+aZaVHXkdgdoR2pl/VYFYSfERg1kHpd8R/Mf2e+KckHZ0+P17Sw+mXk/1B0tj0+eGS7ky/FO8ZSW9If1S5pO8o6RvwQHolsFlmHARmh+rbbGjonTnLtkTEicA3Sb69EpIvvPtB+uVkPwG+nj7/deCP6ZfinQLMT5+fBNwSEccDm4G35/n3MWuTryw2a0bSaxExoIXnlwDnRMTi9AvxVkfEUEnrgZER0ZA+vyoiaiStA0bnfHFZ01dqPxgRk9LHHwd6RcTn8v+bmbXMRwRmnROt3O+M3Tn39+G5OsuYg8Csc96Z89/H0/t/4cAXpb0L+J/0/h+AG2B/05jBhSrSrDP8ScTsUH1zvvEU4L6IaDqFtFrSsySf6q9Mn/sg8H1JNwHrOPBNkB8CbpX0fpJP/jcAmX/duVlzniMw66B0jmB6RKzPuhazruShITOzEucjAjOzEucjAjOzEucgMDMrcQ4CM7MS5yAwMytxDgIzsxL3/wEUgyL9uZzRnAAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "%%local\n", "%matplotlib inline\n", "y = results[\"loss\"] #loss\n", "x = list(range(1, len(y)+1))#epoch\n", "plt.title(\"Loss per Epoch - MNIST Training\")\n", "plt.xlabel(\"Epoch\")\n", "plt.ylabel(\"Loss\")\n", "plt.plot(x,y)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Plot Accuracy/Epoch During Training" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[]" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3deXwV9b3/8deHhLAvgbAnLEKAqLhAXOpWFFBRq1ZbxVart7b+bN3tcvU+ent77b2t2lZrW7tYa2utG9arpYoboHVXgjsk7EvClkAIe/bP74+Z4CGeJCeQk8nyfj4e55HZ55PJyXxmvt85n2PujoiISH1dog5ARETaJiUIERGJSwlCRETiUoIQEZG4lCBERCQuJQgREYlLCUKklZjZVDMrijqO9iw8hotbelmJTwminTKzV8xsm5l1izqW9sjMRpuZm9mueq+Lo44tUWb2o/B3uKHe9BvC6T8Kx6eG47+tt9zrZnZFOHyFmb0eM+8kM3vTzLabWamZvWFmx5jZf8Qcq3Izq4kZX1xv+yfHzNsd53iPbO7v7O6vuPthLb2sxKcE0Q6Z2WjgZMCBc1t536mtub+W0ETM/d29d8zr8VYLrGUsA75Wb9rl4fRYu4HLwvdOo8ysL/AM8GtgADAC+G+gwt1/UnesgKuBt2KO3X4nY3d/LWbZunmxx3tdvf12MTOdk9oQ/THap68BbwN/ITgZ7GNmPczsF2a2Nrz6e93MeoTz6q4Ky8ysMObq8RUz+0bMNupfTbqZXWNmy4Hl4bR7wm3sMLNFZnZyzPIp4ZXmSjPbGc7PMrN7zewX9eKdY2Y3xfslw/1eb2arzGyLmf0s9gRiZl83s/zwTuoFMxvVWMzNYWZ/MbPfm9lL4e/wr3rbP8HMFobHeKGZnRAzb4CZ/dnMNoSxPV1v298xs2Iz22hm/9bc2OpZCPQ0s8PCbR8GdA+nxyojeL/8VwLbHA/g7o+6e42773X3F939o4OM9TPC9+ePzewtgiQ20sy+Ef5dd4bvodj35nQzWxMzXmRmN5vZx+Hf4lEL76qbs2w4/1Yz22Rm683sm+F7aHRL/87tiRJE+/Q14OHwdYaZDYmZ93NgCnACwdXf94Ha8OT2HMFV4SDgKOCDZuzzfOA44NBwfGG4jQHAI8ATZtY9nHczcAlwFtAX+DqwB3gQuKTuJG9mGcD0cP2GfBHIBSYD54XbwszOA/4DuCD8fV4DHm0i5ub6KvBjIIPgWD0c7nsA8CzwK2AgcBfwrJkNDNd7COhJcNU8GLg7ZptDgX4EV+VXAveaWfoBxlfnIT69i7g8HI/nf4ELzWxCE9tbBtSY2YNmNrMF4mvKZQR/175AEbAZODsc/ybwazM7opH1LwJmAIcQvPcva+6yZnYOcB1wKkGCPO3Af50OxN31akcv4CSgCsgIxwuAm8LhLsBe4Mg4690KPNXANl8BvhEzfgXwesy4A6c1Ede2uv0CS4HzGlguH5gRDl8LzG1kmw6cGTP+bWB+OPwccGXMvC4ESWhUIjEDo8Nlyuq9csL5fwEei1m+N1ADZBGcVN6tt723wuM2DKgF0uPsc2r490mNmVYMHH+A74UfAX8DRgLrgK7hz6xw+o9i9lsUDt8JPB4Ovw5c0cDfPCc8BkVANTAHGFJv//ut00Ssdcc7td7014EfNrHuM8A14fB0YE3MvCJgVsz4XcBvDmDZvwI/jpk3MYx39IH+r3aEl+4g2p/LgRfdfUs4/gifNjNlEDQvrIyzXlYD0xNVGDtiZt8NmwG2m1kZwVVxRgL7ehC4NBy+lIavduPtdy0wPBweBdwTNpeVAaWAEVyZx425ARnu3j/mlR9vfXffFe5jePhaW287a8N9ZwGl7r6tgf1tdffqmPE9BMlnP7Z/B2+jT+J40Ja/AvgJsNzdG/u97yC46zyyiW3mu/sV7p4JHE7wO/+ysXUOQv331jlm9o4FneNlwOl8+t6KZ1PMcNzjmcCyw+vFkch7p8Nrdx2OnVnYl3ARkGJmdW/0bkD/8B/+Y6AcGAt8WG/1QuDYBja9m6BJpM7QOMvsK/sb9jd8H5gGLHb3WjPbRnCCrtvXWOCTONv5G/BJGG8O8HScZWJlAXUnyJHAhph9/K+7P9zIugdbqjirbsDMehM0p20IX6PqLTsSeD6Ma4CZ9Xf3sgPdsbu/RuMnuvr+CjwANNqn4e5bzeyXBE1nicZSYGZ/Af5fM+Jpjtj3Vg/g78As4Fl3rzKzZ/j0vZUsG4HMmPGshhbsTHQH0b6cT9DMcShB+/9RBCfZ14CvuXstwUniLjMbHnYWfy7siHsYmG5mF5lZqpkNNLOjwu1+AFxgZj3NbBxB23hj+hA0O5QAqWb2Q4L24jr3Az82s2wLHFHXPu/uRQT9Fw8BT7r73ib29T0zSzezLOAGoO4po98Dt8Z0zvYzsy83sa3mOsuCjv00ghPq2+HV+VxgvJl9JTyWFxP8TZ5x940EzV+/DePuamantHBc8TxOcKU9O4Fl7yLoo8qJN9PMJoYd6ZnheBZBn9LbLRRrY7oBaQTvrZqwb2BaK+x3NnClmU0ws57Af7bCPts8JYj25XLgz+6+zt031b2A3wBfteBxzu8S3EksJGgSuQPoEjZDnAV8J5z+AVDXzHA3UEnQOfggYWdsI14guFpeRtC0Us7+t+R3EfzDvQjsAP4E9IiZ/yAwiaablwD+ASwK43023Bbu/lT4uz1mZjsI7lZmJrC9+sps/2fzb46Z9wjBUz+lBB2al4b73gqcQ3AstxLcTZ0T0+x3GUE/UQFBH8ONBxBXs3jwpNG8BBIu7r6DoC9iQAOL7CTo3H/HzHYTJIZPCH7fpArvum4CniI47l8i6INI9n7/CfwOeJXgqbc3wlkVyd53W2Zhh4xIqwmvqP9G0KHc4BvQzBzIdvcVrRbcp/v+C0HH7g9ae98SPTObBLwHdAvvzDsl3UFIqzKzrgRNRfc3lhxEWpuZfdHM0sLHmG8H/tGZkwMoQUgrMrMcgkdJh5G8J2JEDtQ1wBaCJ8LKw/FOTU1MIiISl+4gREQkrg7zOYiMjAwfPXp01GGIiLQrixYt2uLug+LN6zAJYvTo0eTl5UUdhohIu2Jm9asC7KMmJhERiUsJQkRE4lKCEBGRuJQgREQkLiUIERGJSwlCRETiUoIQEZG4OsznIEREOrKaWqdsTyXb9lRSuruK0t2VlO2ppHRPJf17pPGV40a2+D6VIEREWlltrbN9bxWleyrZtrsyPNnvPx4kgkq27ali255Ktu+toqHSeZNH9leCEBFpa2prnZ3l1ZTWndDDk3vdlf623ZWfnvj3BImgbE8ltQ2c7NNSuzCwVxrpPdNI79WV4f17MCAcH9ArjfReaaT37PrpeM80eqSlJOV3S2qCMLMzgXuAFIL6/7fXmz+K4CsyBxF8e9Sl4VdSYmaXA3Vf1vI/7v5gMmMVEXF3dlZU73cVv213VczVfF0SqApP9sEVfk0DZ/u0lC6k9wpO5uk908gZ2pf0Xl0Z0DM40ded4OuSwYBeafTomoJZsr+COzFJSxBmlgLcC8wAioCFZjbH3ZfELPZz4K/u/qCZnQb8FLgs/MKO/wJyCb7QfFG47rZkxSsiHYu7s7uyZt/Jvu4qftueelf1+674g+nVDZzsU7tYcFIPT+bZg3vHjKcxICYR1F3p90prOyf7A5HMO4hjgRXuvgrAzB4DzgNiE8ShQN13AL8MPB0OnwG85O6l4bovAWcCjyYxXhFpo9ydvVU1+12972vK2Xeyr4o52QfjlTXxvxAupYvta6ZJ75XGmIxeTIltxgl/9u/Zdd/Jvk+31HZ9sj8QyUwQI9j/i+yLCL4IPdaHwAUEzVBfBPqY2cAG1h1RfwdmdhVwFcDIkS3fQSMirae21vlo/XYWFBSzqmTXvjb8srBZp6I6/snejPDKPTiZZw3oyZGZ/fe7qv+07T644u/TPZUuXTrXyf5ARN1J/V3gN2Z2BfAqsB6oSXRld78PuA8gNzdXX40n0s7sqazm9eVbmJ9fzIKlxZTsrKCLwaiBvRjQK40R/btz+PC++07wsc05/cOTfd8eXUnRyT4pkpkg1gNZMeOZ4bR93H0DwR0EZtYbuNDdy8xsPTC13rqvJDFWEWklG7fvZX5+MfPzN/Pmyq1UVNfSp3sqnx8/iOk5Q5g6YRD9e6ZFHaaQ3ASxEMg2szEEiWEW8JXYBcwsAyh191rgVoInmgBeAH5iZunh+OnhfBFpZ2prnU82bGdemBQWb9gBwKiBPfnqcaOYnjOYY8YMoGuKCju0NUlLEO5ebWbXEpzsU4AH3H2xmd0G5Ln7HIK7hJ+amRM0MV0TrltqZj8mSDIAt9V1WItI27e3soY3VmxhfsFm5ucXUxw2HU0Zlc4tMycyPWcwYwf17nSdvu2NeUMfzWtncnNzXV85KhKdTdvLmV+wmQX5xby+YgsV1bX07hY0HU3LGczUCYMZ0EtNR22NmS1y99x486LupBaRdsrd+WT9Dublb2Z+wWY+WR80HWUN6MElx45kes4Qjh0zgLRUNR21V0oQIpKw8qqg6WhefjELCjazeUcFZjB5ZDrfP3MC03OGkD1YTUcdhRKEiDSqeEc58wuCDubXV2yhvKqWXmkpnDJ+ENNyhnDqhEEM7N0t6jAlCZQgRGQ/7s7iDTuCR1ELNvNR0XYARvTvwaxjRjItZzDHjhlAt9TkFIiTtkMJQkQor6rhrZVbmZe/mQUFxWzcXo4ZHJ3Vn++dETQdjR+ipqPORglCpJMq3lnOywXFzMsv5vXlW9hbVUOvtBROzh7EzTMGc+rEwWSo6ahTU4IQ6STcnSUbd7Agv5h5BcV8WFgGBE1HX87NZFrOEI4/RE1H8iklCJEOrLyqhrdWbWV+fvD5hA1h09GRmf357unjmZYzhIlD+6jpSOJSghDpYEp2VoRNR8FTR3sqa+jRNYWTszO4cfp4Tp04mEF91HQkTVOCEGnn3J2CTTuZn7+ZefnFfFhUhjsM79edCyaPYFrOED53yEC6d1XTkTSPEoRIO1RRXcPbq0qZnx/UOlpftheAIzP7cdP08UzLGcyhw/qq6UgOihKESDuxZVfQdDQ/v5jXlpewO2w6Oik7g+unjePUiYMZ3Kd71GFKB6IEIdJGuTvLNu8Kah3lb+b9wqDpaGjf7px/9Aim5wzhc2PVdCTJowQh0oZUVNfw7upS5ucHncxF24KmoyMy+3HjtKDp6LDhajqS1qEEIRKxrbsqeHlpCfPzN/Pa8i3sqqime9cunDQug2tOHcdpEwczpK+ajqT1KUGItDJ3Z3lxXdNRMe+t24Y7DOnbjS8cOZzpOYM5YWwGPdLUdCTRUoIQaQWV1bW8u7p033cnFJYGTUeHj+jL9adlMz1nCIePUNORtC1KECJJUlvrvLFyC0/kFfFyQTE7K6rpltqFE8dlcPXnxzJt4hCG9lPTkbRdShAiLaywdA9PLCriyUVFrC/bS78eXTn7iGFMyxnCSePUdCTthxKESAsor6rhhcWbmJ1XyBsrtmIGJ2cP4tazJjI9Z4geRZV2SQlC5ADVfSfz43nr+McHG9hZXk3WgB7cPGM8F07JZET/HlGHKHJQlCBEmmnb7kqeen89s/MKKdi0k26pXZh5+FAuOiaL48cMpEsXdTRLx6AEIZKAmlrnteUlPJFXxEtLNlNZU8uRmf34n/MP5wtHDqdfj65RhyjS4pQgRBqxbusenlhUyN8XFbFxeznpPbty6fGjuOiYTCYO7Rt1eCJJpQQhUs/eyhqeX7yRxxcW8vaqUroYnDJ+ED8851Cm5QwhLbVL1CGKtAolCBGCDucPi7YzO6+Qf36wgZ0V1Ywa2JPvnTGBCyaPYFg/dThL56MEIZ3a1l0VPPX+ep7IK2Lp5p1079qFsyYN46LcLI4bM0CfbJZOTQlCOp3qmlpeW76FxxcWMr9gM1U1zlFZ/fnpBZM454hh9OmuDmcRSHKCMLMzgXuAFOB+d7+93vyRwINA/3CZW9x9rpmNBvKBpeGib7v71cmMVTq+1Vt280ReIU++V8TmHRUM7JXGFSeM5su5WYwf0ifq8ETanKQlCDNLAe4FZgBFwEIzm+PuS2IW+wEw291/Z2aHAnOB0eG8le5+VLLik85hT2U1cz8OPuH87uqgw/nUCYP573OzOG3iYHU4izQimXcQxwIr3H0VgJk9BpwHxCYIB+qeFewHbEhiPNJJuDvvF5Yxe2Ehz3y0kV0V1YzJ6MX3z5zAhZMz9d0KIglKZoIYARTGjBcBx9Vb5kfAi2Z2HdALmB4zb4yZvQ/sAH7g7q/V34GZXQVcBTBy5MiWi1zapZKdFTz1fhGz84pYUbyLHl1TOPuIYVx8TBa5o9LV4SzSTFF3Ul8C/MXdf2FmnwMeMrPDgY3ASHffamZTgKfN7DB33xG7srvfB9wHkJub660dvESvuqaWV5aWMDuvkAUFxVTXOlNGpXPHhZM4+4jh9O4W9VtcpP1K5n/PeiArZjwznBbrSuBMAHd/y8y6AxnuXgxUhNMXmdlKYDyQl8R4pR1ZWbKLJ/KKePK9Ikp2VpDRuxtXnjSGL+dmMm6wOpxFWkIyE8RCINvMxhAkhlnAV+otsw6YBvzFzHKA7kCJmQ0CSt29xswOAbKBVUmMVdqB3RXVPPvRRmbnFZK3dhspXYxTJwzm4mOymDphEF1T1OEs0pKSliDcvdrMrgVeIHiE9QF3X2xmtwF57j4H+A7wRzO7iaDD+gp3dzM7BbjNzKqAWuBqdy9NVqzSdrk7i9ZuY3Ze0OG8p7KGQwb14taZE/ni5BEM7qMOZ5FkMfeO0XSfm5vreXlqgeooineU839hSe1VJbvplZbCOUcM56JjMpk8Uh3OIi3FzBa5e268eerBkzajqqaWBQXFPJFXyMtLS6ipdY4Znc63Pj+WsyYNo5c6nEValf7jJHIrincyO6+I/3uviC27KhncpxtXnXIIX56SySGDekcdnkinpQQhkdhZXsWzH23k8bxC3l9XRmoXY1rOYC7KzeLz4weRqg5nkcgpQUircXfeXV3K7Lwi5n68kb1VNWQP7s0Pzs7h/KNHkNG7W9QhikgMJQhJuk3by3nyvSKeyCtkzdY99O6WyvlHj+Ci3EyOyuqvDmeRNkoJQpKisrqWBQWbeXxhIf9aVkKtw3FjBnDdadnMnDSUnml664m0dfovlRa1bPNOHl9YyFPvr6d0dyVD+3bn21PH8aUpmYzO6BV1eCLSDEoQctB2lFfxzw83MDuviA8Ly+iaYsw4dAhfzs3ilOxBpHRRE5JIe6QEIQekttZ5Z3UpT+QVMveTjZRX1TJhSB/+85xD+eLRIxjQKy3qEEXkIClBSLP99a013P/aataV7qFP91S+NCWTi3KzmDSinzqcRToQJQhplrdWbuWH/1jMlFHp3DxjPGccNpQeaSlRhyUiSaAEIQlzd25/Lp9h/brz8DeOo3tXJQaRjkwfV5WEPfvxRj4s2s53Tp+g5CDSCShBSEIqq2u58/mlTBzahy8ePSLqcESkFShBSEIeeWct60r3cMvMiXpsVaSTUIKQJu0sr+JXC1ZwwtiBfH78oKjDEZFWogQhTfrDv1ZRuruSW2fm6DFWkU5ECUIatWl7Ofe/vopzjxzOpMx+UYcjIq1ICUIa9ct5y6ipdb53xoSoQxGRVqYEIQ1avnkns/MKuez40WQN6Bl1OCLSypQgpEF3PF9Ar7RUrj1tXNShiEgElCAkrndWbWVefjHfOnWsCu+JdFJKEPIZ7s5PnytgaN/ufP3EMVGHIyIRUYKQz3juk018UFjGzaePV0kNkU5MCUL2U1VTy53PFzBhSB8unJwZdTgiEiElCNnPo++uY81WldQQESUIibGzvIp75i3n+EMGMHWCSmqIdHZJTRBmdqaZLTWzFWZ2S5z5I83sZTN738w+MrOzYubdGq631MzOSGacEvjjq6vYqpIaIhJK2hcGmVkKcC8wAygCFprZHHdfErPYD4DZ7v47MzsUmAuMDodnAYcBw4F5Zjbe3WuSFW9nV7yjnD++tppzjhjGkVn9ow5HRNqAZN5BHAuscPdV7l4JPAacV28ZB/qGw/2ADeHwecBj7l7h7quBFeH2JEnunrec6tpaldQQkX2SmSBGAIUx40XhtFg/Ai41syKCu4frmrGutJAVxUFJja8eN4pRA3tFHY6ItBFNJggzu87M0pO0/0uAv7h7JnAW8JCZJZy0zOwqM8szs7ySkpIkhdjx3fH8Unp0TeE6ldQQkRiJnIyHEPQfzA47nRPtvVwPZMWMZ4bTYl0JzAZw97eA7kBGguvi7ve5e6675w4apKduDsTCNaW8tGQz35o6loG9u0Udjoi0IU0mCHf/AZAN/Am4AlhuZj8xs7FNrLoQyDazMWaWRtDpPKfeMuuAaQBmlkOQIErC5WaZWTczGxPu/92EfytJiLvzk7n5DOnbTSU1ROQzEmrOcXcHNoWvaiAd+LuZ3dnIOtXAtcALQD7B00qLzew2Mzs3XOw7wDfN7EPgUeAKDywmuLNYAjwPXKMnmFreC4s38f66Mm6eMZ4eaSqpISL7s+Dc38gCZjcAXwO2APcDT7t7VdhXsNzdm7qTaBW5ubmel5cXdRjtRlVNLaff/SqpXYznbjiZ1BR9ZlKkMzKzRe6eG29eIp+DGABc4O5rYye6e62ZndMSAUrre2xhIau37Ob+r+UqOYhIXImcGZ4DSutGzKyvmR0H4O75yQpMkmdXRTX3zFvGsWMGMC1ncNThiEgblUiC+B2wK2Z8VzhN2qk/vrqKLbsquXXmRJXUEJEGJZIgzGM6Kty9liSW6JDkKt5Zzh9fW8XZk4Zx9MhkfbxFRDqCRBLEKjO73sy6hq8bgFXJDkyS4555y6msVkkNEWlaIgniauAEgg+qFQHHAVclMyhJjpUlu3hsYSFfPW4kozNUUkNEGtdkU5G7FxN8yE3auTufL6B7aheum5YddSgi0g40mSDMrDtBSYzDCD7pDIC7fz2JcUkLW7S2lBcWb+Y7M8aToZIaIpKARJqYHgKGAmcA/yKoi7QzmUFJywpKahQwuE83rjxZJTVEJDGJJIhx7v6fwG53fxA4m6AfQtqJF5dsZtHabdw4fTw90/QAmogkJpEEURX+LDOzwwm+2EefrmonqmtqueP5AsYO6sVFuZlRhyMi7Ugil5P3hd8H8QOCKqu9gf9MalTSYh7PK2RVyW7uu2yKSmqISLM0miDCgnw73H0b8CpwSKtEJS1id0U1v5y3nGNGpzPj0CFRhyMi7Uyjl5Thp6a/30qxSAu7/7XVlOys4JaZOSqpISLNlkibwzwz+66ZZZnZgLpX0iOTg1Kys4L7Xl3JzMOHMmWUSmqISPMl0gdxcfjzmphpjpqb2rRfzV9OuUpqiMhBSOST1Hpwvp1ZVbKLR99dx1eOHckhg3pHHY6ItFOJfJL6a/Gmu/tfWz4caQk/e2Ep3VK7cL1KaojIQUikiemYmOHuwDTgPUAJog16b902nvtkEzdOz2ZQH5XUEJEDl0gT03Wx42bWH3gsaRHJAXN3bp9bQEbvbnzzZHURicjBOZBPTu0G1C/RBs3LL+bdNaXcOD2bXt1UUkNEDk4ifRD/JHhqCYKEcigwO5lBSfPVldQ4ZFAvLj4mK+pwRKQDSOQy8+cxw9XAWncvSlI8coCeWFTEiuJd/P7SKXRVSQ0RaQGJJIh1wEZ3Lwcwsx5mNtrd1yQ1MknYnspq7n5pGVNGpXPGYSqpISItI5FLzSeA2pjxmnCatBF/em01xTsr+I+zJqqkhoi0mEQSRKq7V9aNhMNpyQtJmmPLrgr+8OoqzjhsCFNGqQKKiLScRBJEiZmdWzdiZucBW5IXkjTHr+cvZ29VDd8/c2LUoYhIB5NIH8TVwMNm9ptwvAiI++lqaV1rtuzm4XfWcfExWYxVSQ0RaWGJfFBuJXC8mfUOx3clunEzOxO4B0gB7nf32+vNvxs4NRztCQx29/7hvBrg43DeOnc/F9nPz15cSlpqF26crpIaItLyEvkcxE+AO929LBxPB77j7j9oYr0U4F5gBsFdx0Izm+PuS+qWcfebYpa/Djg6ZhN73f2o5vwynckHhWU8+9FGrp+WzeA+3aMOR0Q6oET6IGbWJQeA8NvlzkpgvWOBFe6+KuzYfgw4r5HlLwEeTWC7nZ6789O5+WT0TuOqU1RSQ0SSI5EEkWJm+6q+mVkPIJEqcCOAwpjxonDaZ5jZKILyHQtiJnc3szwze9vMzm9gvavCZfJKSkoSCKljWFBQzDurS7lhWja9VVJDRJIkkbPLw8B8M/szYMAVwIMtHMcs4O/uXhMzbZS7rzezQ4AFZvZx2B+yj7vfB9wHkJub63QC1TW13P5cAWMyejHr2JFRhyMiHVgindR3mNmHwHSCmkwvAKMS2PZ6ILYoUGY4LZ5Z7P+Ndbj7+vDnKjN7haB/YuVnV+1cnnyviOXFu/jdVyerpIaIJFWiZ5jNBMnhy8BpQH4C6ywEss1sjJmlESSBOfUXMrOJQDrwVsy09LpmLTPLAE4EltRft7PZW1nDXS8t4+iR/Tnz8KFRhyMiHVyDdxBmNp6g4/gSgg/GPQ6Yu5/a0Dqx3L3azK4luONIAR5w98VmdhuQ5+51yWIW8Ji7xzYR5QB/MLNagiR2e+zTT53VA2+sZvOOCn59yWSV1BCRpGusiakAeA04x91XAJjZTY0s/xnuPheYW2/aD+uN/yjOem8Ck5qzr46udHclv39lJdNzhnDsGJXUEJHka6yJ6QJgI/Cymf3RzKYRdFJLBH69YDm7K6u5ZeaEqEMRkU6iwQTh7k+7+yxgIvAycCMw2Mx+Z2ant1aAAuu27uFvb6/l4mOyGDe4T9ThiEgn0WQntbvvdvdH3P0LBE8ivQ/8e9Ijk31+9uJSUrt04cbp46MORUQ6kWY9J+nu29z9PneflqyAZH8fFpbxzw838I2TxzCkr0pqiEjr0YP0bZi789Pn8hnYSyU1RKT1KUG0Ya8sLeHtVaVcPy2bPt27Rh2OiHQyShBtVE2tc/tzBYwa2JNLVFJDRCKgBNFGPfleEUs37+T7Z0wkLVV/JhFpfTrztEHlVTXc/dIyjszqz71h3UEAAA41SURBVFmTVFJDRKKhBNEG/fmNNWzcXs6tMyeqpIaIREYJoo3ZtruS376ygmkTB3P8IQOjDkdEOjEliDbmNy+vYHdFNf8+c2LUoYhIJ6cE0YYUlu7hr2+t4ctTshg/RCU1RCRaShBtyM9fXEpKF+OmGSqpISLRU4JoIz4u2s4/PtjAlSeNYWg/ldQQkegpQbQBdSU10nt25f99fmzU4YiIAEoQbcK/lpXw5sqtXD8tm74qqSEibYQSRMTqSmqMHNCTrx43KupwRET2UYKI2NPvr6dg006+d8YEldQQkTZFZ6QIlVfV8IsXl3JEZj/OnjQs6nBERPajBBGhB99cw4bt5dwycyJduqikhoi0LUoQESnbU8m9L6/g1AmDOGFsRtThiIh8hhJERO59eQW7Kqq5ZWZO1KGIiMSlBBGBwtI9PPjmWi6cnMmEoSqpISJtkxJEBO56aRlmcPPpKqkhIm2XEkQr+2T9dp56fz1fP2kMw/r1iDocEZEGKUG0sjueL6B/z65crZIaItLGKUG0oleXlfDa8i1cd1o2/XqopIaItG1JTRBmdqaZLTWzFWZ2S5z5d5vZB+FrmZmVxcy73MyWh6/Lkxlna6gNS2pkpvfg0uNHRh2OiEiTUpO1YTNLAe4FZgBFwEIzm+PuS+qWcfebYpa/Djg6HB4A/BeQCziwKFx3W7LiTbZ/fLieJRt3cM+so+iWmhJ1OCIiTUrmHcSxwAp3X+XulcBjwHmNLH8J8Gg4fAbwkruXhknhJeDMJMaaVOVVNfz8hWVMGtGPLxwxPOpwREQSkswEMQIojBkvCqd9hpmNAsYAC5qzrpldZWZ5ZpZXUlLSIkEnw0NvrWV92V6V1BCRdqWtdFLPAv7u7jXNWcnd73P3XHfPHTRoUJJCOzjb91Txm5dX8PnxgzhxnEpqiEj7kcwEsR7IihnPDKfFM4tPm5eau26b9ttXVrCjvIpbZk6MOhQRkWZJZoJYCGSb2RgzSyNIAnPqL2RmE4F04K2YyS8Ap5tZupmlA6eH09qV9WV7+fOba7jg6ExyhvWNOhwRkWZJ2lNM7l5tZtcSnNhTgAfcfbGZ3QbkuXtdspgFPObuHrNuqZn9mCDJANzm7qXJijVZfvHiUkAlNUSkfUpaggBw97nA3HrTflhv/EcNrPsA8EDSgkuyJRt28NT767nqlEMY0V8lNUSk/WkrndQdzu3PF9C3e1e+/flxUYciInJAlCCS4PXlW3h1WQnXnTaOfj1VUkNE2icliBZWW+v89Ll8RvTvwWWfGxV1OCIiB0wJooX986MNLN6wg++eMV4lNUSkXVOCaEEV1TX87IWlHDqsL+cdGfdD4yIi7YYSRAt66K21FG3by61nqaSGiLR/ShAtZPveoKTGydkZnJzdNst+iIg0hxJEC/ndKyvZvlclNUSk41CCaAEbyvby5zdW88WjRnDY8H5RhyMi0iKUIFrAXS8tw10lNUSkY1GCOEgFm3bw5HtFXHHiaDLTe0YdjohIi1GCOEi3P1dAn26pfHvq2KhDERFpUUoQB+HNFVt4ZWkJ15w6jv4906IOR0SkRSlBHKCgpEYBI/r34PITRkcdjohIi1OCOEDPfLyRj9dv5+YZ4+neVSU1RKTjUYI4AEFJjQJyhvXl/KNVUkNEOiYliAPw8NvrKCzdyy0zJ5Kikhoi0kEpQTTTjvIqfr1gOSeNy+CU7IyowxERSRoliGb6/Ssr2bYnKKlhprsHEem4lCCaYeP2vfzp9dWcd9RwDh+hkhoi0rEpQTTD3WFJje+ePiHqUEREkk4JIkFLN+3k74uKuOxzo8gaoJIaItLxKUEk6I7nC+jVLZVrTx0XdSgiIq1CCSIBb6/ayoKCYr49dRzpvVRSQ0Q6ByWIJrg7P52bz7B+3fm3E0dHHY6ISKtRgmjCsx9v5MMildQQkc5HCaIRldW1/OyFpUwc2ocLJmdGHY6ISKtSgmjEI++sZe3WPfy7SmqISCeU1ARhZmea2VIzW2FmtzSwzEVmtsTMFpvZIzHTa8zsg/A1J5lxxrOzvIpfLVjB5w4ZyNTxg1p79yIikUtN1obNLAW4F5gBFAELzWyOuy+JWSYbuBU40d23mdngmE3sdfejkhVfU/7wr1WU7q7k1rNUUkNEOqdk3kEcC6xw91XuXgk8BpxXb5lvAve6+zYAdy9OYjwJ27yjnPtfX8UXjhzOEZn9ow5HRCQSyUwQI4DCmPGicFqs8cB4M3vDzN42szNj5nU3s7xw+vnxdmBmV4XL5JWUlLRY4He/tIyaWud7KqkhIp1Y0pqYmrH/bGAqkAm8amaT3L0MGOXu683sEGCBmX3s7itjV3b3+4D7AHJzc70lAlq+eSez8wq5/ITRjByokhoi0nkl8w5iPZAVM54ZTotVBMxx9yp3Xw0sI0gYuPv68Ocq4BXg6CTGus8dzy+lV1oq152W3Rq7ExFps5KZIBYC2WY2xszSgFlA/aeRnia4e8DMMgianFaZWbqZdYuZfiKwhCR7d3Up8/I3c/XUsQxQSQ0R6eSS1sTk7tVmdi3wApACPODui83sNiDP3eeE8043syVADfA9d99qZicAfzCzWoIkdnvs009JipefzM1naN/ufP3EMcnclYhIu5DUPgh3nwvMrTfthzHDDtwcvmKXeROYlMzY6nvuk018UFjGnRceQY80ldQQEdEnqYGqmlrufL6A8UN6c+EUldQQEQElCAAefXcda7bu4RaV1BAR2afTJ4hdFdXcM285x40ZwKkTBje9gohIJxH15yAit6eimmNGD+DqqWNVUkNEJEanTxCD+3bn95dNiToMEZE2p9M3MYmISHxKECIiEpcShIiIxKUEISIicSlBiIhIXEoQIiISlxKEiIjEpQQhIiJxWVBQtf0zsxJg7UFsIgPY0kLhtCTF1TyKq3kUV/N0xLhGufugeDM6TII4WGaW5+65UcdRn+JqHsXVPIqreTpbXGpiEhGRuJQgREQkLiWIT90XdQANUFzNo7iaR3E1T6eKS30QIiISl+4gREQkLiUIERGJq1MlCDN7wMyKzeyTBuabmf3KzFaY2UdmNrmNxDXVzLab2Qfh64etFFeWmb1sZkvMbLGZ3RBnmVY/ZgnG1erHzMy6m9m7ZvZhGNd/x1mmm5k9Hh6vd8xsdBuJ6wozK4k5Xt9Idlwx+04xs/fN7Jk481r9eCUQU5THao2ZfRzuNy/O/Jb9f3T3TvMCTgEmA580MP8s4DnAgOOBd9pIXFOBZyI4XsOAyeFwH2AZcGjUxyzBuFr9mIXHoHc43BV4Bzi+3jLfBn4fDs8CHm8jcV0B/Ka132Phvm8GHon394rieCUQU5THag2Q0cj8Fv1/7FR3EO7+KlDayCLnAX/1wNtAfzMb1gbiioS7b3T398LhnUA+MKLeYq1+zBKMq9WFx2BXONo1fNV/CuQ84MFw+O/ANEvyl6EnGFckzCwTOBu4v4FFWv14JRBTW9ai/4+dKkEkYARQGDNeRBs48YQ+FzYRPGdmh7X2zsNb+6MJrj5jRXrMGokLIjhmYdPEB0Ax8JK7N3i83L0a2A4MbANxAVwYNkv83cyykh1T6JfA94HaBuZHcbyaigmiOVYQJPYXzWyRmV0VZ36L/j8qQbQP7xHUSzkS+DXwdGvu3Mx6A08CN7r7jtbcd2OaiCuSY+buNe5+FJAJHGtmh7fGfpuSQFz/BEa7+xHAS3x61Z40ZnYOUOzui5K9r0QlGFOrH6sYJ7n7ZGAmcI2ZnZLMnSlB7G89EHs1kBlOi5S776hrInD3uUBXM8tojX2bWVeCk/DD7v5/cRaJ5Jg1FVeUxyzcZxnwMnBmvVn7jpeZpQL9gK1Rx+XuW929Ihy9H5jSCuGcCJxrZmuAx4DTzOxv9ZZp7ePVZEwRHau6fa8PfxYDTwHH1lukRf8flSD2Nwf4WvgkwPHAdnffGHVQZja0rt3VzI4l+Lsl/aQS7vNPQL6739XAYq1+zBKJK4pjZmaDzKx/ONwDmAEU1FtsDnB5OPwlYIGHvYtRxlWvnfpcgn6dpHL3W909091HE3RAL3D3S+st1qrHK5GYojhW4X57mVmfumHgdKD+k48t+v+YesDRtkNm9ijB0y0ZZlYE/BdBhx3u/ntgLsFTACuAPcC/tZG4vgR8y8yqgb3ArGSfVEInApcBH4ft1wD/AYyMiS2KY5ZIXFEcs2HAg2aWQpCQZrv7M2Z2G5Dn7nMIEttDZraC4MGEWUmOKdG4rjezc4HqMK4rWiGuuNrA8WoqpqiO1RDgqfC6JxV4xN2fN7OrITn/jyq1ISIicamJSURE4lKCEBGRuJQgREQkLiUIERGJSwlCRETiUoIQaQYzq4mp4vmBmd3SgtsebQ1U9BWJQqf6HIRIC9gblqwQ6fB0ByHSAsI6/XeGtfrfNbNx4fTRZrYgLOw238xGhtOHmNlTYTHBD83shHBTKWb2Rwu+t+HF8JPPIpFQghBpnh71mpgujpm33d0nAb8hqAgKQaHAB8PCbg8Dvwqn/wr4V1hMcDKwOJyeDdzr7ocBZcCFSf59RBqkT1KLNIOZ7XL33nGmrwFOc/dVYSHBTe4+0My2AMPcvSqcvtHdM8ysBMiMKfpWV7r8JXfPDsf/Hejq7v+T/N9M5LN0ByHScryB4eaoiBmuQf2EEiElCJGWc3HMz7fC4Tf5tMDcV4HXwuH5wLdg35f59GutIEUSpasTkebpEVNBFuB5d6971DXdzD4iuAu4JJx2HfBnM/seUMKn1TVvAO4zsysJ7hS+BUReWl4klvogRFpA2AeR6+5boo5FpKWoiUlEROLSHYSIiMSlOwgREYlLCUJEROJSghARkbiUIEREJC4lCBERiev/AxS8r88Uu/32AAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "%%local\n", "%matplotlib inline\n", "y = results[\"acc\"] #acc\n", "x = list(range(1, len(y)+1))#epoch\n", "plt.title(\"Accuracy per Epoch - MNIST Training\")\n", "plt.xlabel(\"Epoch\")\n", "plt.ylabel(\"Accuracy\")\n", "plt.plot(x,y)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Step 6: Evaluation Using Trained Model and Test Dataset\n", "\n", "Inside the `train_fn` function we saved the trained model to HDFS in the hdf5 format. We can load the weights of this model and use for serving predictions or for evaluation, in this example we will evaluate the model against the test set. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Load Model Weights" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [], "source": [ "model_path_hdfs = hdfs.project_path() + \"mnist/\" + \"mnist_tf_ps.h5\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In future releases of Tensorflow, Keras will be able to read directly from HDFS, but currently it is not supported. To get around this we can download the hdf5 model in the local file system and load it from there using `model.load_weights()`. " ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Started copying hdfs://10.0.2.15:8020/Projects/demo_featurestore_admin000/mnist/mnist_tf_ps.h5 to local disk on path /srv/hops/hopsdata/tmp/nm-local-dir/usercache/FatxKv_Lnvnybr5ulVBa6ZBxZaETIhWRCL9ga70hOV8/appcache/application_1559565096638_0006/container_e01_1559565096638_0006_01_000001/\n", "\n", "Finished copying" ] } ], "source": [ "local_path = hdfs.copy_to_local(model_path_hdfs, overwrite=True)" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "'/srv/hops/hopsdata/tmp/nm-local-dir/usercache/FatxKv_Lnvnybr5ulVBa6ZBxZaETIhWRCL9ga70hOV8/appcache/application_1559565096638_0006/container_e01_1559565096638_0006_01_000001//mnist_tf_ps.h5'" ] } ], "source": [ "local_path" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [], "source": [ "loaded_model = create_model()\n", "loaded_model.compile(loss=keras.losses.categorical_crossentropy,optimizer=keras.optimizers.Adam(),metrics=['accuracy'])\n", "loaded_model.load_weights(local_path)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Evaluate Loaded Model" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [], "source": [ "# We have 10 000 Test examples, we can evaluate in batches of 100 to speed up the process\n", "BATCH_SIZE = 100\n", "NUM_EPOCHS = 100 \n", "# get dataset path from the featurestore\n", "test_dataset_path = featurestore.get_training_dataset_path(TEST_DATASET_NAME)\n", "test_dataset = create_tf_dataset(test_dataset_path, SHUFFLE_BUFFER_SIZE, BATCH_SIZE, NUM_EPOCHS)" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\r", "1/1 [==============================] - 1s 768ms/step\n", "Test loss: 0.0319652184844017\n", "Test accuracy: 0.9900000095367432" ] } ], "source": [ "score = loaded_model.evaluate(test_dataset, verbose=1, steps=1)\n", "print('Test loss:', score[0])\n", "print('Test accuracy:', score[1])" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "PySpark", "language": "", "name": "pysparkkernel" }, "language_info": { "codemirror_mode": { "name": "python", "version": 2 }, "mimetype": "text/x-python", "name": "pyspark", "pygments_lexer": "ipython3" } }, "nbformat": 4, "nbformat_minor": 2 }