{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The goal of this practical session is to learn how to use different classification and regression models from sklearn. It is based on the book \"Python Data Science Handbook\" by Jake VanderPlas. You are encouraged to play with the code provided.\n",
    "\n",
    "\n",
    "\n",
    "# Using sklearn for building ML models\n",
    "\n",
    "\n",
    "The Scikit-Learn API is designed with the following guiding principles in mind, as outlined in the Scikit-Learn API paper:\n",
    "\n",
    "* Consistency: All objects share a common interface drawn from a limited set of methods, with consistent documentation.\n",
    "\n",
    "* Inspection: All specified parameter values are exposed as public attributes.\n",
    "\n",
    "* Limited object hierarchy: Only algorithms are represented by Python classes; datasets are represented in standard formats (NumPy arrays, Pandas DataFrames, SciPy sparse matrices) and parameter names use standard Python strings.\n",
    "\n",
    "* Composition: Many machine learning tasks can be expressed as sequences of more fundamental algorithms, and Scikit-Learn makes use of this wherever possible.\n",
    "\n",
    "* Sensible defaults: When models require user-specified parameters, the library defines an appropriate default value.\n",
    "\n",
    "In practice, these principles make Scikit-Learn very easy to use, once the basic principles are understood. Every machine learning algorithm in Scikit-Learn is implemented via the Estimator API, which provides a consistent interface for a wide range of machine learning applications.\n",
    "\n",
    "## Basics of the API\n",
    "\n",
    "Most commonly, the steps in using the Scikit-Learn estimator API are as follows (we will step through a handful of detailed examples in the sections that follow).\n",
    "\n",
    "* Choose a class of model by importing the appropriate estimator class from Scikit-Learn.\n",
    "* Choose model hyperparameters by instantiating this class with desired values.\n",
    "* Arrange data into a features matrix and target vector following the discussion above.\n",
    "* Fit the model to your data by calling the fit() method of the model instance.\n",
    "* Apply the Model to new data:\n",
    "  * For supervised learning, often we predict labels for unknown data using the predict() method.\n",
    "  * For unsupervised learning, we often transform or infer properties of the data using the transform() or predict() method.\n",
    "\n",
    "We will now step through several simple examples of applying supervised and unsupervised learning methods.\n",
    "\n",
    "## Supervised learning example: Simple linear regression\n",
    "\n",
    "As an example of this process, let's consider a simple linear regression—that is, the common case of fitting a line to  (𝑥,𝑦)  data. We will use the following simple data for our regression example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x: [3.74540119 9.50714306 7.31993942 5.98658484 1.5601864  1.5599452\n",
      " 0.58083612 8.66176146 6.01115012 7.08072578 0.20584494 9.69909852\n",
      " 8.32442641 2.12339111 1.81824967 1.8340451  3.04242243 5.24756432\n",
      " 4.31945019 2.9122914  6.11852895 1.39493861 2.92144649 3.66361843\n",
      " 4.56069984 7.85175961 1.99673782 5.14234438 5.92414569 0.46450413\n",
      " 6.07544852 1.70524124 0.65051593 9.48885537 9.65632033 8.08397348\n",
      " 3.04613769 0.97672114 6.84233027 4.40152494 1.22038235 4.9517691\n",
      " 0.34388521 9.09320402 2.58779982 6.62522284 3.11711076 5.20068021\n",
      " 5.46710279 1.84854456]\n",
      "y: [ 7.22926896 18.18565441 13.52423055 10.67206599  0.64185082  1.4000462\n",
      " -0.29896653 17.38064514 11.36591852 11.3984114  -0.26422614 18.01311476\n",
      " 14.97193082  3.8584585   3.66749887  3.59937032  4.24562734  9.18591626\n",
      "  7.9701638   5.80012793 10.75788366  1.60421824  3.736558    5.13103024\n",
      "  8.93392551 16.05975926  2.92146552 10.28822167 11.2099274  -0.7161115\n",
      " 11.51229264  3.94851904  0.26520582 19.5423544  15.69289556 15.98984947\n",
      "  5.17932245  0.65443493 12.77642131  5.81548096  1.22109281  9.26065077\n",
      "  1.16566447 16.66813782  3.36710603 11.74868864  6.14962364  9.73011153\n",
      "  9.40444538  3.21035654]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXsAAAD4CAYAAAANbUbJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAZmklEQVR4nO3df7Bc9Xnf8fcH6eJcYY+vbN0SdCVbTMvIQ0xBzo5iR4mHH0ZgTC2VujZM61FTZ1R3ILUTh0ROM8Z1OoM6JM4vPPaooAJTB9uxQVZjYqFBniHOxC5XSBgEplCMgxYZXQcEJFLKlfT0j3tW7F3t7t0fZ885u+fzmtFoz4/d813/ePar5zzn+SoiMDOz0XZG3gMwM7PBc7A3MysBB3szsxJwsDczKwEHezOzElic9wCaWbZsWaxatSrvYZiZDY29e/f+NCImWx0vZLBftWoV09PTeQ/DzGxoSPpxu+NO45iZlcCCwV7SSknfkfS4pAOSPpHsf4uk3ZKeSv5e2uL9m5JznpK0Ke0vYGZmC+tkZn8c+FREnA+8G7he0vnAFuCBiDgPeCDZnkfSW4CbgF8A1gI3tfpRMDOzwVkw2EfEoYh4OHn9KvAEMAVsAO5MTrsT2Njk7VcAuyPixYh4CdgNXJnGwM3MrHNd5ewlrQLWAN8Hzo6IQ8mhnwBnN3nLFPBc3fbBZF+zz94saVrS9MzMTDfDMjOzBXRcjSPpjcA3gE9GxCuSTh2LiJDUV0e1iNgGbAOoVCruzmZmI2XHviq37HqS548cY/nEODdesZqNa5rOfQeio5m9pDHmAv2XI+KeZPcLks5Jjp8DHG7y1iqwsm57RbLPzKw0duyr8ul7HqV65BgBVI8c49P3PMqOfdmFw06qcQTcDjwREZ+vO7QTqFXXbAK+2eTtu4D1kpYmN2bXJ/vMzErjll1Pcmz2xLx9x2ZPcMuuJzMbQycz+3XAR4FLJe1P/lwFbAUul/QU8L5kG0kVSbcBRMSLwO8BDyV/PpfsMzMrjeePHOtq/yAsmLOPiO8CanH4sibnTwO/Wre9Hdje6wDNzIbd8olxqk0C+/KJ8czG4CdozcwG7MYrVjM+tmjevvGxRdx4xerMxlDI3jhmZqOkVnWTZzWOg72ZWQY2rpnKNLg3chrHzKwEHOzNzErAwd7MrAQc7M3MSsDB3sysBBzszcxKwKWXZmYZybPzpYO9mVkGap0vaw3Rap0vawb9I+Bgb2aWgVadLz+78wD/7/jJpj8CaQZ85+zNzDLQqsPlkWOzmbQ/drA3M8tAtx0u025/7GBvZpaBVp0vly4Za3p+2u2PF8zZS9oOXA0cjoh3Jvu+CtR6c04ARyLioibvfRZ4FTgBHI+ISkrjNjMbKq06XwLzbtzCYNofd3KD9g7gVuCu2o6I+EjttaQ/AF5u8/5LIuKnvQ7QzGyQsiyHbNf5MvdqnIh4UNKqZseS9Wk/DFya6qjMzDLQrhwyy3bEWbQ/7jdn/8vACxHxVIvjAdwvaa+kze0+SNJmSdOSpmdmZvoclpnZwoqwEHhW+g321wF3tzn+SxHxLuD9wPWS3tvqxIjYFhGViKhMTk72OSwzs4UVYSHwrPQc7CUtBq4BvtrqnIioJn8fBu4F1vZ6PTOztLWqeMlyIfCs9DOzfx/ww4g42OygpLMkvan2GlgPPNbH9czMUlWEhcCzsmCwl3Q38DfAakkHJX0sOXQtDSkcScsl3Zdsng18V9IjwP8GvhUR305v6GZm/dm4Zoqbr7mAqYlxBExNjHPzNRfkulbsoCgi8h7DaSqVSkxPT+c9DDOzpvLsXtmKpL3tnmVyIzQzsy4UpVyzW26XYGbWhWEt13SwNzPrwrCWazrYm5l1YVjLNR3szcy6MKzlmr5Ba2aFVMSKF2jdvbIIY2vHwd7MCmehipe8fwiyaFyWNgd7MyuchSpehrH0MW/O2ZtZ4bSreBnW0se8eWZvZoWzfGKcapOAv3xifGhLH1vJKiXlmb2ZFU67ipdhLX1spnZvonrkGMHrKakd+6qpX8vB3swKp12DsmEtfWwmy5SU0zhmVkitKl6GtfSxmSxTUg72ZjZ0hrH0sZl29ybS5jSOmVlOskxJdbJ4yXZJhyU9Vrfvs5KqkvYnf65q8d4rJT0p6WlJW9IcuJnZsMty8ZRO0jh3ALcCdzXs/8OI+P1Wb5K0CPgCcDlwEHhI0s6IeLzHsZqZdSTvJ2y7kVVKasGZfUQ8CLzYw2evBZ6OiGci4jXgK8CGHj7HzKxjWZYzDpN+cvY3SPpBkuZZ2uT4FPBc3fbBZJ+Z2cD4Cdvmeg32XwT+KXARcAj4g34HImmzpGlJ0zMzM/1+nJmV1Kg9YZuWnoJ9RLwQESci4iTw35lL2TSqAivrtlck+1p95raIqEREZXJyspdhmZmN1BO2aeop2Es6p27zXwKPNTntIeA8SedKOhO4FtjZy/XMzDo1Sk/YpmnBahxJdwMXA8skHQRuAi6WdBEQwLPAf0jOXQ7cFhFXRcRxSTcAu4BFwPaIODCQb2FmlhilJ2zTpIjIewynqVQqMT09nfcwzMyGhqS9EVFpddxP0JqZlYCDvZlZCTjYm5mVgIO9mVkJONibmZWAg72ZWQl48RIzS90wdZ0sCwd7M0tVretkrRlZresk4ICfI6dxzCxV7jpZTA72ZpYqd50sJgd7M0uVu04Wk4O9maXKXSeLyTdozSxV7jpZTA72Znaafksns1pE2zrnYG9m87h0cjQ5Z29m87h0cjQtGOwlbZd0WNJjdftukfRDST+QdK+kiRbvfVbSo5L2S/JqJGZDwKWTo6mTmf0dwJUN+3YD74yIfw78H+DTbd5/SURc1G4FFTMrDpdOjqYFg31EPAi82LDv/og4nmx+D1gxgLGZWQ5cOjma0sjZ/3vgL1scC+B+SXslbW73IZI2S5qWND0zM5PCsMysFxvXTHHzNRcwNTGOgKmJcW6+5gLfnB1yHS04LmkV8BcR8c6G/f8ZqADXRJMPkjQVEVVJ/4S51M+vJf9SaMsLjpuZdWdgC45L+nfA1cC/aRboASKimvx9GLgXWNvr9czMrHc9BXtJVwK/BXwwIo62OOcsSW+qvQbWA481O9fMzAZrwYeqJN0NXAwsk3QQuIm56ps3ALslAXwvIj4uaTlwW0RcBZwN3JscXwz8WUR8eyDfwsx64kVGymPBYB8R1zXZfXuLc58HrkpePwNc2NfozGxg/KRsufgJWrOS8pOy5eLeOGZDKI30i5+ULRcHe7MhsmNflc/uPMCRY7On9vWaflk+MU61SWDv5ElZ5/qHj9M4ZkOilmOvD/Q1vaRfen1StjaO6pFjBK//2OzYV+3q+pYtz+zNCq42i242C6/Xbfql10VG2uX6PbsvLgd7swJrrJhpp5dGZb0sMuJc/3ByGseswJrNopvJslGZu2IOJwd7swLrZLa8dMlYpo3K3BVzODmNY1ZgrSpmACbGx5DgyNHZUzdnswj4XlB8OHXU9TJr7nppNqdVzn587AyOnwxmT0TdvkVuRVxiA+t6aWaDV+stv3TJ2Lz9x2ZPzgv0c/v89Ku15mBvVnAb10yx5MzOMq6uiLFWHOzNhkCnQdwVMdaKb9Ca9SGrtgHtbtTWuCLG2vHM3qxHWbYNaFbuOHaGWLpkzOvEWkc6mtlL2s7cEoSHa+vQSnoL8FVgFfAs8OGIeKnJezcBv5ts/teIuLP/YZvlL8u2AS53tH51msa5A7gVuKtu3xbggYjYKmlLsv3b9W9KfhBuYm5R8gD2StrZ7EfBbNhk3Tagl9YGZjUdpXEi4kHgxYbdG4DaLP1OYGOTt14B7I6IF5MAvxu4ssexmhWK2wbYMOknZ392RBxKXv+EuTVnG00Bz9VtH0z2nUbSZknTkqZnZmb6GJZZNprl0QGOvnbc7X6tcFK5QRtzj+H29ShuRGyLiEpEVCYnJ9MYltlA1R54mhif/8DTS0dn3d/dCqefYP+CpHMAkr8PNzmnCqys216R7DMbCRvXTHHWG06/9eWnWa1o+gn2O4FNyetNwDebnLMLWC9pqaSlwPpkn9nI6ORG7Y59VdZt3cO5W77Fuq17POu3zHUU7CXdDfwNsFrSQUkfA7YCl0t6Cnhfso2kiqTbACLiReD3gIeSP59L9pmNjIVu1HoZPyuCjkovI+K6Focua3LuNPCrddvbge09jc6sINo9KXvjFatP60wp4JJ3zN178jJ+VgR+gtZsAQvNzDeumeJf/fwUqntPAN/YW2XHvqqX8bNCcLA3W0C7mXnNd344c1o5Wu0c1+NbETjYW+l0e7O0k5l5u3O8jJ8VgYO9lUovN0s7mZm3O6dWjz81Me6mZZYbtzi2UunlZmmzG7CNM/OFznFfG8ubg72VSi83SzvpOOmulFZ0DvZWKq0WAVnoZmknM3PP3q3InLO3UvHNUisrz+ytVJxusbJysLfScbrFyshpHDOzEvDM3kZSu142ZmXkYG8jp/bgVK3mvfbgFOCAb6XlNI6NnE562ZiVjYO9jRx3mTQ7nYO9jRx3mTQ7Xc/BXtJqSfvr/rwi6ZMN51ws6eW6cz7T/5DN2vODU2an6/kGbUQ8CVwEIGkRcwuJ39vk1L+KiKt7vY5Zt/zglNnp0qrGuQz4vxHx45Q+z0oozXJJPzhlNl9aOftrgbtbHHuPpEck/aWkn2v1AZI2S5qWND0zM5PSsGxYeFFus8HqO9hLOhP4IPDnTQ4/DLw9Ii4E/hTY0epzImJbRFQiojI5OdnvsGzIuFzSbLDSmNm/H3g4Il5oPBARr0TE3yev7wPGJC1L4Zo2YlwuaTZYaQT762iRwpH0s5KUvF6bXO/vUrimjRiXS5oNVl/BXtJZwOXAPXX7Pi7p48nmh4DHJD0C/AlwbUREP9e00eRySbPB6qsaJyL+AXhrw74v1b2+Fbi1n2tYObhc0myw3AjNCsPlkmaD43YJZmYl4GBvZlYCDvZmZiXgnL31zKtBmQ0PB3vriVeDMhsuTuNYT9zewGy4eGZvC2qWrhmW9gZONZnNcbC3tlqlayaWjPHS0dnTzi9SewOnmsxe5zSOtdUqXRNB4dsbONVk9joHe2urVVrm5WOz3HzNBUxNjCNgamKcm6+5oFAz5mFJNZllwWkca2v5xDjVJsFx+cR44dsbtBu7Wdl4Zm9tDXM3ymEeu1naPLO3tpp1o7zkHZPcsutJfv2r+wtd4eJOmmavUxHby1cqlZiens57GNZEY4ULzM2W+8nXuzzSrH+S9kZEpdXxNNagfVbSo5L2SzotQmvOn0h6WtIPJL2r32taftKucPFC42bZSCtnf0lEXNTiV+X9wHnJn83AF1O6puUg7QoXl0eaZSOLG7QbgLtizveACUnnZHBdG4C014p1eaRZNtII9gHcL2mvpM1Njk8Bz9VtH0z2zSNps6RpSdMzMzMpDMsGIe0KFy80bpaNNIL9L0XEu5hL11wv6b29fEhEbIuISkRUJicnUxiWDcLGNVOpPkzl8kizbPRdehkR1eTvw5LuBdYCD9adUgVW1m2vSPbZkErzYSqXR5plo69gL+ks4IyIeDV5vR74XMNpO4EbJH0F+AXg5Yg41M91bbQU/Ulcs1HQ78z+bOBeSbXP+rOI+LakjwNExJeA+4CrgKeBo8Cv9HlNy5nr4s2GT1/BPiKeAS5ssv9Lda8DuL6f61hxuG2w2XBybxzriuvizYaTe+MUxLCkRlwXbzacHOwLoNfUSB4/EG4bbDacHOwLoF1qpFXwzjJ3Xv+jMrFkjLEzxOzJ1xvouS7erPicsy+AVimQ6pFjnLvlW6zbuue0xmBZ5c4bG5W9dHQWBBPjY4VdocrMTueZfQG0So0ApzpB3vjnjwCvz9qzyp03+1GZPRGc9YbF7L9pfarXMrPB8cy+AJq1DGg0ezL47M4Dp7az6injG7Jmo8HBvgAa+820cuTY7KnX/faU2bGvyrqte1qmiWrcqMxsNDjYF8TGNVP89ZZL+dHWD3R8fq8NybpZMMSNysxGg3P2BbR0ydjcjdAGS8bm/zb32lOmm+ofNyozGw0O9gV007/4OX7ja/s52bA88OzJYMe+at+Btts8vBuVmQ0/p3EKaOOaKd48Pnba/tkTwae+9siCefaFOA9vVj4O9gV1pEkaB+BERN8LczsPb1Y+DvYF1cksu9eHqNJebcrMis85+4K68YrV89ohtNJrvbvz8Gbl0vPMXtJKSd+R9LikA5I+0eSciyW9LGl/8ucz/Q23PBpn34vUvALfeXYz60Q/M/vjwKci4mFJbwL2StodEY83nPdXEXF1H9cprfrZd2PjM3Ce3cw613OwT9aRPZS8flXSE8AU0BjsLQWudzezfqSSs5e0ClgDfL/J4fdIegR4HvjNiDjQ5BwkbQY2A7ztbW9LY1gjx3l2M+tV39U4kt4IfAP4ZES80nD4YeDtEXEh8KfAjlafExHbIqISEZXJycl+h2VmZnX6CvaSxpgL9F+OiHsaj0fEKxHx98nr+4AxScv6uaaZmXWvn2ocAbcDT0TE51uc87PJeUham1zv73q9ppmZ9aafnP064KPAo5L2J/t+B3gbQER8CfgQ8B8lHQeOAddGRDT7MDMzG5x+qnG+C23brxMRtwK39noNMzNLh5+gHbD6xbpdLmlmeXGwH6DGB6FqzcsAB3wzy5SD/QC1WiTkv/yvA57tm1mmHOwHqFWTspeOzp5aicqzfTPLwsgG+zxy5Y3XfPP42LxFwltptSSgmVlaRjLY55Erb3bNsUVi7Awx27i+YBO9tio2M+vESC5e0m5B7SyvOXsieOPPLJ63SMhEk+UGwa2KzWywRnJm3+2C2oO85pGjs+z7zPpT225VbGZ5GMmZfR4Land6TS8JaGZ5GMmZfbMl/dKYPbe76dvNNd2q2MyyNpLBfhALfSx007eba/qpWjPLmorYl6xSqcT09HTew5hn3dY9VJvk5acmxvnrLZd2/DmtcvZO5ZhZPyTtjYhKq+MjmbMfhLRu+uZRKWRmNpJpnG50mlJZPjHedGbf7U3fPCqFzMxKPbOvpVSqR44RvJ6H37Gvetq5N16xmvGxRfP29XLTN49KITOzUgf7blIqaZVMpvWjYWbWjb7SOJKuBP4YWATcFhFbG46/AbgL+HnmliP8SEQ8288109RtSiWNkslBVAqZmS2k52AvaRHwBeBy4CDwkKSdEfF43WkfA16KiH8m6VrgvwEf6WfAaUorD98t19mbWdb6SeOsBZ6OiGci4jXgK8CGhnM2AHcmr78OXFZbgDxtO/ZVWbd1D+du+Rbrtu5pmndv5JSKmZVFP8F+Cniubvtgsq/pORFxHHgZeGuzD5O0WdK0pOmZmZmuBtLNjdb699Ry9ouS3x+3LjCzUVWYG7QRsS0iKhFRmZyc7Oq93dau1/84AJyIODWjd6A3s1HUT7CvAivrtlck+5qeI2kx8GbmbtSmqtsbrX6wyczKpp9g/xBwnqRzJZ0JXAvsbDhnJ7Apef0hYE8MoD9Dt7XrfrDJzMqm52Cf5OBvAHYBTwBfi4gDkj4n6YPJabcDb5X0NPAbwJZ+B9xMJzda62/gntHiHrEfbDKzUdVXnX1E3Afc17DvM3Wv/xH41/1coxML1a43Nh870eQfF67CMbNRNjK9cdrVrjfL0QMskjgZ4QebzGzkjUywb6dVLv5kBD/a+oGMR2Nmlr3ClF4OkpuPmVnZlSLY+0lZMyu7UqRx3HzMzMquFMEe3HzMzMqtFGkcM7Oyc7A3MysBB3szsxJwsDczKwEHezOzEtAAmlD2TdIM8OM2pywDfprRcIrI39/f39+/vFp9/7dHRMvFQAoZ7BciaToiKnmPIy/+/v7+/v7+/t2+z2kcM7MScLA3MyuBYQ322/IeQM78/cvN37/cevr+Q5mzNzOz7gzrzN7MzLrgYG9mVgJDF+wlXSnpSUlPSxrIAuZFJWmlpO9IelzSAUmfyHtMeZC0SNI+SX+R91iyJmlC0tcl/VDSE5Lek/eYsiTp15P/7T8m6W5JP5P3mAZJ0nZJhyU9VrfvLZJ2S3oq+XtpJ581VMFe0iLgC8D7gfOB6ySdn++oMnUc+FREnA+8G7i+ZN+/5hPAE3kPIid/DHw7It4BXEiJ/nOQNAX8J6ASEe8EFgHX5juqgbsDuLJh3xbggYg4D3gg2V7QUAV7YC3wdEQ8ExGvAV8BNuQ8psxExKGIeDh5/Spz/0cvVZN+SSuADwC35T2WrEl6M/Be4HaAiHgtIo7kO6rMLQbGJS0GlgDP5zyegYqIB4EXG3ZvAO5MXt8JbOzks4Yt2E8Bz9VtH6Rkwa5G0ipgDfD9fEeSuT8Cfgs4mfdAcnAuMAP8jySNdZuks/IeVFYiogr8PvC3wCHg5Yi4P99R5eLsiDiUvP4JcHYnbxq2YG+ApDcC3wA+GRGv5D2erEi6GjgcEXvzHktOFgPvAr4YEWuAf6DDf8KPgiQ3vYG5H73lwFmS/m2+o8pXzNXOd1Q/P2zBvgqsrNtekewrDUljzAX6L0fEPXmPJ2PrgA9Kepa5FN6lkv5nvkPK1EHgYETU/jX3deaCf1m8D/hRRMxExCxwD/CLOY8pDy9IOgcg+ftwJ28atmD/EHCepHMlncnczZmdOY8pM5LEXL72iYj4fN7jyVpEfDoiVkTEKub+u98TEaWZ2UXET4DnJK1Odl0GPJ7jkLL2t8C7JS1J/r9wGSW6QV1nJ7Apeb0J+GYnbxqqBccj4rikG4BdzN2J3x4RB3IeVpbWAR8FHpW0P9n3OxFxX45jsmz9GvDlZLLzDPArOY8nMxHxfUlfBx5mrjJtHyPeOkHS3cDFwDJJB4GbgK3A1yR9jLlW8B/u6LPcLsHMbPQNWxrHzMx64GBvZlYCDvZmZiXgYG9mVgIO9mZmJeBgb2ZWAg72ZmYl8P8B/7knEG1AaNcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "rng = np.random.RandomState(42)\n",
    "x = 10 * rng.rand(50)\n",
    "y = 2 * x - 1 + rng.randn(50)\n",
    "plt.scatter(x, y)\n",
    "print(\"x:\",x)\n",
    "print(\"y:\", y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"color:red\">What's the type of x? Of y? Print their value.</span>\n",
    "\n",
    "<span style=\"color:blue\">x and y are one-dimensional arrays with 50 elements.</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now walk through the process of building an ML model\n",
    "\n",
    "1. Choose a class of model\n",
    "\n",
    "In Scikit-Learn, every class of model is represented by a Python class. So, for example, if we would like to compute a simple linear regression model, we can import the linear regression class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LinearRegression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. Choose model hyperparameters\n",
    "\n",
    "An important point is that a class of model is not the same as an instance of a model.\n",
    "\n",
    "Once we have decided on our model class, there are still some options open to us. Depending on the model class we are working with, we might need to answer one or more questions like the following:\n",
    "\n",
    "Would we like to fit for the offset (i.e., y-intercept)?\n",
    "Would we like the model to be normalized?\n",
    "Would we like to preprocess our features to add model flexibility?\n",
    "What degree of regularization would we like to use in our model?\n",
    "How many model components would we like to use?\n",
    "These are examples of the important choices that must be made once the model class is selected. These choices are often represented as hyperparameters, or parameters that must be set before the model is fit to data. In Scikit-Learn, hyperparameters are chosen by passing values at model instantiation. We will explore how you can quantitatively motivate the choice of hyperparameters later.\n",
    "\n",
    "For our linear regression example, we can instantiate the LinearRegression class and specify that we would like to fit the intercept using the fit_intercept hyperparameter:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None, normalize=False)"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = LinearRegression(fit_intercept=True)\n",
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Keep in mind that when the model is instantiated, the only action is the storing of these hyperparameter values. In particular, we have not yet applied the model to any data: the Scikit-Learn API makes very clear the distinction between choice of model and application of model to data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3. Arrange data into a features matrix and target vector\n",
    "\n",
    "Previously we detailed the Scikit-Learn data representation, which requires a two-dimensional features matrix and a one-dimensional target array. Here our target variable y is already in the correct form (a length-n_samples array), but we need to massage the data x to make it a matrix of size [n_samples, n_features]. In this case, this amounts to a simple reshaping of the one-dimensional array:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(50, 1)"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = x[:, np.newaxis]\n",
    "X.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4. Fit the model to your data\n",
    "\n",
    "Now it is time to apply our model to data. This can be done with the fit() method of the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None, normalize=False)"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This fit() command causes a number of model-dependent internal computations to take place, and the results of these computations are stored in model-specific attributes that the user can explore. In Scikit-Learn, by convention all model parameters that were learned during the fit() process have trailing underscores; for example in this linear model, we have the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.9033107255311146"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.coef_\n",
    "model.intercept_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These two parameters represent the slope and intercept of the simple linear fit to the data. Comparing to the data definition, we see that they are very close to the input slope of 2 and intercept of -1.\n",
    "\n",
    "5. Predict labels for unknown data\n",
    "\n",
    "Once the model is trained, the main task of supervised machine learning is to evaluate it based on what it says about new data that was not part of the training set. In Scikit-Learn, this can be done using the predict() method. For the sake of this example, our \"new data\" will be a grid of x values, and we will ask what y values the model predicts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "xfit:  [-1.         -0.75510204 -0.51020408 -0.26530612 -0.02040816  0.2244898\n",
      "  0.46938776  0.71428571  0.95918367  1.20408163  1.44897959  1.69387755\n",
      "  1.93877551  2.18367347  2.42857143  2.67346939  2.91836735  3.16326531\n",
      "  3.40816327  3.65306122  3.89795918  4.14285714  4.3877551   4.63265306\n",
      "  4.87755102  5.12244898  5.36734694  5.6122449   5.85714286  6.10204082\n",
      "  6.34693878  6.59183673  6.83673469  7.08163265  7.32653061  7.57142857\n",
      "  7.81632653  8.06122449  8.30612245  8.55102041  8.79591837  9.04081633\n",
      "  9.28571429  9.53061224  9.7755102  10.02040816 10.26530612 10.51020408\n",
      " 10.75510204 11.        ]\n"
     ]
    }
   ],
   "source": [
    "xfit = np.linspace(-1, 11)\n",
    "print(\"xfit: \",xfit)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"color:red\">What's the effect of linspace?</span>\n",
    "\n",
    "<span style=\"color:blue\">linspace creates an array with 50 (default value) elements the values of which range in between -1 and 11.</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As before, we need to coerce these x values into a [n_samples, n_features] features matrix, after which we can feed it to the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Xfit:  [[-1.        ]\n",
      " [-0.75510204]\n",
      " [-0.51020408]\n",
      " [-0.26530612]\n",
      " [-0.02040816]\n",
      " [ 0.2244898 ]\n",
      " [ 0.46938776]\n",
      " [ 0.71428571]\n",
      " [ 0.95918367]\n",
      " [ 1.20408163]\n",
      " [ 1.44897959]\n",
      " [ 1.69387755]\n",
      " [ 1.93877551]\n",
      " [ 2.18367347]\n",
      " [ 2.42857143]\n",
      " [ 2.67346939]\n",
      " [ 2.91836735]\n",
      " [ 3.16326531]\n",
      " [ 3.40816327]\n",
      " [ 3.65306122]\n",
      " [ 3.89795918]\n",
      " [ 4.14285714]\n",
      " [ 4.3877551 ]\n",
      " [ 4.63265306]\n",
      " [ 4.87755102]\n",
      " [ 5.12244898]\n",
      " [ 5.36734694]\n",
      " [ 5.6122449 ]\n",
      " [ 5.85714286]\n",
      " [ 6.10204082]\n",
      " [ 6.34693878]\n",
      " [ 6.59183673]\n",
      " [ 6.83673469]\n",
      " [ 7.08163265]\n",
      " [ 7.32653061]\n",
      " [ 7.57142857]\n",
      " [ 7.81632653]\n",
      " [ 8.06122449]\n",
      " [ 8.30612245]\n",
      " [ 8.55102041]\n",
      " [ 8.79591837]\n",
      " [ 9.04081633]\n",
      " [ 9.28571429]\n",
      " [ 9.53061224]\n",
      " [ 9.7755102 ]\n",
      " [10.02040816]\n",
      " [10.26530612]\n",
      " [10.51020408]\n",
      " [10.75510204]\n",
      " [11.        ]]\n",
      "yfit:  [-2.88096733 -2.39664326 -1.9123192  -1.42799513 -0.94367106 -0.459347\n",
      "  0.02497707  0.50930113  0.9936252   1.47794926  1.96227333  2.44659739\n",
      "  2.93092146  3.41524552  3.89956959  4.38389366  4.86821772  5.35254179\n",
      "  5.83686585  6.32118992  6.80551398  7.28983805  7.77416211  8.25848618\n",
      "  8.74281024  9.22713431  9.71145837 10.19578244 10.68010651 11.16443057\n",
      " 11.64875464 12.1330787  12.61740277 13.10172683 13.5860509  14.07037496\n",
      " 14.55469903 15.03902309 15.52334716 16.00767122 16.49199529 16.97631936\n",
      " 17.46064342 17.94496749 18.42929155 18.91361562 19.39793968 19.88226375\n",
      " 20.36658781 20.85091188]\n"
     ]
    }
   ],
   "source": [
    "Xfit = xfit[:, np.newaxis]\n",
    "yfit = model.predict(Xfit)\n",
    "print(\"Xfit: \",Xfit)\n",
    "print(\"yfit: \",yfit)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"color:red\">What's the effect of xfit[:, np.newaxis]? Of model.predict(Xfit)? What's the ype of Xfit ? Of yfit ?</span>\n",
    "\n",
    "<span style=\"color:blue\">xfit[:, np.newaxis] creates a matrix (two-dimensional array) from the one-dimensional array xfit. model.predict(Xfit) provides the values predicted by the model learned on the data contained in Xfit, which is a two-dimensional array. The predicted values are stored in yfit, which is a one-dimensional array."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's visualize the results by plotting first the raw data, and then this model fit:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXyV1Z3H8c/JRkLYhASESAgiq6waQcW1Lriz2EWno7ZVqc5Yq3WQoLaitoLaSh2n0ylqWx2p1iFsIkoBrah1CwIJS1hkTwIJYEjIQpZ75o8kNIR7w83dn5vv+x+SJzd5fvclfjk5zzm/Y6y1iIiI88SEuwAREfGNAlxExKEU4CIiDqUAFxFxKAW4iIhDxYXyZikpKTYjIyOUtxQRcbw1a9YctNamtrwe0gDPyMggJycnlLcUEXE8Y8xud9c1hSIi4lAKcBERh1KAi4g4lAJcRMShFOAiIg6lABcRcSgFuIiIQynARUSC6JuKGp54eyNl1bUB/9kh3cgjItJeWGtZlrefx5dsoLSylvEDUrhyWK+A3kMBLiISYAfKqvn5og38bdMBRqR15X/vHMfQ3l0Cfh8FuIhIgFhreStnL798ZzM1dS4euW4IPxrfn7jY4MxWK8BFRAJgz6FKZizM5ZPthxjXvzvP3DySjJTkoN7zlAFujOkLvAb0Aiww11r7gjGmO/BXIAPYBXzXWvtN8EoVEYk89S7Ln/+xi18v30JsjOFXk4dz63npxMSYoN/bmxF4HfCQtfYrY0xnYI0xZgXwA2CVtXa2MSYLyAKmB69UEZHIsvVAOQ/Pz2Xd3lIuH5zKryaPoE+3pJDd/5QBbq0tAooaPy43xmwG0oCJwGWNL3sV+DsKcBFpB2rqXPzPh1/z4vvb6NQhjt9+bzQTR/fBmOCPuptr0xy4MSYDGAN8DvRqDHeA/TRMsbj7nqnAVID09HRf6xQRiQjr95YyPTuX/P3l3DiqD4/fOIyUTh3CUovXAW6M6QRkAw9Ya8ua/0tjrbXGGOvu+6y1c4G5AJmZmW5fIyLij0VrC3hu+RYKS6vo0y2JaRMGM2lMWkDvUVVTz5yVW3n5ox2kdu7AS7dnclWA13W3lVcBboyJpyG851lrFzRePmCM6W2tLTLG9AaKg1WkiIgni9YWMGNBHlW19QAUlFYxY0EeQMBC/NOvDzFjQS67DlVy69i+zLhuKF0S4wPys/1xysWJpmGo/Qqw2Vr7fLMvLQHuaPz4DmBx4MsTEWndc8u3HA/vJlW19Ty3fIvfP7usupZHFuZx60uf4bLwl7vGMWvKyIgIb/BuBD4euA3IM8asa7z2CDAbeMsYcyewG/hucEoUEfGssLSqTde99X7+AR5ZsIHi8mruuqg/D109mKSEWL9+ZqB5swrlY8DTo9UrAluOiEjb9OmWRIGbsPZ1Od+ho8d4cukmFq8rZFCvTvz+Xy9kTPpp/pYZFOpGKCKONm3CYJLiTxwZJ8XHMm3C4Db9HGstS9YXctWc1SzLK+KBKwey9CcXR2x4g7bSi4jDNT2o9GcVStGRKn6+aAMrNxczqm83nr15JINP7xyskgNGAS4ijjdpTJpPK05cLsubX+5l1rLN1LpcPHb9UH44vj+xIdgGHwgKcBFpl3YdrCBrQS6f7TjMBWf2YPbNI+jXI7jNpwJNAS4i7Uq9y/LHj3fymxVbiI+JYfaUEXzvvL4h3wYfCApwEWk3tuwv5+H561m/7whXDu3JLyeN4PSuieEuy2cKcBGJCq1tp6+pc/G7D7bz33/fTpfEeF68dQw3jOztyFF3cwpwEXG81rbTZ6Qk8/D89Ww9cJTJY9L4+Q3D6J6cEM5yA0YBLiKO52k7/fTsXI7VuQDokZzApYNSoya8QRt5RCQKeNo23xTeAIcqapixII9FawtCVVbQKcBFxPG83TYfqCZXkUIBLiKON23CYBK8PPnd3yZXkUQBLiKOdvDoMVZuPkBNvYu4xh2Uad2SOK2j+5avoTyzMtj0EFNEQipQp+dYa1m8rpAn3t5IxbF6fnbVIO65dAAJcTHH79N8ZQr41uQqkinARSRkAnV6TmFpFY8uzOODLSWMSW9oPjWw14nNpwLR5CrSKcBFJGRaOz3Hm2B1uSzzvtjDM+/mU++y/OKGYdxxYYbH5lO+NrlyCgW4iISMP6fn7DxYwfTsXL7YeZiLzkph1pQR9O3eMdAlOooCXERCxpfTc+rqXbzy8U6eX7GVhLgYnr15JN/JPMPx2+ADQQEuIiEzbcLgNj1Y3FRYxvTsXPIKjnD1sF48NWk4vbo4t/lUoCnARSRkvH2weKyunv96fzu/+2D78WsbCo7w6deHonpOu60U4CISUqd6sLhm9zdMz85le/FRYo2h3loACo9U+7RiJZppI4+IRISKY3U88fZGvv0//6Cqpp4eyQnHw7tJtG2F95cCXETC7qNtJUz47Wr+9Mkubju/H8sfvITDFTVuXxtNW+H9pSkUEQmbI5W1/GrZJt7K2ceZKcm89eMLGNu/O+DbipX2RgEuIq0K1Nb3lt7bsJ+fL97A4Yoa/u2yAdx/xUAS42OPf72tK1baIwW4iHjU2tZ38G2bekn5MWYu2cg7eUUM692FP/3gPIandT3pde1hK7y/jG3xkCCYMjMzbU5OTsjuJyL+GT/7fbfTGN2S4jlW5zppdDxrygiPAWutZcFXBTy5dBNVtfX89IqBTL3kTOK9bAPbnhlj1lhrM1te1whcRDzy9MCwtKr2pGut9TTZ900ljy7cwIdbSzi332k8c/NIzurZKeD1NgnWtE+kUYCLiEeeHiR60jLwXS7L65/v5pl387HAzBuHcfsFGcR4aD4VCIHqeOgE+t1FRDyaNmEwSc0eLELDVIk3hyV8XXKU7839lF8s3sg5/U5j+QOX8IPx/YMa3tB6x8NooxG4iHjk6UEi4HGFSG29i7mrd/DCqm0kxcfy6++M4uZz0kLWfMqfjodOowAXkVa1tvW9ZbCf1bMTk373CRsLy7h2+Ok8MfFsenYObfOp9rR+XAEuIj5pHuzVtfX856ptPPR/6zmtYwK///45XDuid1jqak/rx08Z4MaYPwI3AMXW2uGN12YCdwMljS97xFq7LFhFikjkytl1mIezc9lRUsF3zj2D0X278ct3NvNv874KywqQ9rR+3JsR+J+B/wJea3F9jrX21wGvSEQc4eixOp57L5/XPttNn65JvPajsRyuqImIFSDRfpRak1MGuLV2tTEmI/iliIhTfLi1hEcW5FF4pIo7Lshg2oTBJHeIY/zs9/0681Laxp858PuMMbcDOcBD1tpv3L3IGDMVmAqQnp7ux+1EJNxKK2t4aulmsr/ax4DUZObfcwHn9ut+/OvtaQVIJPB1HfjvgQHAaKAI+I2nF1pr51prM621mampqT7eTkTC7d28Iq58fjWL1xVw3+Vn8c79F58Q3uB5pUc0rgCJBD4FuLX2gLW23lrrAl4Cxga2LBGJFMVl1dzzv2u4d95XnN61A4vvG89/TBh8QufAJp42/kTjCpBI4NMUijGmt7W2qPHTycCGwJUkIpHAWsv8Nft4aukmqutcTL9mCHdf3J+4VppPtacVIJHAm2WEbwCXASnGmH3A48BlxpjRgAV2AT8OYo0iEmJ7D1fyyMI8Ptp2kLEZ3Zl18wgGpHrXfKq9rACJBN6sQrnVzeVXglCLiIRZvcvyv5/u4tnlWzDAUxPP5vvj+gW9f4n4RjsxRQSA7cXlTM/OY83ub7h0UCpPTxlBmh4+RjQFuEg7d7z51MptdOwQy/PfHcXkMaFrPiW+U4CLtGN5+47wcHYum4vKuH5Eb2bedDapnTuEuyzxkgJcpJ1ofkpN766JDO3dhb9vLaFHcgJ/uO1cJpx9erhLlDZSgIu0Ay1PqSk8Uk3hkWrO79+dP9yeSdck9wc0SGTTiTwi7YC7U2oA9n5TpfB2MAW4SDvg6VxL9ShxNk2hiESxwxU1PLV0k8evq0eJs2kELhKFrLUszS3kquc/5O31hUw4uxeJcSf+764eJc6nEbiIQzVfVdK858iBsmoeW7SBFZsOMPKMrrx+1ziG9u7i8fXiXMZaG7KbZWZm2pycnJDdTyRatVxVApAYF8OkMWm8k1dETZ2Lh64exI/Gt958SpzBGLPGWpvZ8rpG4CIO5G5VSXWdize/3Mu4/t155uaRZKQkh6k6CRUFuIgDtbZ65I27z1fzqXZCv1uJOJCn1SNp3ZIU3u2IAlzEYWrqXIw4o+tJ17WqpP3RFIqIg6zfW8r07Fzy95dzTno3CkurOVBWrVUl7ZQCXMQBqmrqmbNyKy9/tIOenRN5+fZMrhzWK9xlSZgpwEUi3KdfH2LGglx2Hark1rHpzLhuCF0S1b9ENAcuErHKqmt5ZGEet770GeXVdaQkJ/DmF3u49rcfsWhtQbjLkwigEbhIBFq1+QCPLtxAcXk1lw9O5dOvD1Fd5wIaGlPNWJAHoDnvdk4jcJEQWrS2gPGz36d/1juMn/3+SSPpQ0ePcf8ba7nz1Ry6JsWz4N/Gs/XA0ePh3aSqtp7nlm8JZekSgTQCFwmBRWsLmLlkI6VVtcevNR9JTxzdhyXrC3ni7U2UV9fy4JWDuPeyASTExXjctOPpunqetB8KcJEgc9e3pElVbT2z383n7fWFrMovZlTfbjz37ZEM6tX5+Gv6dEty28/b3WaelvfSdEt00xSKSBAtWlvAQ2+tdxveTfaXVfPJ1wd57PqhLLj3whPCG2DahMEkxceecM3Tph13PVI03RK9NAIXCZKm0XD9KTp+doiL4W8PXEp6j45uv940cvZmWqSt0y3ibApwkSDxdA5lc/GxhtlTRngM7yaTxqR5NQXSlukWcT5NoYgEiTej3uSEOIwJXPOptky3iPMpwEWCxJtRb2lVLTMW5AVsY86kMWnMmjKCtG5JGBq6E86aMkIPMKOUTuQRCZJFawt46P/WUe869WvTuiXxSda3gl+UOJKnE3k0AhcJgsqaOjYUHPEqvEEPGcU3eogp0kwgNsH8Y/tBshbksedwpdffo4eM4guNwEUaNS37KyitwvLPTTDezk8fqaolKzuXf3n5c2IMvDn1fNK8CGY9ZBRfnTLAjTF/NMYUG2M2NLvW3RizwhizrfHP04Jbpkjw+bMJZsWmA1w950PeytnLjy89k/ceuITzz+zhdlVIfIzhtI7xesgofvNmCuXPwH8BrzW7lgWsstbONsZkNX4+PfDliYSOL5tgDh49xswlG1maW8SQ0zvz0u2ZjDyj2/Gvt2UTjkhbnTLArbWrjTEZLS5PBC5r/PhV4O8owMXh2rIJxlrLonUFPPH2JiqO1fGzqwZxz6UNzada8nYTjkhb+ToH3staW9T48X5AZzuJ47mb7jDA5UNST7hWUFrFD//8JQ/+dT39U5JZdv/F3H/FQLfhLRJMfq9CsdZaY4zHxeTGmKnAVID09HR/bycSNJPGpJGz+zDzPttD019oC2SvKSCzX3duGtWHeV/sYfayzbgsPH7jMG6/IIPYmMDtpBRpC18D/IAxpre1tsgY0xso9vRCa+1cYC40bOTx8X4iIfFBfgkt/5JW1dbz9LLN/OXzPXyx6zAXnZXCrCkj6Nu99f4lIsHm6+98S4A7Gj++A1gcmHJEwsvTA8vi8mPk7y/jlvP6sqPkKJc8+4HbE3VEQumUI3BjzBs0PLBMMcbsAx4HZgNvGWPuBHYD3w1mkSLB4G7TjqcHmQBXDuvF4nWFOixBIoZ6oUi75O6UnKT4WG4+N435OftOOoMSGh5ouvu/RX1MJNjUC0WkGU+bdt7bsJ+uHePdfo+noY76mEi4qBeKRB1v+pl4Ct2DR2u82v7enPqYSLhoBC5Rxdt+Jp5CNzkhluUPXuIxxFsuGFQfEwknBbhEFW/7mUybMJjEFhtvEmJj+NXkEXTqEOfxZJvvn5+uwxIkYmgKRaKKt/1MEuNjSYiLOf6wsk/XRB6+ZsjxMFYPE3ECBbhElVP1Mykur2bmko0sy9vPsN5dePbbIxme1tXtz1IPE4l0mkKRqOJp6uM/rh5E9pp9XPX8alZuLmbahMEsvm+8x/AWcQKNwCWquJv6uPOi/ixcV8jqrSWc2+80nrl5JGf17BTmSkX8pwCXqNM09eFyWV7/fDfPvJuPBZ646WxuO78fMWo+JVFCAS6O1dp6769LjpKVncuXu77h4oEpPD1Zzack+ijAxZFaboVvWu9d77LsL6vmhVXbSIqP5dffGcXN56RhjEbdEn0U4OJIntZ7Zy3Ipbbect2I05l509n07JwYpgpFgk8BLo7kab13bb3lf/71HK4Z3jvEFYmEnpYRiiN52grfu0uiwlvaDQW4ONJPvnXWSUeZJcXHMv3aIWGqSCT0NIUiYedN98DmPtxawovvb8flsiQnxFJRU0+atrpLO6QAl7DytJoETj7lprSyhqeWbib7q30MSE1m/r0XcG6/7iGvWSRSKMAlrFrrHtg8wJflFfGLxRsorazlvsvP4r5vnUViiy3zIu2NAlzC6lTdA4vLqvnF4o28t3E/w9O68OqPxnJ2H/UvEQEFuISZp+6Bvbsm8lbOXn65dBPH6lxkXTuEuy7qT1ysnruLNNH/DRJW7roHdoiLoXNiPA/Pz2XI6V1496cXc8+lAxTeIi1oBC5h1bx7YEFpFV2T4qmqqWffN5U8NWk43x+bruZTIh4owCXsJo1JY3haFx6en8tXe0q5bHAqv5o8os2HC4u0NwpwCavaehd/+PBr/nPVdjp2iGXO90YxabSaT4l4QwEufmvrRpwmefuOMG3+evL3l3PDyN7MvOlsUjp1CEHFItFBAS5+actGnCbVtfXMWbmVl1bvIKVTB+bedi5Xn316yGoWiRYKcPGLtxtxmny24xAzFuSx82AFt5zXlxnXDaVrUrxfNfj6G4CI0ynAxWvugvJUG3GalFfXMvvdfOZ9voe+3ZOYd9c4xp+VEpCa2vobgEi0UICLVzwFZbeO8XxTWXvS65u3e/0gv5hHFuaxv6yaOy/qz0NXD6JjQmD+6rX1NwCRaKIAF694CsoOcTEkxcee8LWk+FimTRjM4Yoannx7I4vWFTKwZyey772Qc9JPC2hd3v4GIBKNtLVNvOIpEI9U1TJrSsOabQOkdUvi6cnDiY0xXPX8hyzNLeL+Kway9P6LAh7e4PlgB0/XRaKJRuDiFU89S/p0S2LSmLTj0xUHyqp5dOEGVm4+wMgzujLv7nEMOb1L0OqaNmHwCVM78M/fAESinQJcvOIuKONjDJU1dfTPeofeXRO5eGAqyzYUUVPn4tHrhvLD8Rke+5cEauVI8634WoUi7Y2x1vr+zcbsAsqBeqDOWpvZ2uszMzNtTk6Oz/eT8Goeul2T4qmoqaO2/sS/PwNSk3nljvPISElu9ee4GzXPmjJCwSvihjFmjbt8DcQc+OXW2tGnCm9xvklj0vgk61vsnH09yR3iTgpvgKqa+lbDG1pfOSIi3tNDTPGJu/lwgKIj1af8Xq0cEQkMfwPcAn8zxqwxxkx19wJjzFRjTI4xJqekpMTP20m41dS5eGHlNo9f92b1h1aOiASGvwF+kbX2HOBa4N+NMZe0fIG1dq61NtNam5mamurn7SSc1u8t5cYXP2bOyq2ck96NxLgT//p4u/rD3SEOWjki0nZ+rUKx1hY0/llsjFkIjAVWB6IwiRxVNfU8v2ILr3y8k56dE3n59kyuHNbL55UkWjkiEhg+r0IxxiQDMdba8saPVwBPWmvf8/Q9WoXiPP/4+iAzFuSx+1Al/zIunaxrh9Al0b/mUyLSNp5WofgzAu8FLGxsvB8H/KW18BZnKauuZdayfN74Yg/9enTkL3eP48IBJzafUhdAkfDyOcCttTuAUQGsRSLEyk0HeHRRHiXlx5h6yZk8eOUgkhJOnLNWF0CR8NNOTAcK1sj30NFjPPH2JpasL2Rwr8784bZMRvft5va16gIoEn4KcIdpy8jX26C31rJkfSEzl2zk6LE6HrxyEPdeNoCEOM+LlLSWWyT8FOAO42nk+8Bf1/Hc8i3HQ9rboC86UsVjCzewKr+Y0X278ey3RzKoV2eP92/6R8HTo2+t5RYJHQW4w7Q2wi0orWLa/60HTj3F4XJZ3vhyD7OW5VPncvHY9UP54fj+xMZ4Pg3eXQ+T5rSWWyS0FOAO46mta5Nal2Xmko0cqTr5lBxo+Adg58EKsrJz+XznYS4c0INZU0bQr0fr/UvA/T8KTdK0CkUk5BTgDuOurWtLpVW1pHkI+s6JcVzz29UkxMYwe8oIvndeXxqXgp5yztzT6N8An2R9y783JiJtpmZWDjNpTNrxE3Ba4267ujFQVl3HxQNTWfGzS7llbPoJ4T1jQR4FpVVY/jlnvmhtwfHvVw8TkciiAHegprauHePd/+frGB9zPOj7dE08fj05IY4Xbx3DS7efy+nNroN3LV7Vw0QksmgKxcE6xMdSWetyex0gvUdHkjs0/CeePCaNn98wjO7JCW5/ljfLAtXDRCSyKMAdrLTS/YPKbyprych6B4BuSfH86QfncfmQnq3+rNbOvGyu+fmXIhJemkJxMG/mnqtr6z2uSGlO0yMizqMAdzB3odtSdZ3Lq6PKmj8cNTQsC9QZlSKRTVMoDtYUrk++vYnDlTUeX+ft9nZNj4g4i0bgDlZSfowVmw5wuLKGob27kNqpg9vXaZmfSHTSCNyBrLUsXFvAk0s3UXmsnv+4ehA/vnQA7+QWnbTJR/PYItFLAe4wBaVVPLowj79vKeGc9IbmU2f1bGg+pWV+Iu2LAtwhXC7LvM93M/vdfCww88Zh3HZBxknNpzSPLdJ+KMAdYEfJUbKy8/hi12EuHpjC05NH0Ld7x3CXJSJhpgCPYHX1Ll76aCdzVm4lMS6G5749km+fe8bx/iUi0r4pwCPUpsIyHs5ez4aCMiac3YunJg6nZ5fEU3+jiLQbCvAIU11bz0/eWMuKTQcA6N4xgWuH91Z4i8hJjLWeDscKvMzMTJuTkxOy+znNmt2Huff1ryguP3bC9fhYQ3JCHEeqarWyRKQdMsassdZmtryuEXgEqDhWx3PLt/Dqp7uI4eT57dp6S2ljP5PWDjEWkfZFAe4lb094b6vZ727mpdU7qbeW5IRYKmo8n7TTpPnZliLSfinAveDtCe9tcaSylrtfy+GLXYePX6uoqceAxxPfm/O2v4mIRC/1QvGCN6fVtMV7G4q4cs6HJ4R3EwtuJlFOpv4mIqIA94I3p9V4o7i8mntfX8M9r3/lsfEUNIR4U1vX0zrGE99it6X6m4gIaArFK96eVuOJtZZHFubx5pd7sRa6JMZx50X9eX7FVrc/N61b0gmnvAdr/l1EnE0jcC/4c1rN3sOVXPPCR7zxRUN4Q8PJ8I8t2sDlQ1J1Co6I+EwB7gVfTqtxuSx//mQnE367mq37y0/6elVtPR/kl5zy5zY9QC0orcLyzweoi9YWBP6NioijaCNPEGwvLmd6dh5rdn/DJYNSWb21xO3rDLBz9vWt/qzxs9/3appFRKKXNvIE2aK1BTz7Xj6FR6oB6JgQy2++M4op56Rx0TMf+DyHHqgHqCISfTSFEgCL1hYwPTv3eHhDwxRKbIzBGOPXHLqnkNcyQhHxK8CNMdcYY7YYY7YbY7ICVZSTVNfW89iiDRyrc514vdlp8P6c+O5P+ItIdPN5CsUYEwv8DrgK2Ad8aYxZYq3dFKjiIt2Xuw4zfX4uR4/Vuf1682kOX0/K0TFpIuKJP3PgY4Ht1todAMaYN4GJQNQEuKf110eP1fHse/m89uluuicnEAO43Hx/oKY5dEyaiLjjT4CnAXubfb4PGNfyRcaYqcBUgPT0dD9uF1qe+p9sKipj6fpCisqquWRgCl/sPOw2vDXNISLBFvSHmNbaudbaTGttZmpqarBvFzCe+p/MXb2Djh3imH/PhXxdUkF13cnxHWuM13PcIiK+8mcEXgD0bfb5GY3XokJry/Teuf8iOsTFenyNy1qFt4gEnT8j8C+BgcaY/saYBOAWYElgygo/T/PXad2SeDdvP+Nnv++x7auW+IlIKPgc4NbaOuA+YDmwGXjLWrsxUIWFk7WWiwemnHQ9KT6Wy4ekHt/a7o7mvkUkVPzaiWmtXQYsC1AtEWHv4UpmLMjj4+0HGZCaTHl1HSXlx46vQnE3N94kTUv8RCSEtJW+Ub3L8uo/dvHc8i3ExhiemjSc749NJ6ZFL+4H/7rO7fcbUG8SEQkpBTiw7UA507Nz+WpPKZcNTuXpySNa3cLuT29wEZFAade9UGrrXby4ahvX/+fH7DxYwZzvjeJPPziv1TDW1nYRiRTtdgSeu6+Uh+fnkr+/nBtH9eHxG4eR0soxZ020tV1EIkW7C/Dq2nrmrNjKSx/tILVzB166PZOrhvVq08/Q1nYRiQTtKsA/23GIrOxcdh2q5Naxfcm6dihdk+LDXZaIiE/aRYCXV9cy+9185n2+h/TuHfnLXeO48KyT13mLiDhJ1Af4B/nFPLIwjwNl1dx1UX9+dvUgOiZE/dsWkXYgapPscEUNT769kUXrChnYsxP/fe+FjEk/LdxliYgETNQFuLWWpblFzFyykSNVtdx/xUD+/fIBdIiLPfU3i4g4SFQF+P4j1Ty2aAMrNx9g5BldmXf3OIac3iXcZYmIBEVUBLi1lje/3MvT72ym1uXi0euG8sPxGcTFtut9SiIS5Rwf4LsPVZCVncenOw5x/pndmT1lJBkpyeEuS0Qk6Bwb4PUuy58+2cmv/7aF+JgYnp48glvO63tS8ykRkWjlyADfsr+ch7NzWb+3lCuG9OSXk4fTu6uaSYlI++KoAK+pc/Hff9/O7z7YTufEeF64ZTQ3jeqDMRp1i0j745gAX7e3lOnzc9lyoJyJo/vwixuG0cOL5lMiItHKEQH+4qptzFm5lZ6dE3nljkyuGNq25lMiItHIEQGe3qMjt4xNJ+vaIXRJVPMpERFwSIBPHJ3GxNFq3yoi0px2uoiIOJQCXETEoRTgIiIOpQAXEXEoBbiIiEMpwEVEHEoBLiLiUApwERGHMtba0N3MmBJgd8hu6JsU4GC4iwiAaHkfoPcSiaLlfYAz3ks/a21qy4shDXAnMGubqagAAANaSURBVMbkWGszw12Hv6LlfYDeSySKlvcBzn4vmkIREXEoBbiIiEMpwE82N9wFBEi0vA/Qe4lE0fI+wMHvRXPgIiIOpRG4iIhDKcBFRBxKAd7IGHONMWaLMWa7MSYr3PX4yhjT1xjzgTFmkzFmozHmp+GuyR/GmFhjzFpjzNJw1+IPY0w3Y8x8Y0y+MWazMeaCcNfkK2PMg41/tzYYY94wxiSGuyZvGWP+aIwpNsZsaHatuzFmhTFmW+Ofp4WzxrZQgNMQEsDvgGuBYcCtxphh4a3KZ3XAQ9baYcD5wL87+L0A/BTYHO4iAuAF4D1r7RBgFA59T8aYNOB+INNaOxyIBW4Jb1Vt8mfgmhbXsoBV1tqBwKrGzx1BAd5gLLDdWrvDWlsDvAlMDHNNPrHWFllrv2r8uJyGoHDkeXTGmDOA64GXw12LP4wxXYFLgFcArLU11trS8FbllzggyRgTB3QECsNcj9estauBwy0uTwRebfz4VWBSSIvygwK8QRqwt9nn+3Bo6DVnjMkAxgCfh7cSn/0WeBhwhbsQP/UHSoA/NU4HvWyMSQ53Ub6w1hYAvwb2AEXAEWvt38Jbld96WWuLGj/eD/QKZzFtoQCPUsaYTkA28IC1tizc9bSVMeYGoNhauybctQRAHHAO8Htr7RigAgf9mt5c4/zwRBr+UeoDJBtj/jW8VQWObVhX7Zi11QrwBgVA32afn9F4zZGMMfE0hPc8a+2CcNfjo/HATcaYXTRMaX3LGPN6eEvy2T5gn7W26Teh+TQEuhNdCey01pZYa2uBBcCFYa7JXweMMb0BGv8sDnM9XlOAN/gSGGiM6W+MSaDhocySMNfkE2OMoWGudbO19vlw1+Mra+0Ma+0Z1toMGv57vG+tdeRIz1q7H9hrjBnceOkKYFMYS/LHHuB8Y0zHxr9rV+DQB7LNLAHuaPz4DmBxGGtpk7hwFxAJrLV1xpj7gOU0PFX/o7V2Y5jL8tV44DYgzxizrvHaI9baZWGsSeAnwLzGAcIO4Idhrscn1trPjTHzga9oWPG0FgdtRTfGvAFcBqQYY/YBjwOzgbeMMXfS0O76u+GrsG20lV5ExKE0hSIi4lAKcBERh1KAi4g4lAJcRMShFOAiIg6lABcRcSgFuIiIQ/0/GeAj8B8SyeUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.scatter(x, y)\n",
    "plt.plot(xfit, yfit);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Supervised classification: Naive bayes on Iris dataset\n",
    "\n",
    "Let's take a look at another example of this process, using the Iris dataset. Our question will be this: given a model trained on a portion of the Iris data, how well can we predict the remaining labels?\n",
    "\n",
    "For this task, we will use an extremely simple generative model known as Gaussian naive Bayes, which proceeds by assuming each class is drawn from an axis-aligned Gaussian distribution. Because it is so fast and has no hyperparameters to choose, Gaussian naive Bayes is often a good model to use as a baseline classification, before exploring whether improvements can be found through more sophisticated models.\n",
    "\n",
    "We would like to evaluate the model on data it has not seen before, and so we will split the data into a training set and a testing set. This could be done by hand, but it is more convenient to use the train_test_split utility function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_iris\n",
    "dataset = load_iris()\n",
    "X_iris = dataset.data\n",
    "y_iris = dataset.target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "Xtrain, Xtest, ytrain, ytest = train_test_split(X_iris, y_iris,\n",
    "                                                random_state=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"color:red\">What's the use of random_state?</span>\n",
    "\n",
    "<span style=\"color:blue\">random_state is the seed used by the random number generator.</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the data arranged, we can follow our recipe to predict the labels.\n",
    "\n",
    "<span style=\"color:red\">Based on the folowing instructions, learn a Naive Bayes model (see GaussianNB) on the Iris dataset and store the predictions made on Xtest in a vector called y_model:</span>\n",
    "\n",
    "<span style=\"color:green\">from sklearn.naive_bayes import GaussianNB # 1. choose model class</span>\n",
    "\n",
    "<span style=\"color:green\">model = GaussianNB()                       # 2. instantiate model</span>\n",
    "\n",
    "<span style=\"color:green\">model.fit(Xtrain, ytrain)                  # 3. fit model to data</span>\n",
    "\n",
    "<span style=\"color:green\">y_model = model.predict(Xtest)             # 4. predict on new data</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "y_model: [0 1 1 0 2 1 2 0 0 2 1 0 2 1 1 0 1 1 0 0 1 1 2 0 2 1 0 0 1 2 1 2 1 2 2 0 1\n",
      " 0]\n",
      "ytest:   [0 1 1 0 2 1 2 0 0 2 1 0 2 1 1 0 1 1 0 0 1 1 1 0 2 1 0 0 1 2 1 2 1 2 2 0 1\n",
      " 0]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.datasets import load_iris\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "#load data\n",
    "dataset = load_iris()\n",
    "X_iris = dataset.data\n",
    "y_iris = dataset.target\n",
    "\n",
    "#create train-test datasets\n",
    "Xtrain, Xtest, ytrain, ytest = train_test_split(X_iris, y_iris,\n",
    "                                                random_state=1)\n",
    "#learn model\n",
    "model = GaussianNB() \n",
    "model.fit(Xtrain, ytrain)\n",
    "\n",
    "#compute the values predicted by the learnt model on Xtest\n",
    "y_model = model.predict(Xtest)\n",
    "\n",
    "#simple print to compare the results (predicted in y_model, and true in ytest)\n",
    "print(\"y_model:\",y_model)\n",
    "print(\"ytest:  \",ytest)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can use the accuracy_score utility to see the fraction of predicted labels that match their true value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[13,  0,  0],\n",
       "       [ 0, 15,  1],\n",
       "       [ 0,  0,  9]])"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "accuracy_score(ytest, y_model)\n",
    "\n",
    "#compute the confusion matrix\n",
    "from sklearn.metrics import confusion_matrix\n",
    "confusion_matrix(ytest, y_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"color:red\">What is accuracy? Give its formula and explain it.</span>\n",
    "\n",
    "<span style=\"color:blue\">Accuracy measures the proportion of instances taht are well classified by the learned classifier. It is defined as the ratio of correctly classified instances (#CCI) among the all instances (#I): Accuracy = #CCI/#I.</span>\n",
    "\n",
    "<span style=\"color:red\">What's the confusion matrix of the Naive Bayes model on the Iris dataset? Write a code to compute and visualize this matrix.</span>\n",
    "\n",
    "<span style=\"color:blue\">The confusion matrix C is KxK matrix (where K is the number of classes) in which rows correspond to the predicted classes and the columns to the actual classes. Cij indicates the number of instances that have been predicted in class i but actually belong to class j.\n",
    "\n",
    "<span style=\"color:red\">Redo the above steps with a k-NN model.</span>\n",
    "\n",
    "<span style=\"color:blue\">This one's for you!</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using sklearn to validate ML models and hyperparameters \n",
    "\n",
    "In the previous section, we saw the basic recipe for applying a supervised machine learning model:\n",
    "\n",
    "1. Choose a class of model\n",
    "2. Choose model hyperparameters\n",
    "3. Fit the model to the training data\n",
    "4. Use the model to predict labels for new data\n",
    "\n",
    "The first two pieces of this—the choice of model and choice of hyperparameters—are perhaps the most important part of using these tools and techniques effectively. In order to make an informed choice, we need a way to validate that our model and our hyperparameters are a good fit to the data. While this may sound simple, there are some pitfalls that you must avoid to do this effectively.\n",
    "\n",
    "## Thinking about Model Validation\n",
    "\n",
    "In principle, model validation is very simple: after choosing a model and its hyperparameters, we can estimate how effective it is by applying it to some of the training data and comparing the prediction to the known value.\n",
    "\n",
    "The following sections first show a naive approach to model validation and why it fails, before exploring the use of holdout sets and cross-validation for more robust model evaluation.\n",
    "\n",
    "### Model validation the wrong way\n",
    "\n",
    "Let's demonstrate the naive approach to validation using the Iris data, which we saw in the previous section. We will start by loading the data:\n",
    "\n",
    "<span style=\"color:red\">Load the Iris dataset storing the features in an array X and the targets in a vector y.</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      " 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
      " 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2\n",
      " 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2\n",
      " 2 2]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.datasets import load_iris\n",
    "\n",
    "#load data\n",
    "dataset = load_iris()\n",
    "X = dataset.data\n",
    "y = dataset.target\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we choose a model and hyperparameters. Here we'll use a k-neighbors classifier with n_neighbors=1. This is a very simple and intuitive model that says \"the label of an unknown point is the same as the label of its closest training point:\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "model = KNeighborsClassifier(n_neighbors=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we train the model, and use it to predict labels for data we already know:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X, y)\n",
    "y_model = model.predict(X)\n",
    "\n",
    "#compute accuray of the classifier\n",
    "from sklearn.metrics import accuracy_score\n",
    "accuracy_score(y, y_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we compute the fraction of correctly labeled points:\n",
    "\n",
    "<span style=\"color:red\">Compute the accuracy of y_model. What do you observe? Why?</span>\n",
    "\n",
    "<span style=\"color:blue\">The classifier is perfect. This is not surprising as the the model select the closest instance (1-nearest neighbor) to predict the target. In this case, the closes neighbor is the instance itself as the model is predicted on the training set.</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model validation the right way: Holdout sets\n",
    "\n",
    "So what can be done? A better sense of a model's performance can be found using what's known as a holdout set: that is, we hold back some subset of the data from the training of the model, and then use this holdout set to check the model performance. This splitting can be done using the train_test_split utility in Scikit-Learn:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "# split the data with 50% in each set\n",
    "X1, X2, y1, y2 = train_test_split(X, y, random_state=0,\n",
    "                                  train_size=0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"color:red\">Fit the model on the first subset (X1, y1).</span>\n",
    "\n",
    "<span style=\"color:red\">Predict the classes on the second subset (X2, y2) and compute the accuracy. What do you observe? What's the effect of random_state?</span>\n",
    "\n",
    "<span style=\"color:blue\">In this case, the accuracy is no longer maximum. Instances in the training and test set are no longer equal. As before, random_state is the seed used by the random number generator.</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9066666666666666"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X1, y1)\n",
    "y2_model = model.predict(X2)\n",
    "\n",
    "#compute accuray of the classifier\n",
    "from sklearn.metrics import accuracy_score\n",
    "accuracy_score(y2, y2_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Model validation via cross-validation\n",
    "One disadvantage of using a holdout set for model validation is that we have lost a portion of our data to the model training. In the above case, half the dataset does not contribute to the training of the model! This is not optimal, and can cause problems – especially if the initial set of training data is small.\n",
    "\n",
    "One way to address this is to use cross-validation; that is, to do a sequence of fits where each subset of the data is used both as a training set and as a validation set. For instance we can two validation trials, alternately using each half of the data as a holdout set. Using the split data from before, we could implement it like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.96, 0.9066666666666666)"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y2_model = model.fit(X1, y1).predict(X2)\n",
    "y1_model = model.fit(X2, y2).predict(X1)\n",
    "accuracy_score(y1, y1_model), accuracy_score(y2, y2_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What comes out are two accuracy scores, which we could combine (by, say, taking the mean) to get a better measure of the global model performance. This particular form of cross-validation is a two-fold cross-validation—that is, one in which we have split the data into two sets and used each in turn as a validation set.\n",
    "\n",
    "We could expand on this idea to use even more trials, and more folds in the data—for example we can split the data into five groups, and use each of them in turn to evaluate the model fit on the other 4/5 of the data. This would be rather tedious to do by hand, and so we can use Scikit-Learn's cross_val_score convenience routine to do it succinctly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.96666667, 0.96666667, 0.93333333, 0.93333333, 1.        ])"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "cross_val_score(model, X, y, cv=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Repeating the validation across different subsets of the data gives us an even better idea of the performance of the algorithm.\n",
    "\n",
    "Scikit-Learn implements a number of useful cross-validation schemes that are useful in particular situations; these are implemented via iterators in the cross_validation module. For example, we might wish to go to the extreme case in which our number of folds is equal to the number of data points: that is, we train on all points but one in each trial. This type of cross-validation is known as leave-one-out cross validation, and can be used as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "scores:  [1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.\n",
      " 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.\n",
      " 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 0. 1.\n",
      " 0. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 0. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.\n",
      " 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 0. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 0.\n",
      " 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 0. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.\n",
      " 1. 1. 1. 1. 1. 1.]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.96"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import LeaveOneOut\n",
    "scores = cross_val_score(model, X, y, cv=LeaveOneOut())\n",
    "print(\"scores: \", scores)\n",
    "np.average(scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"color:red\">What's the form of scores? Why? Compute the mean.</span>\n",
    "\n",
    "<span style=\"color:blue\">scores is a one-dimensional array with 150 elements contained a 1 at index i if the ith instance is correctly predicted and a 0 otherwise.</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Selecting the Best Model\n",
    "\n",
    "Now that we've seen the basics of validation and cross-validation, we will go into a litte more depth regarding model selection and selection of hyperparameters. These issues are some of the most important aspects of the practice of machine learning.\n",
    "\n",
    "Of core importance is the following question: if our estimator is underperforming, how should we move forward? There are several possible answers:\n",
    "\n",
    "* Use a more complicated/more flexible model\n",
    "* Use a less complicated/less flexible model\n",
    "* Gather more training samples\n",
    "* Gather more data to add features to each sample\n",
    "\n",
    "The answer to this question is often counter-intuitive. In particular, sometimes using a more complicated model will give worse results, and adding more training samples may not improve your results! The ability to determine what steps will improve your model is what separates the successful machine learning practitioners from the unsuccessful.\n",
    "\n",
    "### Validation curve\n",
    "\n",
    "Dans la suite, nous présentons une façon d'évaluer la meilleure complixité de modèle à choisir"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we imagine that we have some ability to tune the model complexity, we would expect the training score and validation score to behave as illustrated in [this figure](https://jakevdp.github.io/PythonDataScienceHandbook/figures/05.03-validation-curve.png)\n",
    "\n",
    "The diagram shown here is often called a *validation curve*, and we see the following essential features:\n",
    "\n",
    "- The training score is everywhere higher than the validation score. This is generally the case: the model will be a better fit to data it has seen than to data it has not seen.\n",
    "- For very low model complexity (a high-bias model), the training data is under-fit, which means that the model is a poor predictor both for the training data and for any previously unseen data.\n",
    "- For very high model complexity (a high-variance model), the training data is over-fit, which means that the model predicts the training data very well, but fails for any previously unseen data.\n",
    "- For some intermediate value, the validation curve has a maximum. This level of complexity indicates a suitable trade-off between bias and variance.\n",
    "\n",
    "The means of tuning the model complexity varies from model to model; when we discuss individual models in depth in later sections, we will see how each model allows for such tuning."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Validation curves in Scikit-Learn\n",
    "\n",
    "Let's look at an example of using cross-validation to compute the validation curve for a class of models.\n",
    "Here we will use a *polynomial regression* model: this is a generalized linear model in which the degree of the polynomial is a tunable parameter.\n",
    "For example, a degree-1 polynomial fits a straight line to the data; for model parameters $a$ and $b$:\n",
    "\n",
    "$$\n",
    "y = ax + b\n",
    "$$\n",
    "\n",
    "A degree-3 polynomial fits a cubic curve to the data; for model parameters $a, b, c, d$:\n",
    "\n",
    "$$\n",
    "y = ax^3 + bx^2 + cx + d\n",
    "$$\n",
    "\n",
    "We can generalize this to any number of polynomial features.\n",
    "In Scikit-Learn, we can implement this with a simple linear regression combined with the polynomial preprocessor.\n",
    "We will use a *pipeline* to string these operations together:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import PolynomialFeatures\n",
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn.pipeline import make_pipeline\n",
    "\n",
    "def PolynomialRegression(degree=2, **kwargs):\n",
    "    return make_pipeline(PolynomialFeatures(degree),\n",
    "                         LinearRegression(**kwargs))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"color:red\">What's the effect of PolynomialRegression? What's the effect of kwargs ?</span>\n",
    "\n",
    "\n",
    "<span style=\"color:blue\">PolynomialRegression creates a pipeline with the linear regression combinezd with a polynomial preprocessor. **kwargs allows one to pass keyworded variable length of arguments to a function.</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's create some data to which we will fit our model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def make_data(N, err=1.0, rseed=1):\n",
    "    # randomly sample the data\n",
    "    rng = np.random.RandomState(rseed)\n",
    "    X = rng.rand(N, 1) ** 2\n",
    "    y = 10 - 1. / (X.ravel() + 0.1)\n",
    "    if err > 0:\n",
    "        y += err * rng.randn(N)\n",
    "    return X, y\n",
    "\n",
    "X, y = make_data(40)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now visualize our data, along with polynomial fits of several degrees:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdd3RU1drA4d+ZZGaSmfSEAAlpEEINJfQqRaoIomJDLFcvKOi1fHotXBUV7NdGUbArERQQu17UEHoNhBZaAklIAklITyaTafv7YyDSAikzaexnrVmTnJmzz84seGefffZ5X0UIgSRJktR8qRq6A5IkSZJzyUAvSZLUzMlAL0mS1MzJQC9JktTMyUAvSZLUzMlAL0mS1MxVO9ArivKpoig5iqLsP2fbm4qiHFIUZa+iKKsVRfFxTjclSZKk2qrJiP5zYOwF2/4AugohugFHgGcc1C9JkiTJQaod6IUQ64H8C7atEUJYzvy6FWjjwL5JkiRJDuDqwLb+AXxT1YuKokwHpgPo9fpeHTt2dOChJUmSmq7icjPZJRUYzVbULioCPbX46jQoyvnvS0hIOC2EaFHT9h0S6BVFmQ1YgNiq3iOEWAIsAejdu7fYuXOnIw4tSZLUJFltgl/2nWRhXDKns0uICdAzc1g7bugZjNrl0pMtiqKk1eZYdQ70iqLcA0wARgqZOEeSJOmyLFYb3ydmsWhtMsdOl9E+0IP3buvBddGtca0iwNdVnQK9oihjgX8D1wghDI7pkiRJUvNjsthYtSuDRfHJnMgvp1NrLxZNjWFsl1aoVMqVG6iDagd6RVGWAcOAAEVRMoAXsK+y0QJ/KPbJpK1CiAec0E9JkqQmyWi28s2OE3y4LoWTRUa6t/HmhQldGNkpEOXCSXgnqXagF0LcfonNnziqI2azmYyMDIxGo6OavOq5ubnRpk0b1Gp1Q3dFkq46BpOFr7els3j9MXJLKugd5strN3VjaPuAegvwZzly1U2dZGRk4OnpSXh4eL1/CM2REIK8vDwyMjKIiIho6O5I0lWjxGjmq61pfLzhOPllJga28+f923rSv61fg8W2RhPojUajDPIOpCgK/v7+5ObmNnRXJOmqUGQw89nm43y2KZWicjPDOrTg4RGR9Arza+iuNZ5AD8gg72Dy85Qk58svM/HxhmN8uSWN0goLozq35OERkXRr03gywjSqQC9JktRU5JQY+Wj9MZZuTcdosTK+a2seGhFJp9ZeDd21i8jslVWYM2cOb731VkN345IOHTrEgAED0Gq1jbaPktRcZRWWM+fHAwx5fS2fbDzO2K6t+OOxoSycGtMogzzIEb1TWSwWXF0d/xH7+fnx/vvv8/333zu8bUmSLu1EvoFF8SmsTDiBEHBjTDAzh0USHqBv6K5dkRzRn2PevHlERUUxePBgDh8+DEBKSgpjx46lV69eDBkyhEOHDlVu79+/P9HR0fznP//Bw8MDgPj4eIYMGcLEiRPp3LkzAEuXLqVv37706NGDGTNmYLVaAVizZg0DBgwgJiaGKVOmUFpaWq1+BgYG0qdPH7lsUpLqwbHcUp5YsYdhb8WzKiGDW/uEEP/kMN64uXuTCPLQSEf0L/50gKSsYoe22TnIixeu71Ll6wkJCSxfvpzExEQsFgsxMTH06tWL6dOn8+GHH9K+fXu2bdvGzJkziYuL45FHHuGRRx7h9ttv58MPPzyvrV27drF//34iIiI4ePAg33zzDZs2bUKtVjNz5kxiY2MZP348c+fO5c8//0Sv1/P666/z9ttv8/zzz/PYY4+xdu3ai/p422238fTTTzv0c5Ek6dKOZJewIC6Zn/dmoXZRcdeAMGYMbUcrb7eG7lqNNcpA3xA2bNjA5MmT0el0AEycOBGj0cjmzZuZMmVK5fsqKioA2LJlS+XUyR133METTzxR+Z6+fftWrl3/66+/SEhIoE+fPgCUl5cTGBjI1q1bSUpKYtCgQQCYTCYGDBgAwDvvvOPkv1aSpKrszyxi4dpkftt/Cp3GhX8Obcv9g9vSwlPb0F2rtUYZ6C838q5PNpsNHx8fEhMTa7SfXv/36ZwQgrvvvptXX331vPf89NNPjBo1imXLll20vxzRS1L9SzxRyPy/jvLXoRw8ta48PCKSfwyKwFevaeiu1Zmcoz9j6NChfP/995SXl1NSUsJPP/2ETqcjIiKCFStWAPagvWfPHgD69+/PqlWrAFi+fHmV7Y4cOZKVK1eSk5MDQH5+PmlpafTv359NmzaRnJwMQFlZGUeOHAHsI/rExMSLHjLIS5Lj7UjNZ9on27hh4SYS0gv4v1FRbHx6BP83ukOzCPLQSEf0DSEmJoZbb72V7t27V17sBIiNjeXBBx9k7ty5mM1mbrvtNrp37867777LnXfeybx58xg7dize3t6XbLdz587MnTuX0aNHY7PZUKvVLFy4kP79+/P5559z++23V04HzZ07l6ioqCv29dSpU/Tu3Zvi4mJUKhXvvvsuSUlJeHk1zqVdktTYCCHYnJLH+38dZdvxfAI8NDw9riN39g/DQ9v8wqLSECnkL1V45ODBg3Tq1Kne+1JbBoMBd3d3FEVh+fLlLFu2jB9++KGhu3WRpva5SpIzCSGIP5zL/Lij7EovpKWXlhlD23F731DcNS4N3b0rUhQlQQjRu6b7Nb+vrnqSkJDAQw89hBACHx8fPv3004bukiRJVbDZBH8czGZBXDL7MosI9nHn5Ru6MqVXG9zUjT/A15UM9LU0ZMiQyvl6SZIaJ6tN8Nv+kyyIS+bQqRLC/HW8cVM3bugZjMb16rlEKQO9JEnNjsVq48c9WSxcm0xKbhntWuh559buXN8tyGnl+hozGeglSWo2TBYbq3dnsCg+hbQ8Ax1bebLwjhjGdm2Fi5PL9Tndkf/Veter76tNkmopNjaW8PBwVCoV4eHhxMbGNnSXpDOMZitfbUll+FvxPLVqH15uapZM68Wv/xrCdd1aN+0gLwRsng9f31rrJuSIXpKqITY2lunTp2MwGABIS0tj+vTpAEydOrUhu3ZVKzdZ+Xp7OkvWp5BdXEFMqA9zJ3dlWFSL5lGPoaIUfnwYDnwHnSYCS2vVjBzRV6Expyn+4Ycf6NatGz169KB3795s3LixobvU7M2ePbsyyJ9lMBiYPXt2A/Xo6lZaYeGD+BQGvx7Hyz8nERGg5+v7+7HqwYEM71B/RbedKvcIfDQCkr6HkS/AlC9q3ZQc0TuRs9IUjxw5kokTJ6IoCnv37uWWW26pzKopOUd6enqNtkvOUVRu5ovNqXy66TiFBjNDo+zl+vqEN3y5Pofa/519JO/qBtO+h7bX1Kk5OaI/R1NJU+zh4VE5YikrK2seo5dGLjQ0tEbbJccqKDPx1v8OM/i1ON7+4wi9w3z5ftYgvvxH3+YV5M3l8Mv/wcp7IbAzPLChzkEeajCiVxTlU2ACkCOE6Hpmmx/wDRAOpAK3CCEK6tyr356GU/vq3Mx5WkXDuNeqfLmppSlevXo1zzzzDDk5Ofzyyy+O/ayki8ybN++8OXoAnU7HvHnzGrBXDSM2NpbZs2eTnp5OaGgo8+bNc9p1itySCj7ecIyvtqZhMFkZ17UVD42IpEvQpVOONGmn9sHK++D0YRjwkH26xtUxuXZqMq/wObAA+PKcbU8DfwkhXlMU5ekzvz/lkJ7Vs6aWpnjy5MlMnjyZ9evX89xzz/Hnn3/W9SOQLuNsIKuvANdY1ddF6VNFRj5cl8Ky7emYrTau7x7ErOGRRLX0dNgxGg2bDbYugr9eBHc/mLYa2o1w6CGqHeiFEOsVRQm/YPMkYNiZn78A4nFEoL/MyLs+NYU0xUOHDuXYsWOcPn2agICAGvVTqpmpU6dedYH9Qpe7KO2IzyajwMAH8Sms2JmBTQgm9wxm5vBIIpxUyUkIgSk1lfLEPZjS0zBnZiLKjShqNa4tAtBGdUDXtw+akBCnHJ+CVPjxX3B8HXS4DibOB72/448jhKj2A/sUzf5zfi8852fl3N8vse90YCewMzQ0VFwoKSnpom31KSEhQURHRwuDwSCKi4tFZGSkePPNN8WAAQPEt99+K4QQwmazicTERCGEEOPHjxfLly8XQgixePFiodfrhRBCrF27Vlx33XWV7R44cEBERkaK7OxsIYQQeXl5IjU1VeTk5IiQkBBx9OhRIYQQpaWl4vDhw9Xq69GjR4XNZqvsd1BQUOXvF2roz1VqXhRFEcBFD0VR6tTu8dxS8cS3iaLdM7+IyGd/Ec98t1ek55U5qNfns1mtomznTpH1/AviyJChIqlDR/ujU2dxZPhwkTJhgjg6erQ42KNn5WvHptwiin75RdgslovaW7p0qQgLCxOKooiwsDCxdOnSK3fCahFiywdCzG0lxLxgIXZ+JkQV/4fPBewUNYjZZx8OWxIiROU/gqpeXwIsAXv2Skcd11GaUpriVatW8eWXX6JWq3F3d+ebb76RF2SlehEaGkpaWtolt9dGco69XN+Pe+zl+u7sH8aMa9rS2tu9rl29iLWoiIJvvqVw+XLMWVko7u54XHMN+gED0PXuhSY0FOWcOszCZsN0/Dil8esoXLWKzMf/D23njwl65RXcOnYEajmVlXvYvqLmxDaIHAXXvwvebRz+956nJt8KXDyiPwy0PvNza+Bwddrp1avXRd9UTW3kWVZWVjmKXrZsmZg4cWID9+jSmtrnKjVuS5cuFTqd7rzRvE6nq94o9hwHMovEg0t3ivCnfxadnvtNzPslSWQXlzulzxUnMsTJefPEwZ4xIqlDR5F6zz2i8IcfhLW0tNpt2KxWUfjTz+LwwEEiqWu0KFixQgghRFhY2CXPcMLCwi7RkTIh/npZiJcChHgtTIjE5dUaxZ+LBhrR/wjcDbx25rnxJWR3EpmmWLoa1fWi9N6MQt7/K5k/D2bjoXVl5rB23De4LX5OqORkzs7m9MJFFK5aBYqC93Xj8bv33srReE0oKhXeE65DP2ggWU88ycn/PIc5K6t691cIAQd/gv89C0UnIPoWGDMPPAJr+6fVWLULjyiKsgz7hdcAIBt4Afge+BYIBdKwL6/Mv1JbzaHwSFMhP1epMdiZms/8uGTWHcnF213NPwZFcM/AcLx16ivvXEOWggLylnxEwddfI2w2fKdMwX/6P1G3auWQ9oXFwsk5cyhauYovzGZeP5Zy0XvCwsJITU21T9P89hQcWwstu8L4NyFsYK2P7fTCI0KI26t4aWRNDypJUvMnhGDLsTzm/5XMlmN5+Ok1/HtsB6b1D8PTrfYBvqp1/NbSMvK/+Jz8Tz/DVl6O98SJBDw0C00bx85/K66utH7pJTCbufuHHzkSEMDq06crX9fpdLz94pP2efjdS0HjCePegN73gUvDJCOQKRAkSXIoIQTrjuSyIC6ZnWkFBHpq+c91nbijXyg6Td1CzqUufs6aPp0W27cTvn0H1oICPEeNosUj/0IbGemIP+eSFJWK1nPnYs7M4uXERAo8PFiXlkbXyDYsm9WPLidegnQb9HsAhjzhnCWTNSADvSRJDiGE4M+DOSyIO8qejCKCvN14aVIXbukd4rByfeeu41eACV5ePOwfQJv/rUHbvz+Bjz+Ge7duDjnWlShqNcHvvsPxG2/ik7AQwpfcjyphCRT8Ad1ugeGzwTesXvpyJTLQS5JUJzab4Lf9p1iwNpmDJ4sJ9dPx2o3R3BjTxuHl+s5e5Byi1/NYQAs6urmRZDTyz4wTbPgsqd6XGbu6Q+ubOnLig/Wc/mAvgbdcA9c8Ba3r58umumSgr8KcOXPw8PA4L7VBYxEfH8+kSZMq0yzceOONPP/88w3cK+lqY7Ha+HnvSRasTSY5p5S2AXr+O6U7k3o4r1zfmLBwbrVa6KfTk24y8X9ZmfxeUkJoWFj9Bvm8FNj2Iexeioe5HO+e0eTtUeHd+0W0rZ03ZVRbMtA7kbPSFIO9OPnPP//slLYl6XLMVhurd2eyaG0yqXkGOrT0ZP7tPRkf7bxKToYdO8hdtIi3tVryrK68nH2KlYWFmKnH5HJCwLF42PoBHP0fuGig680w+FECXQMpGT2G7DfeIHTJEqccPseQU+t9ZZriczSVNMWS1BAqLFaWbk1j2Jvx/HvlXvRaVz68sxe/PTKE67sHOTzICyEo27yZtGl3kTbtLiqOJhP41FOcePYZNnt7Y1EUwsLCWLJkiXNzEJXm2kv5LewHX90AWbtg2DPw2AGY/AG06ICrry8BDz5I2foNlG5wXCGgk6UnWXFkBfevuZ9rV1xb63Ya5Yj+9e2vcyjfsYU0Ovp15Km+Vedba2ppirds2UL37t0JCgrirbfeokuXLg79vCTprHKTlWXb01l8plxfjxAfXr6hC8M7BCIQFFcUUVhRSJGpCIPZQLmlHKPFSLml/LyHxWbBKqzYhA2bsGEVVoQQWIV94OOqciX5SDI712+hT2o5t+TrCDcoVPh6kHPfaAzjBuHm4UWwq44V41agU+twd3VH56qjxFSCXq1HpTho7Gq12Ne+7/oSDv8GNjO06QuTFkH0zeCqvWgX3zunUhAbS+78+egHD6rVVFKOIYd9uftIyElgc+ZmUorsa/RDPEOY0X0GD/FQrf6cRhnoG0JTSlMcExNDWloaHh4e/Prrr9xwww0cPXrUER+DJCGEoLCikLTCLL7ZvZ/fDx2mzJpPy5YVDOhqQeVi4O2DRTyfWEhxRTGCK990qVJUaFQaFEXBRXGpfFYpqsrg7J6az+A9Jh5IUuFdrie1BSwarmJT53LMrnGQEHfZYygoeGo88dR44qXxwkvrZX++8OdztnmoPfDQeOCp8USruELaJnt1p4M/giEPdP7Qbwb0nAaBl7+jVqXR4P/Pf3JqzhzKNm/G48z/7Ut9viXmEk6UnOB40XGOFx3nWOEx9p3eR7YhGwC1Sk3vlr2Z3H4yg4MH09a7LYqiNK9Af7mRd31qrGmKvby8KreNHz+emTNnyjTFdVSfxTQaA4PZQEZpBhklGZwoOcGJkhNklGSQUZrBydKTmGymv9/sB24ouLj5YVFa4KP2IcgjCG+tNz5aH3y0PnhrvfHWeqNX63F3db/ooVapLznCtRQUUPLnnyQvXoJHhgmLEKwvLeXLggK2HzbAJghrG0bi/kTKLeUYLIbKs4Zzfy41l1JiKqHYVGx/VBRTYiohxZBS+ft5f9MlaITAw2bD0wYerVvg4dENT+8wPLQqPFJ/QHvidzQuGjQuGtQqtf1nlQaVorKfpWBDdLHQwd+LA2/NYZf+RkrNpZSZyyg1l3K6/DQ5hhxyDbkYrcbK47ooLrTxbENMYAzRLaKJDoimk38ntC4XnzXUVqMM9A1h6NCh3HPPPTzzzDNYLBZ++uknZsyYQUREBCtWrGDKlCkIIdi7dy/du3enf//+rFq1iltvvZXly5dX2e7IkSOZNGkSjz32GIGBgeTn51NSUkL//v2ZNWsWycnJREZGUlZWRmZmJlFRUVcc0Z86dYqWLVuiKArbt2/HZrPh79+wN2Q0ZfVVTKO+CSE4XX6alKIUUgpTOFZ4jJSiFNKK0zhdfvq893pqPAnSt0ExBWPKj8Bo9KBLy1Du6NWVIRGRBOgCUKsck67AnJVFSdxaSv78E8OOHWC1km+z8eHp0/xYXMTpM9ew7H8EpB9Lx8fNBx986nRco8VoD/p5Ryk5vpbiE1spyd5PiTBT6qqhxD+CUr8wSt19KbYaKDWVcro4jRJzCWXmMiqsFVhsliseZ3wPG/f8Vczvvy/kZIgOvVqPh9oDf3d/uvp3JTAkkBa6FgR5BNHWuy0hniFoXByf6+dcMtCf0ZTSFK9cuZIPPvgAV1dX3N3dWb58uUxTXAfOLqZRH4oqijiUf4ijBUdJLkzmWNExUgrto9mzvDReRPpEMiR4CCGeIZUPd1ULvt2Wz1dbUikzWRnTpSUPj2hP1+C6l+sTQmDOzKI8MRHDtq2UbduO+cxaeE27dvjffz+eo0bRpVs0VeXdqnNd3qIMSN2EW5r9EZiXbN/uHQJRN0L70RAxFDS6KzZlEzbMNjMmqwmT1YTZZsYqrOdPQ403cHrzDSwqHEPw069esc36UO2kZo7UHJKaGQwG3N3dURSF5cuXs2zZMn74ofEl72xqn2tDUKlUlwwyiqJgs9kaoEeXJqxWbBYLuSUnOXL6EEdyD5KSd4TkgqNkGk9hVYFVBR7uPoT7t6OdTyTtfNpVPvzd/M8bEGQXG1m87hhfb0+jwmJjQrcgHhoeSYdWNS/XJ6xWLKdOYTpxAlN6OqbUNIwHkzAmHcRWVASAytMTXZ8+6Pv1RT9kCNq2bSv3Dw8Pv2See0VR+Oqrr6r/hVtRAif3QtZuOJkIJ7ZD4Zl2td4Q2h/CB0P7UdCiIzhpgHTyxRcpWvUdkfFrcfVzXPFypyc1k84n0xQ3H5cqpuGiQLf2bezZBw15UFEK5jIwGcBsAFOZ/dlcDsIGNisI6wXPAlQqUKlB5Wp/uLhW/iwUVyxlNsyFFVhKLFiKyjEXGbEUlGApKsNmqMBUWoq5tBTKynEx/z2l0eLM49KX+/LsD9ddKGo1ikZDvkZNoVqDotGQV2bgZEkFZncf2ru48prWhY7tgvDcoUPZo+Gkxv4++0ON4uKKMJkQpgpsxgpERQU2oxFrQQHW/DwseflYCwvttU/PUDQatFFReI0Zg1vnTrh1jcatU0cUl0unQrhU8XVFUXjggQcuHeTN5fablk4fhtwjcPoIZO+H00fh7MVhr2AIjoH+D9ozRrbsCirHpGK4Er+pUylctpyi1avxv+++ejnm5chAX0tDhgxhz549Dd0NqTYM+fZRXlEmFGfy62O9OLA5j9Z6G4F6hQCdCj93BSiChX0v35armz1wKyr7Q+UCisuZZxXYrAirBXOxFeNpQUUBmIpUmIpVmEpcsVnOXw4oVIIKvaBQD3nuKgo9FcoDwKgBnYvAT4EAlQuBLhpaqnVoNe7g6oZw0SIUDQI1KGqEokbgihAuCJuCsCqUllvYdfQUxS461F4WVMW5qIpzsWLFYCpArdNhM5sQJvOZwG5/YLPZvzC02jMPDSqtGy6+vmjCw3GP6YWLny/qVq3RhIWiCQ3FtWXLKoP6pVyY575zuxBeefZRJl7T054BsijDnsu9KMNeZ7UgjcqAjmLPKRPY2X4DU1BPCOpRr/neL6SNjMS9Rw8KV6/G7x//aPCpVRnopeZJCChMh5wk+yjv9BHIS7Y/G/LOe2tnFy0h3VqyP72AhJPlVLjoiBl0Ld36DQOdn/2h9eLnNWt57e33OZqahV+rNvxnzlym3jntokNbCgow7NyJce8+jEkHMB44gPXM9AWKgjooCHXnMGytvcnysXLEvZA9ShZ7lVOUuAtQFMLcAojWBRGtDaS/xpcolRsakwGMxfbpiYoSqCi2P4w5f/9+uYuFOmjvD+VWF4qMVkqMVsrMUGYSlLoUETNyLKh19rlqtf7Msw7hqkPRuIPLuWcmZ39W/32WAmCrgPLDcDzJPsI/e3ZjM198NnT2ubwADHlMLctj6kNaMPiBrRDS5sCX5/Tfo5W95F5QDHS/HQLaQ0AH8G8HaseXHqwr7xtu4NScORj3H8A9umuD9kUGeqnpE8I+Qs9MgJN7ICvR/mws/Ps9+hYQEAUdJ9iffcPBOxi82oA+AE9FYQAwoIpDxMbGMv3hOZVTCzkp6Uyf8QAoKm4dMwbD9u0Ydu7EsGMHFUfPXOxTq3Fr3x7P0aOxdgjneCuFBH0uiUVJJOXto9xSDoCP1oeuAV2ZFnBj5fI6b20tLoQKARZj5RdBamYWP2w/zIHjGfi5VDAszI3Ny97FQzHgpVXw0iro1aDXKOhVFZB//OLpKQROG4uq7V8kuPva16v7hpNs9OaPPQUcO2XE5ubHuNv+ybU33WOfhrnETUqNmdf4cWS/8gpF33/f4IFeXoxt5prl52qz2Ufq6VsgbbP9ueSk/TUXjf0UvnV3++l7y672kZ+7b50Oee7FQhXQzc2doR56rvX1I1Jln35R6XS49+qFe+9eFHYKZq9/KbsL9pOYm0hasX1fV5Urnfw6ER0QTXSLaLoFdCPEM8Shp/b7MoqYH3eUNUn2cn13DQjjvsER+Htoq7zoWVkR6VxnvzhMZfZnm8V+x6jNfObnM89nf4a/p64qp7HOPKvU558puLrbr1+c48JlrmDPY+P0FAdOEhsbS8mcOXQD7jSbedkB92bIi7FSg6i3G41KcyHlLzj6B6TEQfmZipWeQfYLbaEDIKQvtOgEro5fk5yfkcF1nl5c46FnsN4DHxcXLEKQWF6OzzP/JqtTALu8C9mdt4c9uV9SfLwYjoOv1pfugd25sf2N9AzsSWf/zg69EeZcCWkFzI87SvzhXLzcXHlkZHvuHRSOj+7vz+NSFz2rTAqmKPYpkXqaFmkOy1zPOvulNVCl4prgNrTMymrQezPkiL4KjTlN8ZtvvklsbCxgz5B58OBBcnNz8bvEMi5nfq5OHYEJATkHIekHe6bArERAgC4AIq+FtsMgbAD4hDltiZy1sNB+Y8+aNRSsXYtaUThtsbDRbGCHn4V9US6ou/ugbaPFIuxz422929IzsCc9AnvQo0UPwrycnz5367E85scdZVNyHr46NfcPacu0AWF4VVGur7HeBdxUlrlWx9kzJzdFYWNke34oKuLlnOxLnznVQG1H9DLQV8ERgd6ZaYrP+umnn3jnnXeIi7t0HhBnfq41mgaojsrg/j0cWG2/cIoCbfrY1z1HXgute1x0yu9I1sJCiv/4g5L/raFs61awWMhRBHH6crZEQXqMG+oW9hGyzWQjTBPG2Oix9AjsQfcW3Ws3t36B6gRiIQQbjp5mQVwy21PzCfDQMmNoW+7oF4pe2zRP1B3+76kBnful9XbrIHrrdAxLSUbU8UtLTt04wLx58/jiiy8IDAwkJCSEXr16kZKSwqxZs8jNzUWn0/HRRx/RsWNHUlJSmDp1KmVlZUyaNIl3332X0tJS4uPjee655/D19eXQoUMcOXKEpUuX8v7772MymejXrx+LFi3CxcWFNWvW8MILL1BRUUG7du347LPPKtMdV9eyZcu4/faq6rY719lqP9XdXqWSbNizDBK/tq+LVlQQNgj6TtUYqGEAACAASURBVIdOE8GzpQN6WzWbyURpfDxFP/5Iafw6sFgwtPRmWx8Pfg8v4niYCyhemAvMGI4aMPyRh1eJF3NmzuGuO+9yaF+ulI5BCEHcoRzej0tmz4lCWnm5Mef6ztzWN9Rh5foaSo2mlRq5c+/NWFNawlgvL3q6u3M6sGGWfDbKQH/qlVeoOOjYNMXaTh1p9eyzVb7e1NIUg33+8vfff2fBggWO+6Bq4FI3Gp3dfkVWCyT/Abu+giO/25fhhfSD8W/VS3AXQmDYtYuTq5ZhXBOHS2k5JZ6urIuxsaGLC2mtDJhPWihMKsTwuwHDUQPmPPsFx7CwMI6lHnNKv6qep/4Pft1GMD8umaSTxbTxdeeVydHc1CsYrWvTDvBnXbiWvjFNK9XUuV9a60vLMAvBKB9fujbQl5ZDAr2iKI8B92O/g2EfcK8Qwnj5vRqXppSm+KyffvqJQYMGXXJuvj7UagRWmgsJn8GOT6D0lH3Z44BZ9jSwLa6c56cuzFYzB/fEkfPdt3jG78brdDlGNeyIUtjeXYdLnxh6tI7hmcAeRAdE4+Xmdck54xqfsdTARW0rKnQdB2MacCsPxu4iIkDPmzd344aewaidVK6vIU2dOrVJBvYLXfiltc9mY0rbtsQ00N9W50CvKEow8C+gsxCiXFGUb4HbgM9r2+blRt71qbGmKT5r+fLlDTZtAzUcgZ3aD9s+gL0rwFphn2/v9V+IGmO/+cYJCo2FJOYmciBlC5Y/1tF26wkiM220Bo62c2PP9T3xHT2Wa8L6cZ9PJC4X3B5fpzOWWqo8psoFfedheA+4BbVfMBSd5L3bejChm+MrOUnOce6XVt7nn5Pz2uuYMjLRtAmu/84IIer0AIKBE4Af9i+On4HRl9unV69e4kJJSUkXbatPCQkJIjo6WhgMBlFcXCwiIyPFm2++KQYMGCC+/fZbIYQQNptNJCYmCiGEGD9+vFi+fLkQQojFixcLvV4vhBBi7dq14rrrrqts98CBAyIyMlJkZ2cLIYTIy8sTqampIicnR4SEhIijR48KIYQoLS0Vhw8frnZ/CwsLha+vrygtLb3s+xr6cxWpm4T48gYhXvASYm4rIX5+XIic6v+d1WWz2URKYYr47sh34rmNz4nJ314n7nu6s/hiQiext1NHkdSho9h67QCx+Y1/i6zj+6vV5tKlS4VOpxPYz1QFIHQ6nVi6dKnD+3/W518uFX59rhfBMz4WYU/9LFrf857wiR4uvvzKeceUnM+Yckwkdego8r/+uk7tADtFLeJ0nUf0QohMRVHeAtKBcmCNEGLNhe9TFGU6MB2cOyKqraaUphhg9erVjB49+ryzh0bjbBHl9W/aK/boW8C1c6DXPXW+cekso8XIgbwD7M7ZTWJOInty91BoLKBjBlyb5MqNB61oy23Y/H3wvWsifpNvxK1Dhxodoz7njI1mK8u3p7M4oxWeI2ZAXio5q14i0JzNm010nlr6myYiHHVoKKXx6/BtgLPwOi+vVBTFF1gF3AoUAiuAlUKIpVXt0xSWV16JTFNchdRN8NeLcGKb/WamQY9Ar7vrfNNNUUURiTmJJOQksCt7FwfyDlQWgehtCmLcET0dtp9Ck12A4u6O56hr8Z44Cf2A/jVKrlXfyiosfL0tnSUbjpFbUkGfcF8eHtGeIe0DGjwRluRYp156icLvf6DDtq0o6tpNVzbk8sprgeNCiNwzHfkOGAhUGeibA5mm+ALZSfYAf+R3e4C/7m3oeWet85Nkl2WzK2cXCdkJ7MrZRXJBMgKBq8qVLv5d+EfIFPoeMOO/bh/mvQdApULfvz/ej0/E89prUTWCM53LrYcvMZr5cksaH284RoHBzKBIf+bf3pP+bWWlsOZK178/BV8vo3zfPnQxMfV6bEcE+nSgv6IoOuxTNyOBnZffpemTaYrPKM6CuLn2NfBaL/sUTb8HajSCF0KQXpLOzlM7K4N7ZmkmAO6u7vRo0YPRPUbTy7srEQfyKf91DaXrloPZjCoqisAnn8RrwgTULRsuLe2FqloPX25VKGndm882HafYaGFYhxY8PKI9vcIcM6UlNV66Pn1AUSjburXpBXohxDZFUVYCuwALsBtYUsu25OmqA9V1Wu6yLCbYugjWvWFPcjVgFgz5P3tK32o4VXaK7ae2s+3kNrad3Ea2IRsAPzc/egb25I6Od9CrZS/a68MxbtxMyRe/UxL/EbkGAy4tAvC78068J03ErWNH5/2NdXDheniVuxea3pOZu0cDSUcZ3bklD42IpFubutVBlZoOV19ftJ06Yti6DWbOrN9jO6IRIcQLwAt1acPNzY28vDz8/f1lsHcAIQR5eXm4ubk5vvGUOPj135B3FDqMhzGvgF/EZXcpNBay/dT2yuCeWpwK2FP09m3Vl36t+9G7VW8ivCIQZjNlmzZR/MUXHP8rDltZGS6+vnhffz1e48ah69O7Xufda5Mb5ux6eBe9L159b8SjxzgUtYayw5tY/8GzdGrtVR9dlxoZfb/+FCxdis1oROWM/5tVaDR3xrZp04aMjAxyc3MbuivNhpubG23atHFcgyXZ8Nu/7blofCPgjhUQNfqSb7UJG4fyD7EhYwMbMjew7/Q+bMKGzlVH71a9uTnqZvq37k973/aoFBW2sjJKN27iZNxiSuLWYispQeXtjee4sXiNG4e+Xz8UJ+cNupQrpSSoSmjHbhQHD8Cz+2hQuVCWtI6iLd8S7OlCp9av1UvfpcZHP6A/+Z99Rvnu3egHVFX9wPEaTVIzqRETAvYsh9+fttfqHPoEDPwXqM8fkZSaStmUtYkNGRvYmLmRPGMeCgpdA7oyOHgwA4MG0iWgC2qVfcWBOSeH0rXxlMT9hWHLVoTJhIu3Nx7Dh+M1fhz6/v1RNI5POVwTNU20lZ5n4IN1yXy7Ix2LxUrp/r8o3roCS+GpJp1bXXIMa0kJR/r2I2DWLFo8NKvG+8ukZpJzFGXAT4/a89KE9IOJC85LVVBoLGTtibX8mf4nW7K2YLaZ8dJ4MShoEEPaDGFg0ED83e0rSYTNhjHpIEUbN1KyNg7jnr0AqNu0wff22/AYMRJdr5gGGblXpbqJ21JyS1m0NoXvEzNxURTu6BdOy4K9vLXiRwqK7Olpm2reFslxXDw90XboQPmuhHo9buP5HyXViNNzip8dxf/6pD3h2NjXoe8/QeVCgbGAP9L+4I+0P9hxagdWYSVIH8RtHW9jZOhIurfojuuZGqLm7GwKf1tN2caNlG3ZgrWgAAC3rl1p8ci/8BgxEm1U+0Z7XeZKaRAOnyphwdpkftmbhcZVxd0DwplxTVtaerkBXXno3jvqucdSY6eL6UnR9z8gLJZ6G9TIQN8E1XbeuNrKC+Dnx+HAd/Z0wTcsosKrNfHpf/Jzys9szNyIRVgI9wrn3q73cm3YtXT262wvEGE0Yti0lbJNmyjbtLGyfqpLQAAeQ4egHzQI/cCBuAYE1L2f9aCqxG2z/vMaM77ayf8OZKPXuDB9aDvuHxJBgEfTqmsq1T/3mF4UfL2MiiNHcOvcuV6OKefomyCnFmhI3QTfTYfSU4hrnmZ/p1GsSv6eNalrKDGX0MK9Bde1vY4JbScQ5Wufwqk4coSyjZso27QJw86dCJMJRaNB17uXPbAPHow2KqrRjtqv5Lyzpx5D6XDToxwuUePp5sq9A8O5d1AEvvqGvZYgNR3mrCySR4yk5ezZ+E27s0b7NvkKU1L1OaXkms1qXxO/7nXK/cL5rf9dLM/eysH8g7i7ujMydCTXt7uefq36IQoKKdu8mbKNmyjdvAlr7mkAtO0j0Q8chH7wIHS9e6Nyr59ao/Vh27E8FqxNZsPR0/jo1Nw/OIJpA8LxdndO5k2peTs6bDi6mJ4Ev/12jfaTF2OvIlXNG9c6L31ZHnx3P+lp61nWoS8/iFJKDn5GpE8ks/vN5rqQMagOHKVs+SbSN/4XY1ISAC4+PugHDrSP2gcNRN2qVV3+rEZHCMGm5DzejzvK9uP5BHhoeGZcR+7sH9Zky/VJjYN7zx6Un1mMUB/kv9YmaN68edx7772YzebztpeUlBAbG1uzefrMBJJW3c2nrkb+CAlGZcllVOgobvUaTkRSAWXvbyBr65vYDAZwccG9Zw9aPPoI+kGDcevSGcWJ9VsbihCCtYdzmB+XzO70Qlp6aXl+Qmdu7xuKu6bxJkiTmg73rtGU/PY7lvx8XOuhcJCcummiAgICyMvLu2h7Tebpd6x7kY8PxrLZXYuHyo379GMYleaFdf1WKg4eBEAdHIx+8GD0gweh798fF09PR/4ZjYrNJliTlM2CtUfZn1lMsI87Dw5rx5TebZpNuT6pcTDs2EHatLsIWfwhHtdcU+395NTNVSY/P/+S26tT5m5fdiLv/fUI20x5dC92452jXQjdnY01axUGRcG9Z08Cn3wSj+HD0USEN9mLqNVltQl+3XeSBXHJHM4uIcxfxxs3dWNyTPMs1yc1PLfOnUGlonzvvhoF+tqSgb6Jqk2Zu5TCFObvfJvEpHWM2m9l5mE33LONoD6A+8CBeM6cicewYU1m6WNdWaw2fkjMYmF8Msdyy4gM9ODdW3swoVtrXGWAl5xIpdejbdeW8v376uV4MtA3UTUpzF1oLOT9hHdJ+X0V43ZZmZ5iQ0HBPaYz3g9ej9fYsbj4OCeLotNv7KoFk8XGd7syWBSfQnq+gY6tPFl4RwzjurZCJeuxSvXELbobpWvX1kvWXhnom6jqlLmz2qx8tyeWA5+9y7Dt5UwpBJW7wO+O6/G+919oQkIc1p9LBXTAuTd21ZDRbOXbnSf4MD6FrCIj3dp489yE3ozsGCgDvFTv3KO7UvTdd5gzM9E4MvngJciLsc1U0vHtrH3nSfqsz8HTCEpLC0HddXg++w1KK8fmcL/wTl2wn124u7vX+YKxIxhM9nJ9i9fby/X1CvPl4RGRXBPVotlff5Aar/J9+0idcgvB77+H1+hLZ4G9kLwYKwFQXlLIX689TOufdjLCBIboVoS1OoAuugtMXQF6x8+/X1hkA+w1dS/cdlZ1Lhg7QmmFhS+3pPLJhuPklZkY0Naf927rwYC2suaB1PC07duDSkXFoUNQzUBfWzLQNxPCauXAVwsoW/gR7UqspPcKpte4ngQkL4F2I+GWL0Hr4ZRj1zRwX+6CsSMUlZv5fFMqn246TlG5maFRLfjXiEh6hzt/vbIkVZfKzQ1N2wiMSQedfiwZ6JuBst27SXr6ETzScskL0cDcJxgjDttL/UXfApMWgqvzcrFUtQLI39+f8vLyal0wdoT8MhOfbDzGl5vTKKmwcG2nljw8IpLuIbJcn9Q4uXXqjGHHDqcfR64ha8JsZWUcf/E/pN1xBxX5ucTd35NBP8bTz7zHHuT7PQiTFzs1yIN9BZBOpztvm06n47333mPJkiWEhYWhKAphYWFOKbyRU2LklV8PMvj1OBbFpzAkKoBf/zWEj+/uLYO81Ki5deyI5dQpLGfSdzuLHNE3UYaEBI4//giq7Dz+7K0m5ImnmBl9K8rPj8DupTDoUbh2DtTDXPSVVgA5a4XNyaJyFq87xrLt6ZitNiZ2D2LW8Ejat2y+d+9KzYtbJ/vCiIpDh3B1YmlBGeibGGG1cvqDD8lZuIAcb3hnjIX9myuYu9+GcmwW7F0OQ/8Nw5+tlyB/1tSpU+ttyeSJfAMfrEth5c4MbEIwuWcwM4dHEhGgr5fjS5KjaDvaA73x4CGn1pCVgb4JsZw+zYnHHsW4I4GNXRTebVnGkSUZKBU2XH+cCZ1VMHw2XPPvhu6qUxw/XcbCtcms3m0v1zeldxseuKYdIX66K+8sSY2Qq58fri1bYjzk3AuyDgn0iqL4AB8DXQEB/EMIscURbUt25QcOkD5rFhV5OSyeoGJlymlyPsxBEfDZDW7c0lnFazvdeXpO8wvyR7Pt5fp+2pOF2kXFtP5hzLimLa29m0++e+nqpY2KqqzE5iyOGtG/B/wuhLhZURQNIIdYDlT8229kPv0MBW5W/nu3lj+/OUbxzmIAFl7nxl3dNcyOM/LqxhKebuC+OtKBrCIWxCXz+4FTuKtd+OeQttw3JIJAT7eG7pokOYw2MhLD9u0IqxXFxTlZUusc6BVF8QaGAvcACCFMgKmu7Up2+V9+RfYrr5AcqmbRFA9embSIhAWTKaaYt0ZrebC3hlc3VvDKBhNhYWEN3V2HSDxRyIK4o/x5MAdPrSuzhkXyj8ER+MlyfVIzpI1sh6iowJyRgcZJ/4cdsbwyAsgFPlMUZbeiKB8riiKvip0RGxtLeHg4KpWK8PBwYmNjq7WfEILc998n+5VX2NlRzcJ7WrBgylJ6BvZk3rx5zL3Wg/8boGX+dhPP/lXh1PXp9WVHaj7TPtnGDQs3sSO1gMdHRbHx6RE8MaaDDPJSs6WNjASgItl50zeOmLpxBWKAh4UQ2xRFeQ94Gnju3DcpijIdmA7OvzOysbgwB0x1k3oJIch++WUKvl5GfA9X/ndLOJ+PWUIrvb1U39SIAhik4tujGh79vYSwsLBGkRWyNoQQbEmxl+vbeiwff72Gp8Z2ZNqAMDxkuT7pKqBp1w6AiuQUPEeOdMoxHPE/KQPIEEJsO/P7Srh4qlgIsQRYAvakZg44bqNXVQ6Y2bNnVxmUhRBkz3uFgq+X8csANZsmteWTMZ/g7+5vf8O+lbBmNnSehCVyFCGbniM9PZ3Zs2cDDZMVsjaEEMQfyWVBXDIJaQUEemr5z3WduKNfKDqNDPDS1cPFwwPX1q0b94heCHFKUZQTiqJ0EEIcBkYCSXXvWtNXVQ6YqrYLIch58y0Kli7lt35qNk6MOD/IH1sHqx+AsEEsqxjBPx94oNGkAK4um03w58FsFqxNZm9GEUHebrw8qQtTeofgppbl+qSrkzYysnEH+jMeBmLPrLg5BtzroHabtJpWgTq9YCH5n37Kn73VrL8hgo/HnhPkT+2Db+4E/0i4LZZnOvWs8dlCQ7LaBL/tt5frO3SqhFA/Ha/dGM2NMW3QuMpMHNLVTduunVNX3jgk0AshEoEa50hu7mpSBarg2285vXAhG3to+GNyCJ+O/YQA9zMphQvTYenNoPGAO1eCu2+NzxYaisVq46e9WSyISyYlt4y2LfS8fUt3JnYPkuX6JOkMbftIp668kZOhTlSdKlAApRs2curFFznQXss3k3z5Ysziv4O8sYiiRaNQSrMZ9EkpJfMHM2/evFrVjK1PJouN1bvt5frS8gx0aOnJ/Nt7Mj66NS6ympMknUdbeUE2WQb6puhKOWCMhw6R8egjZAW6suhGdz4Y8yHBHsH2F60Wst4fS4vyk4xeZmB/jhWwz8XffffdfPHFF/WWAri6jGYrKxIy+DA+hczCcroGe7F4Wi9GdWopy/VJUhU0lUssnbPyRgb6BmQpKODEgzMpdrXwys0q3hg3nw5+Hf5+w5r/EGRI4p+/GolPtVZuNhgM/PrrryxZsqTRFN4uN1n5ens6S9ankF1cQc9QH+be0JVhHWS5Pkm6EmevvJGBvoEIq5XM/3uCitxs5t2p8OjYufRudc5ljoTPYdsHvLfVxMe7zBftn56eXq8ZI6tSWmFh6dY0Pt5wjNOlJvpF+PH2LT0Y2E6W65OkmtC2aycDfXOTO38+hs2b+WiciuGj7uf6dtf//eLxDfDL/0Hktbz/xU7g4gusDT0XX1Ru5svNqXyy6TiFBjND2gfw8Ij29I2Q5fokqTY0bSMwJCQghHD4IEkue3CiqtIflMStJe/DxcR1VyGuH8m/Yv719075x+HbaeDXDm7+lJfmvnLJ6k0NNRdfUGbiv2sOM/i1OP77xxF6hfqyeuZAvrqvnwzyklQHmrAwRHk5lpwch7ctR/ROUlX6A3VxMd0+/5zUliqW9LBxYMIifgz80T6/PmUyfDMNhIA7loObd7VX7jhbbkkFH288xtItaZSZrIzt0oqHRkTSNdi7XvshSc2VJiwcAFNqGuqWLR3atiJE/Wcj6N27t9i5c2e9H7c+hYeHX7T8UQG+at+eLlpXnrwDNiw4RkVWBWAfpe9/aSARJTtg6gpoP6oBen2xU0VGFq9PYdn2dEwWGxO6BfHQiEiiZLk+SXIoU0YGKdeOotVLL+J7yy2XfI+iKAlCiBrfsyRH9E5yqRuXpvn6EqNyYfFIhR0/ZlYGeYC7O5uJKNkOw55tFEE+o8DAB/EprNiZgVUIbugRzKzh7WjbwqOhuyZJzZK6dWsUtRrTJe6PqSsZ6J3kwhuaorRaHg8MZEd7hRX5hRRtLap8rX8bF94d68bPRyxMeP7JhuhupdTTZSyKT+a7XZkoCtzcK4QHr2lHqL+sJSNJzqS4uKAOCZGBvik5N/2BK/BqcGvK3BVWjm+B+r9/3+QUqFdYOcWdE0WCZ3f6M0HVMNfHk3NKWLg2hR8SM3F1UTG1XygzrmlHkI8s1ydJ9UUTFoZZBvqm49yLqKNLSumkduPtca68ddtH7PDawfTp0zGWG/j6Rnf83BWGx1p5at6r9d7PgyeLWRCXzK/7T+Lm6sJ9gyP455C2BHrJcn2SVN80YWGUbd6MsNlQHDjok4HeiaZOncrNAweSPHEi29pauebOp4jyjSJqahQAJ795nJFtjTy5UcfD896q15U0ezMKmR+XzB9J2XhoXXnwmnbcNzgCfw9tvfVBkqTzacLDEBUVWE6dQh0U5LB2ZaB3ImGzkfrsU5S7WNgzrR//7XRH5WtTB0dAsgm63sKbLyyBerqLNCEtn/f/SmbdkVy83Fx59Nr23DMwHB+dLNUnSQ3tbEIzU1qaDPRNRcGKFdh27+Pb6915etzrqJQzp2KGfFh1P/iGw4S3nR7khRBsOZbHgrhkNqfk4afX8OSYDtw1IAxPN7VTjy1JUvWdG+j1AwY4rF15Z+wl1Lag97mshYVkvvUGSSHQ7/5naak/cwOEEPDDQ1CaAzd/ClrnrUcXQrDuSC5TPtzCHR9t40h2KbPHd2LjU8OZNTxSBnlJamRcW7VC0WoxpTr2gqwc0V+gtgW9L5T29uuoSg3seLAHr0bd9PcL2z+Cw7/AmFchqKdD+36WEIK/DuYwP+4oezKKaO3txosTu3BrH1muT5IaM0WlQhPq+CWWMtBfoDYFvS9UfvAg5Su+56/eamZNeePvBEXZB+yFvaPGQv8HHd11bDbB7wdOMT8umYMniwnxc+fVG6O5MSYYrasM8JLUFKjDwjAdO+7QNmWgv0BdS/QJITj83JNUuIHfQ7MI8Qyxv2CpgO+mg5sPTFro0Hl5i9XGL/vs9ViP5pTSNkDPW1O6M6lHEGpZrk+SmhRteDhl69Y7tH6sDPQXqGuJvtM//4B2fwq/3tSKf/e57+8X1s6D7P1wx7egD3BIX81WG6t3Z7JobTKpeQaiWnrw3m09mNAtSJbrk6QmSh0aijCb7Ussg4Md0qYM9BeoSUHvC9lMJtLfeIWcQBgz6w1cVWc+3rTNsOl96HUPRI2pcx8rLFZWJmTwQXwKGQXldAny4sM7YxjduZUs1ydJTdzZ4G7KzJSB3lnqkhY45bOF6HJLSHl0EOOC+tg3Goth9Qz7UsrRdcshbzRbWbY9ncXrjnGq2EiPEB9emtSF4R0CZTUnSWomNGeCuzkzy2FtOizQK4riAuwEMoUQExzVbkOoTYk+S3ExJUs+5XiEK7fedU4qg/89A0UZcO/voK1d5seyCgux29JYsv44p0sr6Bvux5tTujE4MkAGeElqZlyDgkBRMGdmOq5Nh7UEjwAHAS8HttlkJL47B32ZBWbeSwtdC/vGQ7/C7qUw+HEI7VfjNouNZ8r1bTxOgcHMoEh/FozoSf+2/g7uvSRJjYVKo8E1MBBzRobD2nRIoFcUpQ1wHTAPeNwRbTYlxlNZqFf8zq5uem4Z/5h9Y3kB/PwotOwKw56pUXuFBhOfbkrl803HKTZaGN6hBQ+NaE+vMF8n9F6SpMZGHRzcKEf07wL/Bq7KskM7X38ab6sg+PEnUbuoiY2NRf3ro9zYroJJXxu4w2tFtaaCTpdW8PGG43y1JZUyk5UxXVry0PD2RLeR5fok6WqibhNM+c4Eh7VX50CvKMoEIEcIkaAoyrDLvG86MB2qv1SxKSg6kYLXmh3s79uC2/rdQmxsLN+8MoMfp7jwygYTv+7OJP4Kd9bmFBtZvP4YsdvSqLDYuC66NQ+NiKRjq6tyFkySrnrq4GCKf/4FYTajqOueqqTONWMVRXkVmAZYADfsc/TfCSHurGqf5lQzds1DN9M67gB8s5Do6BF0iQzj1wn5GMyCnovLqLDa3xcWFkZqaup5+2YWlvNhfArf7DyB1SaY1COImcMiiQyU5fok6WpWuHIlJ//zHO3+/ANNmzaV2xusZqwQ4hngmTOdGAY8cbkg35xkHz9A67gDpAwK5YboEQDMjMohxFvN4E/LK4M8nH9nbXqegUXxyazaZb/YclNMGx4c1o4wf3299l+SpMZJfSa4mzMyzwv0tSXX0dfBrjdn0wbo9sTL9g1pm5nVV8O7WyvYkmE9772hoaEk55SyaG0yP+zJwkWlcFufUB4Y1o5gWa5PkqRzqCvX0jvmgqxDA70QIh6Id2SbjVVW8h7axB/m+NC2TOzQF8xG+PFhSlz9mbc1+7z3eoZ0IHrGq4x6Zx1aVxX3DAxn+tC2tJTl+iRJugR1q1agUmHOdMwSSzmir6Wd7z5PBNDz7Gh+07uQl4zntNW8G3GS2bNnc7JCTeuR90CbHhwrd2HG0HDuHxJBgCzXJ0nSZShqNa6tWjbOEf3V4sSJJELWHSFzQATRkTGQlwIb/gtdb4J2I+ikLmD4C+1ZezgXTzdX7h0UwT8GyXJ9kiRVnyYoGJMM9A1n84Ln6WaG6Eeet1eM+uVxcHUjoeOTvP3xVjYl5+GrU/PE6CjuGhiOl6zkJElSDamDgynbtu3/27vz8Kirc4Hj3zeZJJAQloSYYAIJQv5+UgAAFKFJREFU+yIg+yK4ANZbU5dWrUuBam2NStXn1rb31lLr7fXyPPX61FaLFumlihDFrbKJBS2IG1sAARGQACEQCIRgEkK2mcx7/5gpRLYMmclMZvJ+nifPM/Ob8zvznkx4OXPO73dOQOqyRH+Rio7tJWvFdoqHZNB/8Bh065vI3g+ZkziNJ+fvpXO7OH6d3Y/JozNJiLNfrzGmaWIyMnAtXozW1SGx/o0GWCa6SB+9+DuGVMOlP/0Fq7fsZtCiX3LA3YM5NRN44sbe3DWqm23XZ4zxW0x6OqjiLC4m1s+bTC3RX4SSE8WkL95AUWYn/ripLbeXPMZ4Rznrxv+FVRMn2XZ9xpiAaXiJpb+J3vaZ81G9W8l9bjop5cqc9AlkVO/ih44PYNR9XH/d9ZbkjTEBFdMlDQDn4WK/67JE3whnvZu3Nh5k0jPL6LN8LUVJbbhjWg6zkl9HElKInvibUIdojIlAjtRUAFxH/E/0NnRzHrWuet7eWMRfVudz4Hg1ExyL6H7UDf95L/0dn0HRBvjuX6CNLTxmjAm8qLg4opOSAtKjt0R/hhpnPa9vOMCs1Xs4XF7D5RkdmJ7dm9LfTqc63sGQW+6Ev46D9OEw+M5Qh2uMiWCOtFSc1qMPnKo6F7lrC5n98V5KTtQyIrMTv791MFf17sy7a+YyZEctdXdcT9T6P0PlEbjzNYiykS9jTPOJSesSkLtjW32iP1Hj5JU1+5nzyT6On6zjip7JPHvnEMb2SEZEUFWKXvk/ugOX3XYb/P0mGDIZMoaHOnRjTISLSUulaqP/G5C02kRfXuXkpc/28dKnBZRXO7m6TwqPTOrF8Mykb5RbX/AxQ9eWUjn2MmK3/AkcbWDSEyGK2hjTmjjSuuAuL8ddVUVUfHzT6wlgTGGhtLKWOZ/s45U1+6msdfGtAak8NKEXl3fteM7yefP+yMQa6PKdq2Db43Dd/0BiapCjNsa0RjFpnlzjLD5CXI/uTa6n1ST6oxU1/PXjvcxfW0iNq57sgV346YReDLj0/FfN7CvbR8/3d1LZLZkORXMhuReMuj+IURtjWjNHqudaeteRYkv0F3KorJoXV+/htQ0HcNW7uXlIOtOu6Unv1Mb3MV+x6E9cVQLt7hmIHH/NMwHrsBUojTHBceqmqeIjjZS8sIhN9AeOe7bre2vjQVThlmHpTLumF1mdfduur8pZRduFK6lNiKGv+33IHA99r2/mqI0x5rRTN00VH/avnkAE05LsLank+VV7WPh5EdEi3D6iKw9c3ZOuSRc3kbF80wKG7nLBVZlE1a2B654EkWaK2hhjznbqpinr0Xt8deQEM1fms3TrIWKio/jh2Ezuv6onaR0ufrs+VeXAglfo54buiVtg0O2QPqwZojbGmAtzpKXibO09+i+Kypm5Mp9/bC8mPjaa+67swU+u7EFKYtO369t69HMGrTlCZVZb2rQvhUmPBzBiY4zxXUxqGs5Dh/yqI2wT/ebCr5m5Mp9/7jxKYpyDhyf24t5x3emU4P9k6ccLn2dSGSQPOAxjHoCO/i0RaowxTRXTJY3qTZv8qiPsEv36fcf588rdfLz7GB3jY3j0W324+4osOrQNzHZ9ZTVlJL63ltq2QudebWD8owGp1xhjmsKRmkZ9eTnu6uqm1xHAeJqNqvLZnlKe++du1u07TnJCLL+6vh9TxmTSLsDb9S3Ly2XYV/XE9KsiauJj0PbcN1IZY0wwnL5pqumLm/mdJUWkK/AKkAooMFtVn/W3XvAk+A93lfDcyt1sLizjksQ4Hr9hAD8Y1Y22sYHf6ENVOfLmAoa6ofvgRBhxb8DfwxhjLoYjrQsArlAmesAF/FxVN4lIIrBRRN5X1S+bWqHbrby/4wgzV+azraic9I5tefK7A/n+8Ixm3Y9129EtDF13jMouLnIW7mH1y32YMWMGkydPbrb3NMaYC2m4DEJT+b3OrqoeVtVN3scngB1AelPqqncrS7YcIvu5j7l/3kbKq508desgVv3iGqaOyWz2TbffmvVbUsvA0bmc+Vud7N+/nylTptC5c2dyc3Ob9b2NMeZcHGmnl0Foch2BCgZARLKAocC6c7yWA+QAdDtjo1tXvZvFWw7x/Kp89pScpGdKAn+843JuHHwpjujgrPle7aqm2+bd1MYpM7Z9jVtPv1ZaWkpOTg6A9e6NMUEVFRdHVPv2uI6WNL2OQAUjIu2At4F/V9WKM19X1dmqOkJVR6SkpABQ53KzYH0hE/+wmkff2EJMdBQzfzCUFT+7mu8NzQhoks/NzSUrK4uoqCiysrLO6qGv2raQkflQllbD27tcZ51fVVXF9OnTAxaPMcb4ypGSgquk6Yk+ID16EYnBk+RzVfXvjZVXhXlrCpi1ei9FZdUMSu/A7KnDubZ/KlFRgV9mIDc3l5ycHKqqqgDYv3//WT30/Neep7sLXso//y+zsLAw4LEZY0xj/E30fneZRUSAOcAOVX3Gl3N2Flfw+KLtpLaP46UfjWTxQ+O47rK0ZknyANOnTz+V5P+lYQ+9qHQ3mXmlVCQJb11gGOzMISdjjAmGkCd6YBwwFZgoIp97f7IvdEKcI5rcn4zm7QevYELfS5BmXizsfD3xwsJCcnNz+d/f/Bt9DsGHzhruvvtukpOTzyobHx/PjBkzmjVOY4w5l5AnelX9RFVFVQer6hDvz7ILndMjJYFxvTo3e4L/l/P1xJOSkvj5Qzn0qYylXpSnNxcyd+5cnn32WebPn09mZiYiQmZmJrNnz7aJWGNMSDhSUtC6uqafH8BYWqwZM2Z8Y4wePD10gNuujmbIDmFzQi2l9fXgHdIpKCiwxG6MaREc3gtYmio41y6G2OTJk5k9ezaZmZkAREdHU1VVRd2JUgZnJZNUCfMKSk+Vt0lXY0xL4m+ibxU9ejh9dU3Dnv1DY2JJ2NeGiljlw9LTV4TapKsxpiWxHv1FaHj1TWIsTBzfkUH5sIyTOL1lbNLVGNPSOC6xRO+zhkMyj4yO5VBJIjH18ObeYzbpaoxpsaISEpC2bZt+fgBjafH+NSTTPg5+ekUcnXbHsCehnqrOqcybNw+AqVOnnvPOWWOMCRUR8Wv4plUl+hkzZhAfH8/Do2LZ4WhHr8OwvK6a7OxscnJy2L9/P6p66s5ZS/bGmJbCEr2PJk+eTM49P+DRsXHs2hOPG2gz8lqWLVt2wTtnjTEm1PxJ9K3mqhvwrHnT5ovXaDvBQe+CGLYluvjDy3NR1XOWt8ssjTEthfXoffS7x3/NwyOE3INtSP8a3q2sOG+SB7vM0hjTclii99GEpGIuTYyi/FACzihYkn/svGXtMktjTEtiid4X9S4euyqez4rcjD4ay4YOTspd7nMWtcssjTEtjY3R++LLhWS1d/PUlrY8UgnvRp+1NwrgSfIFBQXBjc0YYxphib4RufPnM2z9I1BfT093e6pj4B87zx62seEaY0xL5c/dsRE/dJObm8vbTz1A/6R6fr/GyVXHY/i0o5Nq1zcnYW24xhjTkkV37NjkcyO+Rz99+q/JnajsL3NT6mpPYh0s/br8G2VsuMYY09L5s39HxPfoMznEuG4Onv6sluzEDpS3hVW7Ti9JbMM1xphIF/GJ/nfXtufoSTcLttRzRbmD1UlOnN5hGxuuMca0BpE9dFP8BddkuHjiI+XqbknEuYR3K8qJj4+3BG+MaTUiu0e/5nmIiWfgPc/w3c7JHG0PB04kWJI3xrQqkZvoTxTDtjdh6BS+d8NtDC0X9o1KZ99O2wvWGNO6RG6iXz8b3C4Y/QD733mVaDe0+052qKMyxpigC0iiF5Fvi8guEckXkV8Fok6/1J2EDXOg33cguSelixdSmAJjxt8e6siMMSbo/E70IhINPA9cDwwA7hKRAf7W65fPX4WaMrjiYeoOHiRxVxE7hqeQkZgR0rCMMSYUAtGjHwXkq+peVa0DFgA3B6DepnHXw9oXIH0EdB3N0UVvA/DW0WKioqJsm0BjTKsTiESfDhxo8Pyg99g3iEiOiOSJSF5JSUkA3vY8dr0Hx/fCFQ+BCIULXmVnBmxae/DUNoFTp05l2rRpzReDMca0IEGbjFXV2ao6QlVHpPixCluj1syEDt2g343U7PqKTiUVfNQDqguqG8bCrFmzrGdvjGkVApHoi4CuDZ5neI8F38GNULgGxjwI0Q7KliymXmBFXSWcsZGUqtqesMaYViEQiX4D0FtEuotILHAnsDgA9V68NTMhrgMMm4q63Rxfuogt3YWinSfOWdz2hDXGtAZ+J3pVdQEPAcuBHcAbqrrd33ovWtkB+HIRDL8b4hKp3rwZKT7Gp5dFcXL7yXOeYnvCGmNag4CM0avqMlXto6o9VTU0S0HmzQEURt0HQPnSpThjhJqxg7hv6n1nLfFpq1YaY1qLyLgz1lkNG+dC32zo2A11Oil/7z3W94LRPa/hhRdeYN68eWRmZiIitmqlMaZViYzVK794G6qPw+j7ATj52WdoWTmfXBvFLzPGAzB58mRL7MaYVin8E70qrHsRUvpD1pUAlC99l9r4GA7070T/pP4hDtAYY0Ir/IduDqyH4q2esXkR3FVVnPjgA9b2i2JM5niiJPybaIwx/gj/LLj+Rc8llYPvAODEqlVodTUr+7u4Mv3KEAdnjDGhF96JvuKw55LKoVMgrp3n0JKl1Ca1Y1dXYXSX0SEO0BhjQi+8E/3Glz2LmI38MQCur7+m8pNP+PzydvRN7k+nNp1CG58xxrQA4ZvoXXWw8SXo/S1I7gnAieUrwOViYffjjL10bIgDNMaYliF8E/2Xi6DyCIy6/9ShiqVLcXVLY88l9YzpMiaEwRljTMsRvol+/YuQ1BN6TgTAefgwVXl55I/oQpyjDcNSh4U4QGOMaRnCM9Ef+hwObvBcUhnlaUL50qUALO1VztBLhhIXHRfKCI0xpsUIz0Sf9zdwtIXL7zp1qGLJUhyDLmN9dKGNzxtjTAPhl+hrKmDbWzDwVmjb0XNo1y5qv/qKw+N7A9j4vDHGNBB+iX7bG+A8CSPuPXWoYskSiI7mwz5OOsZ1pF9SvxAGaIwxLUt4JXpVyHsZ0gZBumeyVd1uyt9dRsK4cXx4YhOju4y2ZQ+MMaaB8MqIRRvhyDYY/iPwri9flZeH6/BhaieNpqS6xIZtjDHmDOGV6PNegpgEGPT9U4cqlixB4uPZ3DcGgNFptuyBMcY0FD6JvrrMs+784O9Dm/YAuOvqqFi+gsRrJ7G+fAup8alkJGaEOFBjjGlZwifRb30DXNWeYRuvytWrcVdU0P6GG9hQvIGRaSPP2jLQGGNau/BI9KqedW0uHQqXDjl1uGLJUqKTkzk6oAvHa44zMm1kCIM0xpiWKTwS/YF1cPTLb/Tm6ysqqFy1ivbZ2eQd2wTAyFRL9MYYc6bwSPR5L0FsoucmKa8TK1agTicdbryBDUc22Pi8Mcach1+JXkSeFpGdIrJVRN4RkY6BCuyUquOw/R24/I5Tm4sAlL2zkNisLOIGDrTxeWOMuQB/e/TvAwNVdTDwFfCY/yGdYesbUF8Lw+85daiuoIDqjRvpcMstFFQU2Pi8McZcgF+JXlVXqKrL+3QtEPixk83zocsQz92wXmULF0JUFB1uvokNxRsAG583xpjzEVUNTEUiS4DXVXX+eV7PAXK8T/sCuwLyxhenM3AsBO8bStbm1sHa3Dr0VdXEiz2p0UQvIh8Aaed4abqqLvKWmQ6MAG7RQP3P0QxEJE9VR4Q6jmCyNrcO1ubWoaltdjRWQFWvbeSN7wFuACa15CRvjDGtVaOJ/kJE5NvAfwBXq2pVYEIyxhgTSP5edTMTSATeF5HPRWRWAGJqTrNDHUAIWJtbB2tz69CkNgdsMtYYY0zLFB53xhpjjGkyS/TGGBPhIjLRi8i3RWSXiOSLyK/O8XqciLzufX2diGQFP8rA8qHNj4rIl97lKv4pIpmhiDOQGmtzg3K3ioiKSNhfiudLm0Xkdu9nvV1EXg12jIHmw992NxFZJSKbvX/f2aGIM1BE5G8iclREvjjP6yIiz3l/H1tFZFijlapqRP0A0cAeoAcQC2wBBpxRZhowy/v4Tjw3eoU89mZu8wQg3vv4wdbQZm+5ROAjPHdujwh13EH4nHsDm4FO3ueXhDruILR5NvCg9/EAoCDUcfvZ5quAYcAX53k9G3gPEGAMsK6xOiOxRz8KyFfVvapaBywAbj6jzM3AXO/jt4BJEt4rojXaZlVdpacvgW2e5SqCy5fPGeBJ4CmgJpjBNRNf2nwf8Lyqfg2gqkeDHGOg+dJmBdp7H3cADgUxvoBT1Y+A4xcocjPwinqsBTqKSJcL1RmJiT4dONDg+UHvsXOWUc9aPeVAclCiax6+tLmhH+PpEYSzRtvs/UrbVVXfDWZgzciXz7kP0EdEPhWRtd57XcKZL23+L2CKiBwElgEPBye0kLnYf+/+3TBlwo+ITMGzXMXVoY6lOYlIFPAMcE+IQwk2B57hm2vwfGv7SEQGqWpZSKNqXncBL6vqH0RkLDBPRAaqqjvUgbUUkdijLwK6Nnie4T12zjIi4sDzda80KNE1D1/ajIhcC0wHblLV2iDF1lwaa3MiMBD4UEQK8IxlLg7zCVlfPueDwGJVdarqPjzLh/cOUnzNwZc2/xh4A0BV1wBt8Cx4Fql8+vfeUCQm+g1AbxHpLiKxeCZbF59RZjFwt/fxbcBK9c5yhKlG2ywiQ4EX8ST5cB+3hUbarKrlqtpZVbNUNQvPvMRNqpoXmnADwpe/7YV4evOISGc8Qzl7gxlkgPnS5kJgEoCI9MeT6EuCGmVwLQZ+6L36ZgxQrqqHL3RCxA3dqKpLRB4CluOZsf+bqm4Xkf8G8lR1MTAHz9e7fDyTHneGLmL/+djmp4F2wJveeedCVb0pZEH7ycc2RxQf27wcuE5EvgTqgV+qath+W/WxzT8H/ioiP8MzMXtPOHfcROQ1PP9Zd/bOOzwBxACo6iw88xDZQD5QBfzo3DU1qDOMfx/GGGN8EIlDN8YYYxqwRG+MMRHOEr0xxkQ4S/TGGBPhLNEbY0yEs0RvjDERzhK9McZEuP8HG8QxxbwGfCwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "X_test = np.linspace(-0.1, 1.1, 500)[:, None]\n",
    "\n",
    "# plotting polynomials with increasing degrees\n",
    "plt.scatter(X.ravel(), y, color='black')\n",
    "axis = plt.axis()\n",
    "for degree in [1, 3, 5, 7]:\n",
    "    y_test = PolynomialRegression(degree).fit(X, y).predict(X_test)\n",
    "    plt.plot(X_test.ravel(), y_test, label='degree={0}'.format(degree))\n",
    "plt.xlim(-0.1, 1.0)\n",
    "plt.ylim(-2, 12)\n",
    "plt.legend(loc='best');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"color:red\">Plot polynomial models with increasing degrees. What do you observe?</span>\n",
    "\n",
    "<span style=\"color:blue\">The fit to the data seems to increase with the degree of the polynomial. For the polynomial of degree 7, however, the sharp decrease towards x=1 may not be relevant. Indeed, if the model perfectly fits the training data, its ability to generalize to unseen data may be limited.</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The knob controlling model complexity in this case is the degree of the polynomial, which can be any non-negative integer. A useful question to answer is this: what is a good degree of polynomial?\n",
    "\n",
    "We can make progress in this by visualizing the validation curve for this particular data and model; this can be done straightforwardly using the validation_curve convenience routine provided by Scikit-Learn. Given a model, data, parameter name, and a range to explore, this function will automatically compute both the training score and validation score across the range:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0, 0.5, 'score')"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXhU5dn48e9NCMQgS0LYg4IWRJQdAUUExERwAUEiblXcKNRq6aK1i1Spvm2tr6305wZVW60boEh8RdnEolWUBAOyyKKgkAXCDrJleX5/PDPJJJlJJsmcOTOZ+3Ndc83MOWdm7sxMzj3PLsYYlFJKxa5GbgeglFLKXZoIlFIqxmkiUEqpGKeJQCmlYpwmAqWUinGaCJRSKsY5lghE5AUR2SMi6wPsFxGZJSLbRGSdiPR3KhallFKBOVki+Ccwupr9Y4BunssU4BkHY1FKKRWAY4nAGLMS2F/NIeOAl4y1CmglIh2cikcppZR/jV187U7ATp/7uzzb8isfKCJTsKUGmjVrNqBHjx5hCVAppRqK7OzsvcaYNv72uZkIgmaMmQ3MBhg4cKDJyspyOSKllIouIvJtoH1u9hrKBTr73E/1bFNKKRVGbiaCTOAWT++hIcAhY0yVaiGllFLOcqxqSEReA0YAKSKyC/g9EA9gjHkWWARcAWwDjgG3ORWLUkqpwBxLBMaYG2rYb4C7nXp9pZRSwYmKxmKllHJTaSkcPAj798ORI1BcbC8lJVVvV3ddUuL/+UWC2z50KJx7bmj/NtBEoJSqA2PKT4DGBL54j61uf11fvz6PPX4c9u2zJ3bvte/tytsOHKjfa4bKM89oIlAq5hkDJ07AoUPll8OH4fvv7fbjx2t/fepU+Und91JU5H97cbH9hdwQtWgBrVtDcrK97tq1/HZysr20aAHx8RAXB40bV732t8173ahR1V/5gRKMv+2tWoX+bwZNBCpGGWOL6adO2RNeUVH5CbG01O73vfa3zd++QNUB1VUVeE+uR49WPLlXPtl7bxcX1+5vTUiA007zf92sGSQl2ROb9yRW+VLdvrg4e2Kr7gI176+L+jw2IcGe3H1P+q1a2b81FmkiUEErKan4a9LfxXefv1+Svie+mvbXdFKt7mTrPcFXPtH73o5EjRpBy5b20qKFvU5NrXjf3+3TT/d/om/atH4nTBUbNBE0MEeOwNq1sGaNvXz3nT05en+ten/B+rtdeZv3xO97Yg+16orT3tv+itmVbzdpYk9+3u1Nmthfd5Wva9rmLb57i/CVb1e+rrytujhr2ta8OSQm6olbhZ8mgih24AB88UX5SX/NGtiypbxusV07+MEPyk+M3hNXXFzF60C34+Iq/rr0dwm0r2lT+5qVT/L+qhWUUu7SRBAl9uypeMJfswa2by/f37kz9O8PN95or/v3h44d3YtXKRU9NBFEsG++gUcfhSVLYNeu8u1nnw0DB8KUKfaE368ftPE7p6BSStVME0EE2rkTHnkEXnjBVqGMH29P/P37Q9++znUhU0rFJk0EEaSgAP74R3j2WVvPP3Uq/OGcf9Mq50MoSIZVybAlubxDs++lWTOtcFdK1Ykmggiwbx889hj8/e+2W+Ntt8Hvfgdnbv8QLr3F9g88cQJOngz8JPHxtkN45QTRvHl5K7C/luFAl7g4+5wpKbbVuW1be52SErudrZVqoDQRuOjgQfjrX+3l6FG46Sb4/e9tTx8OH4bhk+GssyAnx/7iP37cdhXyjn0PdDlwAHJz4csvbX9S7+gpb99Qf5faaN26PDH4XvveTknxPwTW936g7UeO2ElVZsyAkSO1pKOUwzQRuODoUZg1Cx5/3J6zJ06Ehx+Gnj19Dpo+3TYWfPSRHS0EtpN5YiJ06hT6oHyHxnqTw8mTUFhouyzt3m2vfW/v3m2T1O7d9gQerLi48tFQ3ssZZ5TfbtoUXn8dRo0qTwhpaZoQlHKIJoIwOn7cThr1pz/Z8+vVV8PMmbYBuIKFC+HFF+HXv4aLLgpPcCLlI5u8EhNtdVP37jU//uTJ8kSxZw/s3WsHGVQeBtuypd1e00n90Udta/kf/wiXXw5DhtiEMHq0JgSlQkxMJEypVwvRuGbxqVPwj3/Yc1tenv1x+4c/wODBfg7eswfOP98OAvj8czsqK5adPAn//Cf8z//YYdIDB9qEcNVVmhCUqgURyTbGDPS3T0sEDjtxAgYMgI0b4eKL4dVXYfjwAAcbAz/6ka1mWb5ckwDYaqIf/ci2oL/0kk0IY8fawRMzZsC4cfVPCIWFkJUFq1fD119XnJQo0MU7PafvpaSkfLi195KYGNz9Zs1s/+Dzz9cEp8JOE4HDFiywSeDFF+HWW2v4H3/pJXj7bduFqFevsMUYFZo0gTvvtG/iK6/YgRbjx0OfPvDgg/Z2oyCW4D58GLKz7Unfe/n2W7tPxA7RTkgon4TIO/1mfLw9YXvnIPa9eI+Ji7MlmGPHbD2g93LwYPlt333+Jm/q2BHS0211WFqabZhXymFaNeSwkSPteWbbthrOU99+C7172xPbihUV6+pVVcXF8NprNiFs2WJ/ST/4IFx7bfl7d/y4bcz2Pelv3lz+HF272qqmCy6wl/797Yk+XIqKypPCoUPw8ceweDEsXWp7EYjY+C6/vLydpLH+dlN1U13VkCYCB23ZAuecY9sGfvObag4sLYXLLrMnqrVrbZdRFZySEnjjDdvo8tVXdvmmCy+0v/rXry9fG7B9+/IT/gUX2BNsSoq7sQdSUmKrqhYvtpdVq+x3pEUL25PKmxi6dHE7UhVFNBG45L777BiBnTuhQ4dqDvzb3+BnP4M5c2z1h6q9khJ4802bdXftsg0zvif+jh2jt+794EHbZuRNDN99Z7d3716eFEaNslVaSgWgicAFJ0/aBUWGDYO33qrmwI0bbZVEWhpkZkbvyUqFhzG2esubFD780FYtdetmx1707+92hCpCVZcIgmhdU3WxcKHtSj9lSjUHFRXBLbfYAWNz5mgSUDUTgR494Kc/hUWL7EjyhQttMhgyBJ54ouEuKKwco4nAIbNnw5ln2h/6AT36qK3Lfu45W4etVG0lJNjutDk5cOWV8Itf2Ovdu92OTEURTQQO+PprW6V7xx3VdP75/HPb4+Xmm21PF6Xqo3VrWwf59NO211mfPnYhC6WCoInAAf/4h+0qevvtAQ44dsxWCXXoYKccVSoURGDaNNv7LCXFNiLff78dIKdUNTQRhNipU3aKnKuuqmZuuF//2jb4vfiirjKjQq9XL1vinDoV/vIXO3Hftm1uR6UimCaCEHvnHTtdUMBG4uXL7dSj99xjxw4o5YTERDvD4Ztv2iTQrx/8+99uR6UilCaCEJs923YbHT3az86DB2HyZDvK7E9/CndoKhZNmGAHKfbtCz/8oa2SPHLE7ahUhNFEEELbt9vZAQI2Et97L+Tn2zmFEhPDHp+KUWecYRuQH3rIztPUv78duayUhyaCEHr+edte57eR+M034eWX4be/hUGDwh6binGNG9vl7z780I52vPBCuzKSjjlQaCIImaIi20g8Zoz9AVZBQYGdSnnAALsYsVJuGTbMjjm4+mo7B8qYMfb7qWKaJoIQefddW+vjt5H4Rz+y61O+/LIu/K7cl5xsS6jPPAMrV9p5iqJsqhkVWpoIQmTOHDuv2RVXVNqxebOdQ+jBB+3MmEpFAhHbvXTWLDvf1Zo1bkekXKSJIAS++w7ee8+2DVSZLv6dd+z1zTeHPS6lajRhgu3ZMG+e25EoFzmaCERktIhsFpFtIvKAn/1niMgKEflCRNaJSOXf01Hh+eft9R13+Nm5cKEd7n/mmWGNSamgtG5tq4bmzdPqoRjmWCIQkTjgKWAM0BO4QUR6Vjrsd8BcY0w/4HrgaaficUpxsU0EftcJKSyETz6xk4IpFakyMuCbb+CLL9yORLnEyRLBIGCbMeYbY8wp4HVgXKVjDOBdG7AlkOdgPI54/33IzQ3QSLxoke2eN67yn61UBLnmGq0einFOJoJOwE6f+7s823w9BNwsIruARcA9/p5IRKaISJaIZBUWFjoRa53Nng3t2tm5harIzLQtyLpYiIpkKSlw6aVaPRTD3G4svgH4pzEmFbgCeFlEqsRkjJltjBlojBnYpk2bsAcZyK5dttvo7bf76RV64oRdQWrsWF1wRkW+jAw7f3pOjtuRKBc4mQhygc4+91M923zdAcwFMMZ8CiQAEbqieFUvvGBrfvwuM/zBB/D991otpKLD+PFaPRTDnEwEq4FuItJVRJpgG4MzKx3zHTAKQETOxSaCyKr7CaCkxK47kJYGZ53l54DMTLsE5ciRYY9NqVpLSbHfVa0eikmOJQJjTDHwE2AxsAnbO2iDiMwUEW83ml8Ad4nIWuA1YLIx0fEtXLIEdu4M0EhcWmrHD1x+OTRtGvbYlKqTjAw7ZfXatW5HosKs8vCnkDLGLMI2Avtum+FzeyMw1MkYnDJ7NrRpE6BnaHY25OVpt1EVXcaPhx//2JYK+vZ1OxoVRm43FkelvDz7g/+226BJEz8HZGbatSqvvDLssSlVZ23awIgRWj0UgzQR1MGLL9o2Ar+NxGATwcUX21GbSkWTjAzYuhXWrXM7EhVGmghqqbTUNhJfeil06+bngB077D+RVgupaDR+vC3Nau+hmKKJoJaWLbPn+oBrEmd6OkZpIlDRqG1brR6KQZoIamn2bFvjc801AQ7IzLTTTfstLigVBTIyYMsW+PJLtyNRYaKJoBZ277aTiU6eHKBX6MGD8J//aGlARbcJE7R6KMZoIqiFf/7TzjZ6110BDnj/fXuAJgIVzdq2heHDtXoohmgiCFJpqV2FbPhwOOecAActXGi74A0eHNbYlAq5jAy7ut769W5HosJAE0GQVqywc3IFbCQ+dcouU3b11XbOFqWimVYPxRRNBEGaPduu+T1hQoADPvoIDh3SaiHVMLRrB5dcotVDMUITQRAKC2HBArjlFkhICHBQZqbdedllYY1NKcdkZMBXX8GGDW5HohymiSAI2dlQVFRNacAY2z6QlgbNmoU1NqUcM2GCXUtDq4caPE0EQcjPt9epqQEO+PJL+PZbrRZSDUv79uXVQ6pB00QQBG8i6NAhwAHe0cR+16tUKoplZMCmTVo91MBpIghCfj60alVN+8DChbbLaPv2YY1LKcdde61WD8UATQRByM+vpjSQmwtZWbokpWqY2reHYcM0ETRwmgiCUG0i+L//s9faPqAaqowM2LjRXlSDpIkgCPn51dT6ZGbaRYt79gxrTEqFjVYPNXiaCGpgDBQUBCgRHD0Ky5fb0oBI2GNTKiw6dLALLWkiaLA0EdTg8GE4fjxAIliyBE6e1PYB1fBlZNieQ5s2uR2JcoAmghpU23U0MxOSkmDo0LDGpFTYafVQg6aJoAYBE0FJiW0ovuIKiI8Pe1xKhVXHjvYHjyaCBkkTQQ0CJoJPPoF9+7RaSMWOjAw7LfVXX7kdiQoxTQQ1CJgIMjNtSeDyy8Mek1KuuPZae62lggZHE0EN8vPhtNOgRYtKOzIzYeRIPzuUaqA6ddLqoQZKE0ENvGMIKvQO3bzZLu6tg8hUrMnIsJMsbt7sdiQqhDQR1MDvGIKFC+21JgIVa7R6qEHSRFADv9NLZGZCv37QubMrMSnlmtRUuOgiTQQNjCaCGlRJBIWFtseQlgZUrMrIgHXrbPWoahA0EVTj+HE4eLBSInj3XTvvhCYCFasmTrTXWipoMDQRVKOgwF5XSAQLF9ricb9+rsSklOtSU+HCCzURNCCaCKpRZQzB8eN2fiGdZE7FuowMWLsWtm51OxIVApoIquFNBGVTUH/wARw7ptVCSmn1UIOiiaAaVaqGFi6E5s1hxAi3QlIqMnTuDEOGaCJoIBxNBCIyWkQ2i8g2EXkgwDHXichGEdkgIq86GU9t5edDXBy0aQOUlsI778Do0dC0qduhKeW+jAzIyYHt292ORNWTY4lAROKAp4AxQE/gBhHpWemYbsCvgaHGmPOA6U7FUxf5+dCuHTRqhF2XuKBAq4WU8hozxl4vXepuHKrenCwRDAK2GWO+McacAl4HKk/VeRfwlDHmAIAxZo+D8dRahTEEmZm2eHDFFa7GpFTE6NHD9iDSRBD1nEwEnYCdPvd3ebb56g50F5H/isgqERnt74lEZIqIZIlIVmFhoUPhVlUhESxcCMOGQXJy2F5fqYgmAmlpdrnWkhK3o1H14HZjcWOgGzACuAGYIyKtKh9kjJltjBlojBnYpk2bsAVXlgi2b7fzsGu1kFIVpafDgQOQne12JKoenEwEuYDvZDypnm2+dgGZxpgiY8x2YAs2MbiuuBj27PF0Hd2wwW686CJXY1Iq4owaZa+XLHE3DlUvTiaC1UA3EekqIk2A64HMSse8jS0NICIp2KqibxyMKWh79tiZJDp0AHI9+atT5ZotpWJcmzbQv7+2E0Q5xxKBMaYY+AmwGNgEzDXGbBCRmSLirWNZDOwTkY3ACuA+Y8w+p2KqjQpjCPLybH1o2cgypVSZtDT49FM4csTtSFQdOdpGYIxZZIzpbow52xjzqGfbDGNMpue2Mcb83BjT0xjTyxjzupPx1EaF6SVyc20/0saNXY1JqYiUng5FRfCf/7gdiaojtxuLI1aFRJCXp9VCSgUydKhdz1Wrh6KWJoIAKswzlJsLHTu6Go9SEatpUxg+XBuMo5gmggDy86F1a2jSBC0RKFWTtDT46ivYubPmY1XE0UQQgHfRek6ehL17tUSgVHXS0+21Vg9FJU0EAZQNJvPWEWmJQKnAzjvP/sNoIohKQScCEblYRG7z3G4jIl2dC8t9BQWVxhBoiUCpwETgsstg2TI7U6+KKkElAhH5PfAr7EyhAPHAv50Kym3G+JQI8vLsRi0RKFW99HRbjZqT43YkqpaCLRGMB8YC3wMYY/KA5k4F5bYDB+DUKS0RKFUrl11mr7X3UNQJNhGcMsYYwACISDPnQnJflTEETZvqrKNK1aR9e+jdW9sJolCwiWCuiDwHtBKRu4BlwBznwnJXlVHFHTvqYvVKBSMtDT7+2K7traJGUInAGPM4MB94EzgHmGGM+buTgbmpwmAyHUOgVPDS02296sqVbkeiaqHGyXM8S04uM8aMBGKizFelaqhvX1fjUSpqDBtmq1KXLLHre6uoUGOJwBhTApSKSMswxBMR8vOhWTNofrrR6SWUqo3TTrPJQNsJokqw02keBb4UkaV4eg4BGGPudSQql5WNIThyBL7/XquGlKqNtDT41a8qrfWqIlmwjcVvAQ8CK4Fsn0uDVPb91a6jStWeTjcRdYIqERhj/uVZZay7Z9NmY0yRc2G5Kz/f0yygg8mUqr3eve3KZUuXwi23uB2NCkKwI4tHAFuBp4CngS0icomDcblKSwRK1UOjRrZ6aOlSO0xfRbxgq4b+F0g3xgw3xlwCXA781bmw3PP997ZpoML0EpoIlKqdtDTYvRu+/NLtSFQQgk0E8caYzd47xpgt2PmGGpwqC9K0bGm7ECmlgpeWZq91uomoEGwiyBKRf4jICM9lDpDlZGBu0SUqlQqBTp2gZ09tMI4SwSaCacBG4F7PZaNnW4NTUGCvK0wvoZSqvfR0O8L4xAm3I1E1CDYRNAaeNMZMMMZMAGYBcc6F5R4tESgVImlpNgl8/LHbkagaBJsIlgOn+dw/DTvxXIOTnw/x8dA6qdTe0RKBUnUzfLj9Z9J2gogXbCJIMMYc9d7x3E50JiR3edcqlr2FUFysJQKl6qpZMxg6VNsJokCwieB7EenvvSMiA4HjzoTkLh1DoFQIpaXZFct273Y7ElWNYBPBT4F5IvKRiHwEvA78xLmw3OMtEeioYqVCwDvdxPLl7sahqhVsIugK9MP2FFoKbMazWllDoyUCpUKoXz+7up9WD0W0YBPBg8aYw0ArYCR2molnHIvKJUVFdu3tsh5DIp7igVKqTuLi7FrGS5bodBMRLNhEUOK5vhKYY4x5F2jiTEju8VZjliWCdu2gcbAzdSul/EpLs/9Pmza5HYkKINhEkOtZs3gSsEhEmtbisVHD71rFSqn60ekmIl6wJ/PrgMXA5caYg0AycJ9jUblEB5Mp5YAzz4Tu3bWdIIIFu3j9MWPMW8aYrZ77+caYBpfetUSglEPS0+HDD+HkydA836FDUFoamudSDa96pz7y8237cNuWJ22rsZYIlAqNtDQ4dgw+/bT+z/Wf/9j/zcceq/9zKUATQQX5+ZCSAvF7PUUDLREoFRojRtiOF/VtJ/jgAxgzxi4csn17SEJTmggqKBtDoIPJlAqtFi1gyJD6tRMsXQpXXglnnw2pqbB/f+jii3GOJgIRGS0im0Vkm4g8UM1x14qI8Uxd4ZqCAh1MppRj0tMhOxv27av9Y99/H66+2jY6f/CBbYDWRBAyjiUCEYnDrnE8BugJ3CAiPf0c1xw7hcVnTsUSLC0RKOWgtDQ7qKy2000sWgTjxsG559ok0KaNHa2siSBknCwRDAK2GWO+Mcacws5PNM7PcX8A/gy4unpFaWmlEkHTpvbLppQKjYEDoVWr2rUTvPMOXHMN9OplE0jr1na7JoKQcjIRdAJ2+tzf5dlWxjOjaWfPSOWARGSKiGSJSFZhYWHoI8WWVouLfUoEHTvaLkRKqdBo3BguvdTW9Qcz3cSCBXDttXa+omXLKv4w00QQUq41FotII+AJ4Bc1HWuMmW2MGWiMGdimTRtH4qmyaL22DygVemlp8N13sHVr9cfNnw/XXQcDBtgSRKtWFfcnJ8PRo3DqlHOxxhAnE0Eu0Nnnfqpnm1dz4HzgQxHZAQwBMt1qMNZRxUqFgXda6uqqh954A66/HgYNgsWLoWXLqsckJdnrAwdCH2MMcjIRrAa6iUhXEWkCXA9kencaYw4ZY1KMMV2MMV2AVcBYY0yWgzEFVJYI2hstESjllLPOspdA3UhffRVuvBEuusj2FGrRwv9x3moiTQQh4VgiMMYUYxevWQxsAuYaYzaIyEwRGevU69ZVQYG97nD6ETtYRUsESjkjPR1WrLDzvvt66SX44Q/hkkvgvfegefPAz+FNBNpOEBKOthEYYxYZY7obY842xjzq2TbDGJPp59gRbpUGwJYIWrSAxIOerqNaIlDKGWlpcOQIfObTY/zFF2HyZBg5Et591653XB1NBCGlI4s9qqxMpiUCpZxx6aXQqFF5O8GcOXD77TZBvPMOJCbW/ByaCEJKV13xqDKYTEsESjmjVSvbELx0qf0/mzYNRo+23UUTEoJ7Dk0EIaUlAo+yRet1egmlnJeebquGpk2z8wfVJgmA7UkkookgRDQRYMe2VCgRtGxZcx2lUqrurrjC/uONHQtvvlm7JAC2aikpSRNBiGjVELbd6tgxTyJYpV1HlXLc4MGQkwM9e0J8fN2eQ0cXh4wmAnQwmVKu6NOnfo/XRBAyWjWEzxgCXaJSqeihiSBkNBHgUyJoV2rvaIlAqciniSBkNBFQngg6xhfaKUi1RKBU5NNEEDKaCLCJoGlTaPm9LkijVNRIToaDB6GkxO1Iop4mAsrHEEiejiFQKmokJ9suqIcOuR1J1NNEgC5RqVRU0tHFIaOJgErzDIlAu3Zuh6SUqokmgpDRRIDPWsV5eTYJ1HWAi1IqfDQRhEzMJ4KTJ+33SMcQKBVlNBGETMwnggqDyXRUsVLRQxNByMR8IqgyvYSWCJSKDt51izUR1JsmAm8iSD4JhYWaCJSKFo0b22UFNRHUmyYCTyLoFOepI9KqIaWih44uDglNBPl2avPWJ3QwmVJRRxNBSGgiyIe2bSFutw4mUyrqaCIIiZhPBGVjCHSJSqWijyaCkIj5RFBheokmTaB1a7dDUkoFSxNBSGgiyK80mEzE7ZCUUsHyJgJj3I4kqsV0Iigpgd277cyjOphMqSiUlGT/kY8ccTuSqBbTiaCwEEpLdXoJpaKWd3TxgQPuxhHlYjoR6KL1SkU5nWYiJDQRAKktDsPRo1oiUCraaCIIiZhOBN4J5zqJjiFQKippIgiJmE4E3hJBm1M6hkCpqKSJICRiPhEkJUGTvVoiUCoq6QykIRHziaB9e3RUsVLR6rTT7EUTQb3EfCIo6zHUsiU0a+Z2SEqp2tLRxfWmiUAXpFEqumkiqDdHE4GIjBaRzSKyTUQe8LP/5yKyUUTWichyETnTyXh8GeNnegmlVPTRRFBvjiUCEYkDngLGAD2BG0SkZ6XDvgAGGmN6A/OBx5yKp7KDB+3C9TqYTKkop4mg3pwsEQwCthljvjHGnAJeB8b5HmCMWWGMOea5uwpIdTCeCsoWrW9XqlVDSkUzTQT15mQi6ATs9Lm/y7MtkDuA9/ztEJEpIpIlIlmFhYUhCc47huCMxL1QXKwlAqWilSaCeouIxmIRuRkYCPzF335jzGxjzEBjzMA2bdqE5DW9iaCj0a6jSkW15GQ4cQKOH3c7kqjlZCLIBTr73E/1bKtARC4DfguMNcacdDCeCryJoG2xDiZTKqrp6OJ6czIRrAa6iUhXEWkCXA9k+h4gIv2A57BJYI+DsVSRn2/HoSQe0BKBUlFNE0G9OZYIjDHFwE+AxcAmYK4xZoOIzBSRsZ7D/gKcDswTkRwRyQzwdCHn7Toq+Xl2VbL27cP10kqpUNJEUG+NnXxyY8wiYFGlbTN8bl/m5OtXp8IYgrZtIT7erVCUUvWhiaDeHE0EkaygAM4/Hx1DECZFRUXs2rWLEydOuB2KqoeEhARSU1OJj6QfTpoI6i1mE0F+PqSlAStzoXPnGo9X9bNr1y6aN29Oly5dEBG3w1F1YIxh37597Nq1i65du7odTjlNBPUWEd1Hw+34cTh0SEcVh9OJEydo3bq1JoEoJiK0bt068kp1zZrZql1NBHUWk4mgbAxB65N2BXvtMRQWmgSiX0R+hiI6qKyeYjoRnNnUu1allgiUimqaCOolphNB2VrFWiJo8A4ePMjTTz9dp8deccUVHDx4sNpjZsyYwbJly+r0/CoENBHUS0wngrZFOpgsVlSXCIqLi6t97KJFi2jVqlW1x8ycOZPLLnOtN3RANf1tDYYmgnqJyV5D+fnQuDE0P6rTS7hh+nTIyQntc/btC3/7W+D9DzzwAF9//c8VyVsAABMgSURBVDV9+/YlLS2NK6+8kgcffJCkpCS++uortmzZwjXXXMPOnTs5ceIEP/3pT5kyZQoAXbp0ISsri6NHjzJmzBguvvhiPvnkEzp16sTChQs57bTTmDx5MldddRUTJ06kS5cu3HrrrbzzzjsUFRUxb948evToQWFhITfeeCN5eXlceOGFLF26lOzsbFJSUsriLCkp4Y477iArKwsR4fbbb+dnP/sZ27ZtY+rUqRQWFhIXF8e8efM466yzuP/++3nvvfcQEX73u98xadIkPvzwwyp/27///W9mzZrFqVOnGDx4ME8//TRxcXGh/RDclJwMa9e6HUXUiskSQUEBtGsHjfJyoUkTaN3a7ZCUw/70pz9x9tlnk5OTw1/+Yuc2XLNmDU8++SRbtmwB4IUXXiA7O5usrCxmzZrFvn37qjzP1q1bufvuu9mwYQOtWrXizTff9Pt6KSkprFmzhmnTpvH4448D8PDDD3PppZeyYcMGJk6cyHfffVflcTk5OeTm5rJ+/Xq+/PJLbrvtNgBuuukm7r77btauXcsnn3xChw4deOutt8jJyWHt2rUsW7aM++67j3xPcdf3b9u0aRNvvPEG//3vf8nJySEuLo5XXnml/m9qJNESQb3EbImgwhKVkdgTogGr7pd7OA0aNKhCf/hZs2axYMECAHbu3MnWrVtpXelHQteuXenbty8AAwYMYMeOHX6fe8KECWXHvPXWWwB8/PHHZc8/evRokpKSqjzurLPO4ptvvuGee+7hyiuvJD09nSNHjpCbm8v48eMBO6jL+3w33HADcXFxtGvXjuHDh7N69WpatGhR4W9bvnw52dnZXHDBBQAcP36ctm3b1v4Ni2RJSXD0KBQV6SwBdRCziaBzZ3SJyhjXrFmzstsffvghy5Yt49NPPyUxMZERI0b47S/ftGnTsttxcXEcDzD1sfe4uLi4WtXTJyUlsXbtWhYvXsyzzz7L3LlzefLJJ4N+vJfv32aM4dZbb+WPf/xjrZ8nangHlR04YKeMUbUSk1VDFUoE2j4QE5o3b86RI0cC7j906BBJSUkkJiby1VdfsWrVqpDHMHToUObOnQvAkiVLOHDgQJVj9u7dS2lpKddeey2PPPIIa9asoXnz5qSmpvL2228DcPLkSY4dO8awYcN44403KCkpobCwkJUrVzJo0KAqzzlq1Cjmz5/Pnj12gt/9+/fz7bffhvzvc5WOLq6XmEsExcV2DJkuWh9bWrduzdChQzn//PO57777quwfPXo0xcXFnHvuuTzwwAMMGTIk5DH8/ve/Z8mSJZx//vnMmzeP9u3b07x58wrH5ObmMmLECPr27cvNN99c9iv+5ZdfZtasWfTu3ZuLLrqIgoICxo8fT+/evenTpw+XXnopjz32GO39zKLbs2dPHnnkEdLT0+nduzdpaWllbQkNhiaC+jHGRNVlwIABpj527TIGjPnHXw/bG489Vq/nU8HZuHGj2yG47sSJE6aoqMgYY8wnn3xi+vTp43JEdRORn+Xnn9v/53fecTuSiAVkmQDn1ZhrI/D+EOoSr2MIVHh99913XHfddZSWltKkSRPmzJnjdkgNh5YI6iXmEkGBZ1aJjqJjCFR4devWjS+++MLtMBomTQT1EnNtBGWjik9piUCpBqNlS9sNXBNBncRsImh5TOcZUqrBaNTIjiXQRFAnMZkIWreGxgW50KIFnH662yEppUJBRxfXWUwmAh1DoFQDpImgzmI3EegYAlWD0z2lxby8PCZOnOj3mBEjRpCVlVXt8/ztb3/j2LFjZfeDmdZa1YEmgjqL3USgJQIVpI4dOzJ//vw6P75yIghmWms3lJSUuB1C/WgiqLOY6j5qjO0+2rF9qc0IWiJwhwvzUD/wwAN07tyZu+++G4CHHnqI008/nalTpzJu3DgOHDhAUVERjzzyCOPGjavw2B07dnDVVVexfv16jh8/zm233cbatWvp0aNHhbmGpk2bxurVqzl+/DgTJ07k4YcfZtasWeTl5TFy5EhSUlJYsWJF2bTWKSkpPPHEE7zwwgsA3HnnnUyfPp0dO3YEnO7a17x583j44YeJi4ujZcuWrFy5kpKSEn71q1/x/vvv06hRI+666y7uueceli9fzi9/+UuKi4u54IILeOaZZ2jatCldunRh0qRJLF26lPvvv58LLriAu+++m8LCQhITE5kzZw49evQI1afkLE0EdRZTiWD/fjs5Ydfme+0NTQQxY9KkSUyfPr0sEcydO5fFixeTkJDAggULaNGiBXv37mXIkCGMHTs24Nq8zzzzDImJiWzatIl169bRv3//sn2PPvooycnJlJSUMGrUKNatW8e9997LE088wYoVKyqsOwCQnZ3Niy++yGeffYYxhsGDBzN8+HCSkpLYunUrr732GnPmzOG6667jzTff5Oabb67w+JkzZ7J48WI6depUVtU0e/ZsduzYQU5ODo0bN2b//v2cOHGCyZMns3z5crp3784tt9zCM888w/Tp0wE7/caaNWsAOy/Rs88+S7du3fjss8/48Y9/zAcffBCaD8Fpyclw8CCUlEBDWmshDGIqEZStVRyvg8lc5cI81P369WPPnj3k5eVRWFhIUlISnTt3pqioiN/85jesXLmSRo0akZuby+7du/3O2QOwcuVK7r33XgB69+5N7969y/bNnTuX2bNnU1xcTH5+Phs3bqywv7KPP/6Y8ePHl80UOmHCBD766CPGjh0b1HTXQ4cOZfLkyVx33XVl014vW7aMqVOn0rix/ddOTk5m7dq1dO3ale7duwNw66238tRTT5UlgkmTJgFw9OhRPvnkEzIyMspe4+TJkzW/uZEiOdkW+w8dKh9gpoISk4mgEzqYLBZlZGQwf/58CgoKyk5+r7zyCoWFhWRnZxMfH0+XLl38Tj9dk+3bt/P444+zevVqkpKSmDx5cp2exyuY6a6fffZZPvvsM959910GDBhAdnZ2nV7Lm4hKS0tp1aoVOaGutgsX39HFmghqJaYai72JoE2Rlghi0aRJk3j99deZP39+2a/eQ4cO0bZtW+Lj41mxYkWN0zNfcsklvPrqqwCsX7+edevWAXD48GGaNWtGy5Yt2b17N++9917ZYwJNgT1s2DDefvttjh07xvfff8+CBQsYNmxY0H/P119/zeDBg5k5cyZt2rRh586dpKWl8dxzz5WtgbB//37OOeccduzYwbZt2wA7k+nw4cOrPF+LFi3o2rUr8+bNA+yElGujaflHnWaizmIyESQdy7XD0QMU/1XDdN5553HkyBE6depEhw4dALsEZFZWFr169eKll16qsWF02rRpHD16lHPPPZcZM2YwYMAAAPr06UO/fv3o0aMHN954I0OHDi17zJQpUxg9ejQjR46s8Fz9+/dn8uTJDBo0iMGDB3PnnXfSr1+/oP+e++67j169enH++edz0UUX0adPH+68807OOOOMsumpX331VRISEnjxxRfJyMigV69eNGrUiKlTp/p9zldeeYXnn3+ePn36cN5557Fw4cKg43GdJoI6Ezs7afQYOHCgqanfdiDTp8Pzz8ORG6ZAZmb5DHTKcZs2beLcc891OwwVAhH7WW7eDD16wCuvwI03uh1NxBGRbGPMQH/7Yq5EoIPJlGqgtERQZzGVCAoKdDCZUg1WUpK91kRQazGVCPLzPc0CWiJwRbRVQ6qqIvozbNzYTiSpiaDWYi4RpLY9ZRct1hJBWCUkJLBv377IPpGoahlj2LdvHwkJCW6HEpiOLq6TmBlHcPSovfygmafrkJYIwio1NZVdu3ZRWFjodiiqHhISEkhNTXU7jMA0EdRJzCSCslHFTXQMgRvi4+Pp2rWr22Gohk4TQZ04WjUkIqNFZLOIbBORB/zsbyoib3j2fyYiXZyKpWxUsdFRxUo1WJoI6sSxRCAiccBTwBigJ3CDiPSsdNgdwAFjzA+AvwJ/diqeslHFxVoiUKrB0kRQJ06WCAYB24wx3xhjTgGvA+MqHTMO+Jfn9nxglASa9rGevGPHko7lQXy8Xa9SKdWweBOBdkqoFSfbCDoBO33u7wIGBzrGGFMsIoeA1sBe34NEZAowxXP3qIhsrmNMKYmzPM/dKKI6TKVQ6W+OEBpX7URqXBC5sTkTV/3/vxvi+3VmoB1R0VhsjJkNzK7v84hIVqAh1m7SuGpH46q9SI1N46odp+Jy8mdxLtDZ536qZ5vfY0SkMdAS2OdgTEoppSpxMhGsBrqJSFcRaQJcD2RWOiYTuNVzeyLwgdERR0opFVaOVQ156vx/AiwG4oAXjDEbRGQmkGWMyQSeB14WkW3AfmyycFK9q5cconHVjsZVe5Eam8ZVO47EFXXTUCullAqtiOo6o5RSKvw0ESilVIxrkIkgkqa28HnNziKyQkQ2isgGEfmpn2NGiMghEcnxXGY4HZfndXeIyJee16yy/JtYszzv1zoR6R+GmM7xeR9yROSwiEyvdEzY3i8ReUFE9ojIep9tySKyVES2eq6TAjz2Vs8xW0XkVn/HhDCmv4jIV57PaYGItArw2Go/c4die0hEcn0+rysCPLba/18H4nrDJ6YdIpIT4LGOvGeBzg1h/X4ZYxrUBdsw/TVwFtAEWAv0rHTMj4FnPbevB94IQ1wdgP6e282BLX7iGgH8nwvv2Q4gpZr9VwDvAQIMAT5z4TMtAM506/0CLgH6A+t9tj0GPOC5/QDwZz+PSwa+8VwneW4nORhTOtDYc/vP/mIK5jN3KLaHgF8G8VlX+/8b6rgq7f9fYEY437NA54Zwfr8aYokgoqa28DLG5Btj1nhuHwE2YUdWR4NxwEvGWgW0EpEOYXz9UcDXxphvw/iaFRhjVmJ7tvny/R79C7jGz0MvB5YaY/YbYw4AS4HRTsVkjFlijCn23F2FHb8TdgHer2AE8//rSFyec8B1wGuher0gYwp0bgjb96shJgJ/U1tUPuFWmNoC8E5tERaeqqh+wGd+dl8oImtF5D0ROS9MIRlgiYhki53Oo7Jg3lMnXU/gf0433i+vdsYYz3SGFADt/Bzj5nt3O7Yk509Nn7lTfuKptnohQFWHm+/XMGC3MWZrgP2Ov2eVzg1h+341xEQQ0UTkdOBNYLox5nCl3Wuw1R99gL8Db4cprIuNMf2xM8XeLSKXhOl1ayR2MOJYYJ6f3W69X1UYW06PmL7YIvJboBh4JcAhbnzmzwBnA32BfGw1TCS5gepLA46+Z9WdG5z+fjXERBCxU1uISDz2g37FGPNW5f3GmMPGmKOe24uAeBFJcTouY+wiDcaYPcACbPHcVzDvqVPGAGuMMbsr73Dr/fKx21tF5rne4+eYsL93IjIZuAq4yXMCqSKIzzzkjDG7jTElxphSYE6A13Tlu+Y5D0wA3gh0jJPvWYBzQ9i+Xw0xEUTk1Bae+sfngU3GmCcCHNPe21YhIoOwn4+jCUpEmolIc+9tbGPj+kqHZQK3iDUEOORTZHVawF9pbrxflfh+j24FFvo5ZjGQLiJJnqqQdM82R4jIaOB+YKwx5liAY4L5zJ2IzbddaXyA1wzm/9cJlwFfGWN2+dvp5HtWzbkhfN+vULeAR8IF28tlC7b3wW8922Zi/zkAErBVDduAz4GzwhDTxdii3Togx3O5ApgKTPUc8xNgA7anxCrgojDEdZbn9dZ6Xtv7fvnGJdhFhr4GvgQGhulzbIY9sbf02ebK+4VNRvlAEbYe9g5su9JyYCuwDEj2HDsQ+IfPY2/3fNe2Abc5HNM2bJ2x9zvm7R3XEVhU3WcehvfrZc/3Zx32JNehcmye+1X+f52My7P9n97vlc+xYXnPqjk3hO37pVNMKKVUjGuIVUNKKaVqQROBUkrFOE0ESikV4zQRKKVUjNNEoJRSMU4TgVJ+eGbK/KXbcSgVDpoIlHKIZ7SqUhFPE4FSHiLyWxHZIiIfA+d4tp0tIu97Jhr7SER6+Gxf5Zmf/hEROerZPsJzXCaw0bPtZhH53DOP/XMiEufZni4in4rIGhGZ55lrRqmw00SgFCAiA7DTGfTFjuq8wLNrNnCPMWYA8Evgac/2J4EnjTG9sCNUffUHfmqM6S4i5wKTgKHGmL5ACXCTZ06k3wGXGTuRWRbwc8f+QKWqoUVXpaxhwALjmZ/H84s+AbgImOezXEVTz/WFlM8P/yrwuM9zfW6M2e65PQoYAKz2PMdp2MnDhmAXH/mvZ3sT4NOQ/1VKBUETgVKBNQIOen7J18b3PrcF+Jcx5te+B4jI1dgFRW6oZ4xK1ZtWDSllrQSuEZHTPLNMXg0cA7aLSAaUrd3cx3P8KuBaz+3rq3ne5cBEEWnreY5kETnT8/ihIvIDz/ZmItI95H+VUkHQRKAUYOxSgW9gZ5d8DzsdMsBNwB0i4p110rts4nTg5yKyDvgBdpU7f8+7EdsWsMRz7FLsrJuFwGTgNc/2T4EeDvxpStVIZx9Vqg5EJBE4bowxInI9cIMxJmRr6yoVTtpGoFTdDAD+n2dRkYPYOeGVikpaIlBKqRinbQRKKRXjNBEopVSM00SglFIxThOBUkrFOE0ESikV4/4/tUSj/a1EoxAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.model_selection import validation_curve\n",
    "degree = np.arange(0, 21)\n",
    "train_score, val_score = validation_curve(PolynomialRegression(), X, y,\n",
    "                                          'polynomialfeatures__degree', degree, cv=7)\n",
    "\n",
    "plt.plot(degree, np.median(train_score, 1), color='blue', label='training score')\n",
    "plt.plot(degree, np.median(val_score, 1), color='red', label='validation score')\n",
    "plt.legend(loc='best')\n",
    "plt.ylim(0, 1)\n",
    "plt.xlabel('degree')\n",
    "plt.ylabel('score')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"color:red\">Does this correspond to the expected result?</span>\n",
    "\n",
    "<span style=\"color:blue\">Yes! The performance on the training set keeps on increasing (even though slightly) while they decrease on the validaiton set (bias-variance tradeoff).</span>\n",
    "\n",
    "<span style=\"color:red\">What's the best polynomial to use? Plot the dataset with this polynomial.</span>\n",
    "\n",
    "<span style=\"color:blue\">The best polynomial is a third-order polynomial. Its plot is given below.</span>\n",
    "\n",
    "<span style=\"color:red\">Redo this study by varying the size of the dataset. How do the results vary?</span>\n",
    "\n",
    "<span style=\"color:blue\">This one's for you!</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(-0.05673314103942452,\n",
       " 0.994263633135634,\n",
       " -0.7459943120970807,\n",
       " 10.918045992764213)"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXyU5b338c81kz0kARJABDIBZFFxQSKCUut6XOvSo1YN1q2N1rbHoz1Wberx6RL7tH1sa6tdYutKrAsiVcQdFTdAdmSTxSQkbCFAEjIkmeV6/kigBBJIMvvM9/165ZXJPXcmv1viN9dc97UYay0iIhJ7HJEuQEREekcBLiISoxTgIiIxSgEuIhKjFOAiIjEqKZw/LC8vzxYUFITzR4qIxLxFixbtsNYOOPh4WAO8oKCAhQsXhvNHiojEPGNMZWfH1YUiIhKjFOAiIjFKAS4iEqMU4CIiMUoBLiISoxTgIiIxSgEucoDy8nIKCgpwOBwUFBRQXl4e6ZIkwb2/ZnuXz4V1HLhINCsvL6e4uBi32w1AZWUlxcXFABQVFUWyNElQMxZXc8/05V0+rxa4SLuSkpL94b2P2+2mpKQkQhVJInt87kbufnEZpw3v3+U5CnCRdlVVVT06LhIKPr/l56+tonT2ai45YTBP3nxql+cqwEXa5efn9+i4SLA1Nnv4ztOf88QnX3HzGQX88brxpCY5uzxfAS7SrrS0lIyMjA7HMjIyKC0tjVBFkkg27XRz1V8+Y+66HZReOY4Hv3E8Toc57PccMcCNMU8YY7YbY7444Fh/Y8w7xph17Z/7BaF+kYgqKiqirKwMl8uFMQaXy0VZWZluYErILazYyRWPfcKW+r08c8tEik5zdev7utMCfwq48KBj9wHvWWtHAe+1fy0S84qKiqioqMDv91NRUaHwRkMrQ8lay7PzKrn+8flkpyfzyvfP4Ixj8rr9/UccRmitnWuMKTjo8OXAWe2PnwY+AO7t9k8VkZigoZWh42718pMZK5i5dDNnjRnAH751Mn0zUnr0Gr3tAx9krd3S/ngrMKirE40xxcaYhcaYhbW1tb38cSISCRpa2bVA3plsqN3DFY99wr+WbeZH54/miRtP7XF4QxAm8lhrrTHGHub5MqAMoLCwsMvzRCT6aGhl5wJ5Z/Lqss3c//JyUpOdPHPLRL426pCNdrqtty3wbcaYwQDtn7ue6ykiMUtDKzvXm3cmDc0e7nphKf/1zyWMOSqLWT+cElB4Q+8D/FXgxvbHNwL/CqgKEYlKGlrZuZ6+M5m/sY6L/vARry7bzH+fN4oXb5vM0X3TA66jO8MI/wl8BowxxlQbY24F/i9wvjFmHXBe+9ciEmc0tLJz3X1n0uL18es313Dt4/NIchpeun0y/33eaJKcwZmCY6wNX7d0YWGh1abGIhLrDu4Dh7Z3Jgf+cVtYsZP7Zqxg/fY9XHvqMB649DgyU3t329EYs8haW3jwca1GKCLSQ/tCuqSkhKqqKvLz8yktLaWoqIjGZg+/eXMtz86rZEjfdJ68+VTOHjMwJHVoKr2IRL1onEx08KSv66+/nrdXbuX8381l2vxKbj6jgLfvOjNk4Q1qgYtIlIuFyUTrtjXy81mr+GjdDsYMyuIvU09hfH7oVxhRH7iIRLWCggIqKysPOe5yuaioqAh/QQeod3v4/btf8uy8SjJTnNx1/mimTnKRHKSblPuoD1xEYlI0TiZq9vgon1/Fo3PWUb/Xw/Wn5XP3+WPon9nz2ZSBUICLSFTLz8/vtAUeiclEHp+flxZW86c569hS38wZx+RScvFxHHd0dthrAQW4iES50tLSTofshXMykdfn59Vlm3nkvXVU1rk5Jb8vD19zEqeP7P7KgaGgABeRqHa4IXuhtrfVxwufV/H4R19Rs3svxw7O5ombCjl7zECMOfxmC+Ggm5giIgfZ2dTKM59V8PSnFexye5jg6sftXx/JuWMH4jjCLjmh0NVNTI0DF5GIipYx3tZaFlXu5K4XljLpV+/xh3fXMcHVj+m3T+bl753O+ccNikh4H466UEQkYqJhjHf9Xg+zlm9m2rwqVm9poE9qEteeOowbJrkYNSgrLDX0lrpQRCRi8vLyqKurO+R4qMd4t3r9fLB2OzOX1vDu6u20ev0cNzibqZNcXH7y0b1esyRUNA5cElp5eXlEboJJ18rLyzsNbwjNGO9Wr59PN+zgrZXbeOOLLex2e8jNTOH6iflcOX4IJw7NiYobkz2hAJe4Fw1v0+VQh9v8IFhjvOvdHuauq+XtVdv4YM12Glu8ZKQ4OffYQXxz/BCmjMoL+qzJcFIXisS9aJ6KncgcDgdd5c+0adN69ce11etncdUuPl63g4/W72BF9W78FnIzUzjv2EFcMG4Qp4/MIy3ZGWj5YdVVF4oCXOJeV0FhjMHv90egIoGu/7Dm5uayY8eObr3Gjj0tLK7cxeKq3Syu2sXy6t00e/w4HYaThuYwZdQAzhyVx/j8fjijbARJT6gPXBLWIVOxHUk407MYOnIsn27Ywc6mVnY1tbLb7aGp1cfeVi/uVh9uj4+9rW0fPr/FZy0+v8Vv2z58/rahZ06HIcnpINlhcDoMyU4HSU5DksOQmuQkPcVJRkr75+Skfz9u/8hMTSI7LZns9GSy05LITk8Oytv6aO/372qG5SOPPHLIuX6/pWb3XlZvaWDN1kbWbG3gi5oGqna2fW+y03D80TlcP9HFpBH9mTQyl+y05LBdS6SoBS5xp9Xrp7KuiQ21e6jetZc585by/oKlkNGfpOwBODP7dvm9yU5DerKTjJQkMlLbgzfZibM9nB2m7bPTGIwxOAz4/BaP3+L1+fF2+Gxp8bb9AXB7fLhbfLT6utfiT092kpOeTHb6oeHe9nXb8awOj9uez0pLYvoLzx9xx5ho0OGPjKuAHz/4Swq/fiGVdU1U7XRTUeemqq6JDbVN7Gnx7v8+V24Gxx6Vzfj8vkxw9WPckJyY6xbpCXWhSNzx+y0VdU2sqKln1eYGNtTuYUNt2//4Pv+/f68zU5xkmla2f7WGxu2byHJ6ueyCc7jw7Cn0y0ihf2YK/TKTeWPmyzz4QGhbrF6ff3/L3t3qw93qpanFR2Ozh4ZmDw17vTTsbXtcv7f96wOfa/bQsNeD/0j/23pb8TU34W/Zg7/Fjb+lCX/zHjJTHNx287c7hH16spPUZCdp7e8W0pIdpCU5SUtuf5zsJMnx7z9YnY3UsNbi9Vs8Pj+t3vYPn58Wr5/G5rZramz27r/O+r0eahtb2N7YwvaGts91TS0cGEcOA0P6pVOQm8nwvEyOHZzN2KOyGD0oK+qG+YWaAlxiXm1jCwu+2sniql37Q3tfqyzF6WDEgExGDujDyAGZjBjQhxEDMnHlZpKdlnTE4WHd2eMwWlhr2dPibQ/Ef4f6vseNzV4e+PlDmNRMHKkZONL6tH1OzcSR2oeMvrm0eAPr+3cYcJi2dyQAHr+fnkSJw0Ben1QGZqfiaahj7dIF1G/bRE4K3HjVpdx81aUM6Zce0yNEgkkBLjFna30z8zbWMf+rncz/qo6NtU0ApCY5OHZwNicMyeGEITmMG5LDqEF9AvqfPd5Gqhzpelq8vv0t470eH80ePy0eH83etsfNngM+e334fBa/BUv7Z7vvXgBYCynOtr7/lCTH/s8pTgepyQ6y0pLaunrau3my0pLITEnC4TAx9YczkhTgEvV8fsuSql3MWbOd99fWsnpLAwBZqUmcOrw/pw3vz8Th/Rk3JCfoLbN4G6kSK8EYb3849wn2DWSNQpGo1Or188mGHcxatoX31mxjt9uD02GY4OrHfReNZcoxeRw7ODvkQ8CiadOAYIjkEqw9EY277QQqnBPH1AKXQ4R6+JnPb/m0PbTfXLmV+r0estKSOO/YQZwzdiBnjhpATkZ4h4DFSos13sRjCzwU19RVCxxrbdg+JkyYYCW6TZs2zWZkZFhg/0dGRoadNm1awK9dVddkH35rjZ380LvWde8se9wDb9g7/7nYvrNyq232eINQfc9NmzbNulwua4yxubm5Njc31xpjrMvlCso1y+GF8vctUowxHa5n34cxptevCSy0nWSqAlw6cLlcnf7yuVyuXr2ex+uzry2rsdc//pl13TvLFtw3y97wj/n2tWU1dm9rZEJ7n0iGx4F/OBL9j0W8/bcI9v9D1nYd4OpCkQ6CdTOv3u3hn59X8cynFWyub2ZI33SuLhzK1YXDGNI3PZgl91qk3r6ruya+heLfV6NQpFsCDbVNO92Uzd3I9EXV7PX4mDwil1umDOecsQOjbi2KSI08icd+X+koXKNQFODSQW9bDxU7mnjs/fXMWFKD0xguO/lobj6jgOOPzglH2b0SqSCNtyGLEnohGUZojLkL+A5tfTwrgJuttc2BvKZEVk+Hn22o3cNjc9Yzc2kNyU4H357s4rYzR3JUTlo4y+6VrhZTKi0tDenPjbchixJBnXWMd+cDGAJ8BaS3f/0icNPhvkc3MePHtoa99v4Zy+2I+1+3Y3462/5y1kq7rWFvpMvqsUjcQIvHkRcSWnRxEzPQiTxJQLoxxgNkAJsDfD2Jck0tXh7/aCNlczfS6vUz9bR8fnjuKPL6pEa6tF4pKioK+43DWJlkI9EvoD5wY8ydQCmwF3jbWnvIb6AxphgoBsjPz5/Q2VtHiX5+v+XlxdX85q211Da2cPEJR3HPBWMZnpcZ6dJE4l7Q+8CNMf2Ay4HhwG7gJWPMVGvttAPPs9aWAWXQdhOztz9PImfN1gYemPkFn1fsYnx+X/46dQITXP0iXZZIwgtkRaDzgK+stbXWWg8wAzg9OGVJNGhq8fLQ7NVc8sePWb99D7/5zxN5+fbTFd4HKC8vp6CgAIfDQUFBAeXl5ZEuSRJIIH3gVcAkY0wGbV0o5wIaIxgn3l+7nZ/MWMGW+maumziMH18wln6ZKZEuK6pot3uJtED7wH8GfAvwAkuA71hrW7o6X+PAo19Ds4fSWat5YeEmRg/qw6++eaJa3F3QhBwJl5CMA7fWPgg8GMhrSPT4aF0t905fztaGZu44ayR3njeK1KT43WcwUPG4FKrEFu1XJDR7fPx05gpu+McC0lOczLjjDH584diECe/e9mN3NfFGE3IkXBTgCW799kaueOwTps2r4jtThvP6f32Nk4d1vWt7vNnXj11ZWYm1dn8/dndCvLS0lIyMjA7HwjGTU2QfrYWSoKy1vLSomgf/tZL0FCcPX3MSZ48ZGOmywi7QfuxQb34hAlrMSg7Q1OKl5JUVzFy6mckjcvnDtSczKDv61y4JBS0sJbFAe2IKAJV1TRQ/s4h12xu567zR/OCcY6Jumddw0sJSEsvUBx4lwjEhZO6XtVz26Cdsa2zmmVtO487zRiV0eIP6sSW2KcCjQCA30rrDWkvZ3A3c9OQCBuek8er3pzBlVF5QXjvWFRUVUVZWhsvlwhiDy+XSzjgSM9QHHgVCOSGkxevj3unLmbl0MxefcBS/veokMlPVcyYSS9QHHsVCNSFkt7uV4mcXseCrnfzPf4zm+2cfgzGJ3WUiEk8U4FEgFDfSNu10c9OTC9i0cy9/vG48l510dCAlikgUUh94FOjsRpoxhosvvrhXr7eiup4r//wptY0tPHPrRIW3SJxSgEeBoqIibrzxxg7dG9Zann766R7fyPxg7Xa+VfYZqUkOZtxxOpNG5Aa7XBGJEgrwKDF79uxDJpS43W5KSkq6/RpvrNjCd59ZSEFuJq/ccTrHDMwKdpkiEkXUBx4lAr2R+fKiau6ZvoyTh/XlyZsnkpOeHMzyRCQKqQUeJQJZ2e7ZeZX86KVlTBqRy7O3nqbwFkkQCvAo0dsZgX/7cAMPzPyC844dyBM3nRq2Md7aSkwk8hTgUaI3MwL/+uEGfvXGGi49cTB/mTqBtOTwrN8d6pmjItI9mokZo/7x8Vf8YtYqLjvpaH7/rZNDsqZJV0ulaisxkfDSTMw48uxnFfxi1iouGncUv7vmpJCFd1cb9morMZHooC6UGPP8gioe+NdKzjt2II9cO54kZ2j+CUtKSvaH9z77hjVqKzGR6KAAjyGvLKnm/ldW8PXRA3is6BRSkkL3z3e4VraWYBWJDgrwGDFnzTb+56XlTB6Ry99umBDyDYcP18rWEqwi0UEBHgMWV+3ijvLFHDs4i7JvF4ZltMmRWtlFRUVUVFTg9/upqKhQeItEgAI8yq3f3sgtT33OoOw0nrxpIn3CNM5brWyR6KdhhFFsa30z3/zzJ7T6/Lz8vdNx5WZGuiQRiYCuhhGqBR4lDp7Z+PdnnuPGJxZQv9fDUzdPVHiLyCE0DjwKHDLmelM1//tmJemubJ6+ZRLjhuREuEIRiUZqgUeBg8dc9z/vNlLyT8Q3b5o2HxaRLgUU4MaYvsaY6caYNcaY1caYycEqLJEcOOY6a8I3yBp/MfXzplP14YsRrEpEol2gLfBHgDettWOBk4DVgZeUePaNuU4bMYF+53wH95efsfvDpzWzUUQOq9cBbozJAc4E/gFgrW211u4OVmHRKFRLqJaWlpI1dAwDLrsXT20FO2Y9TEZGumY2ishhBXITczhQCzxpjDkJWATcaa1tOvAkY0wxUAyxvVbG4RZ3CnRs9CVXXs3INVnsamik9uVfkH/0oP0r/4mIdKXX48CNMYXAPOAMa+18Y8wjQIO19oGuvieWx4GHaglVn99yy1Of89mGOl64bRLj8/sFUKWIxKNQjAOvBqqttfPbv54OnBLA60W1UC2h+si7X/Lhl7U8eNlxCm8R6ZFeB7i1diuwyRgzpv3QucCqoFQVhUKxhOq7q7bxxznruXrCUK6fmK9tykSkRwIdhfJDoNwYsxw4GXgo8JKiU7CXUK3Y0cRdLy5l3JBsfnHFOJ577jltUyYiPaK1UHqgqy3Gesrd6uXKxz5lW2Mzr/1gCsP6Z2ibMhHpUld94ArwCLj7xaW8sqSGp2+eyJmjBwDgcDjo7N/CGIPf7w93iSISRbSYVZR4eVE1MxbX8F/njNof3hCaPnYRiW8K8DDaWLuHB/71BcMzvTxSfEGHm5XapkxEekoBHiYtXh8/eG4J+Dws/NP3qayo6HCzEtAGCiLSI1pONkx+NXsNq7Y0wId/ZU9tTYfn9u32rq3JRKQn1AIPg7dXbuWpTyu4dcpwqua/0ek5gU4IEpHEowAPsa31zfz45eWcMCSHH184RjcrRSRoFODd1JtZktZa7pm+jBaPnz9eN57UJKduVopI0CjAu2HfSoQ9nSU5bV4lH63bQcklxzI8r21PS+32LiLBook83XCkWZKdzdA8/YIruOiRuZw2PJenbj4VY0wEKheReNDVRB6NQumGw61E2Ok64bfdzin39CE1KZ3fXHWiwltEQkJdKN1wuBuPB29IDJB04sVs2pvEL68Yx6DstHCUKCIJSAHeDYe78Xhw6zx54Aj6nnE9Tas/5BsnHR3OMkUkwSjAu+FwNx47tM4dSeRdchc+dz2Zq2dFrmARSQgK8G4qKiqioqICv9/fYcbkga3znElXkTJwOE0f/J3SB0siWa6IJADdxAzQviAv+fWfsKd/Cyo/59Gf3KZhgSIScgrwILj2uuuZXl9AxY4m3n38fnL7pEa6JBFJAOpCCYKnP61gSdVuHvzG8QpvEQkbBXiANu1089u31nL2mAFcfrJGnYhI+MR1gId6l3drLffPWIHDQOmVJ2jCjoiEVdz2gXc6Q7J944Rg3WB8ZUkNH6/fwS8uP56j+6YH5TVFRLorblvgnc2Q3LdxQjDUuz08NHs14/P7UnSaKyivKSLSE3HbAj/c+iXB8P/eXsvOplaevmUiDoe6TkQk/OK2BR7KjROWV+9m2vxKbjy9gOOPzgn49UREeiNuAzxUGyf4/JafzvyCAX1Sufv80QG9lohIIOI2wEO1ccJzC6pYXl3PTy89jqy05CBVKyLSc9rQoQdqG1s45+EPOHFoDtNuPU3DBkUkLLra0CFuW+Ch8Ks3VtPs8fHzy8cpvEUk4gIOcGOM0xizxBgT1+unLq7axYzFNRSfOYKRA/pEuhwRkaC0wO8EVgfhdaKW32/52WurGJiVyh1nHRPpckREgAAD3BgzFLgE+HtwyolOryypYdmm3dx30VgyU+N26LyIxJhAW+B/AH4M+Ls6wRhTbIxZaIxZWFtbG+CPC78nninnR898SMvmtdz9za8FfT0VEZHe6nWAG2MuBbZbaxcd7jxrbZm1ttBaWzhgwIDe/riIKC8v596n34P0vux8r4zKygqKi4sV4iISFQJpgZ8BXGaMqQCeB84xxkwLSlVRouShh8kY/w32rHyf1s1rgeCupyIiEoheB7i19n5r7VBrbQFwLTDHWjs1aJVFAfeoC7B+P7s/fKrD8WCtpyIiEgiNA+/C/I11ZIydQsP86fga6zo8F4z1VEREAhWUALfWfmCtvTQYrxUNrLU8NHs1Ocl+PF+82eG5YKynIiISDGqBd+L1FVtYVl3PA1eMp+zPjwZ9PRURkWDQoOaDtHr9/PattYw9Kosrxw/BOaFIgS0iUUkt8IM8N7+Syjo39140Fqc2ahCRKKYAP0Bjs4c/zlnP5BG5nDU6tsasi0jiUYAfoGzuRnY2tXL/xWO12qCIRD0FeLvtDc38/aOvuPTEwZw4tG+kyxEROSIFeLvfv7sOr9/PPReMiXQpIiLdogAHNtbu4cWFmyg6zYUrNzPS5YiIdIsCHPjDu+tIcTr4wTla61tEYkfCB/jarY28tnwzN51RQF6f1EiXIyLSbQkf4L9/50v6pCRx25kjIl2KiEiPJHSAr6iu582VW7n1a8Ppm5ES6XJERHokoQP8d++sJSc9mVumDI90KSIiPZawAb6ochfvr63ltq+PIDstOdLliIj0WMIG+O/eWUtuZgo3Ti6IdCkiIr2SkAH+2YY6Pllfx/fOGqld5kUkZiVkgP/+3S8ZlJ3K1EmuSJciItJrCRfg8zfWseCrndz+9ZGkJTsjXY6ISK8lXID/ac568vqkcO2p2tdSRGJbQgX44qpdfLx+B9/92gjSU9T6FpHYllAB/uic9fTLSFbft4jEhYQJ8C9q6pmzZju3ThmukSciEhcSJsD/NGcdWWlJfPv0gkiXIiISFAkR4Gu3NvLWym3cfHqBZl2KSNxIiAB/9P31ZKY4teaJiMSVuA/wjbV7mLV8MzdMLtCKgyISV+I+wMvmbiTF6eA7X1PrW0TiS1wH+PaGZmYsruHqwqHabUdE4k6vA9wYM8wY874xZpUxZqUx5s5gFhYMT3xSgdfv57tf0247IhJ/AhkQ7QV+ZK1dbIzJAhYZY96x1q4KUm0BaWz2UD6vkotOGKyd5kUkLvW6BW6t3WKtXdz+uBFYDQwJVmGBem5+FY0tXm4/c2SkSxERCYmg9IEbYwqA8cD8YLxeoFq8Ph59ZxVsXcNJ+f0oKCigvLw80mWJiARVwAFujOkDvAz8t7W2oZPni40xC40xC2trawP9cd1y319m0Oh1sO3Dcqy1VFZWMnXqVPLy8hTkIhI3AgpwY0wybeFdbq2d0dk51toya22htbZwwIABgfy4bvH7La+saqB12waaK5Z0eK6uro7i4mKFuIjEhUBGoRjgH8Bqa+3vgldS18rLyykoKMDhcHTZLfLu6m2QcxT181/u9DXcbjclJSWhLlVEJOQCGYVyBnADsMIYs7T92E+stbMDL+tQ5eXlFBcX43a7AaisrKS4uBiAoqKi/ef9be5G2LMD95qPu3ytqqqqUJQoIhJWgYxC+dhaa6y1J1prT27/CEl4A5SUlOwP730Obk0vqdrFospdXDIqg4z0tC5fKz9fu/GISOyLmZmYXbWaDzz+xCcVZKUm8evbr6SsrIzc3NxDzs/IyKC0tDRkdYqIhEvMBHhXreZ9x//81HO8tmQTNR+/xLgxxwCwY8cOpk2bhsvlwhiDy+WirKysQ5eLiEisMtbasP2wwsJCu3Dhwl5978F94NDWmi4rKwPgrqc+IH38N6j523fxNWzf/5zCWkRinTFmkbW28ODjMdMCLyoqoqysrNPWdMn//ozU487F/eVn+Bq2AxptIiLxL2YCHNpCvKKiAr/fT2lpKSUlJTgcDuqyj8GZnkXjwn91OF+jTUQknsXk7r4du1MM2YWX07L5S1pqVnc4T6NNRCSexVQLfJ8DhxSmjZhAcu5QGg5qfWu0iYjEu5gM8AO7RrJPvRxv4w7ca9sm7mi0iYgkipgM8H1dI8l5LtILxtO4aBb4fbhcLp599lkAbrjhBq1CKCJxLSYDvLS0lIyMDLImXIrf08yeZW+RkZHBxRdfTHFxMZWVlftXIdTiVSISr2IywIuKinjkL4+TNe4c3KvnMmxQf8rKypg9e/YRp9uLiMSLmAzw8vJyfjntHUhKpc/mRZSWllJUVNSt6fYiIvEi5gK8bQjhbXgKJtNcs5rKZZ9wyy23kJeXR1ezSjWcUETiUcwFeElJCf6Bo0nuP4Q9i18HoLW1lbq6uk7P13BCEYlXMRfgVVVVZJ1yCb6mXTSt7XrNb0DDCUUkrsXcTMz8sSdjR55Kw7yXwOft8jxjDBUVFeErTEQkzGKuBX7mrSWAoXHpm4c9T/3eIhLvYirAmz0+VjRlcZRvO/49nfd5g/q9RSQxxFSAv758C7vcHrZ9/GKXI07U7y0iiSKm+sB/P2sR1O+gcsHbnT6vfm8RSSQx0wL/7d+fp3pvEjsXzOzyHPV7i0giiZkAf+yNpfg9LexZ+X6nz6vfW0QSTUwEeFOLF3/+BNxrPsK2NB3yvPq9RSQRxUQf+GvLNuNIzeh06KDL5VK/t4gkpJhogf9zQRWDUn0k7e64KJW6TUQkkUV9gH9RU8+y6nq+9x8ndLkrvYhIIor6LpTnP68iNcnBleOHknNGkQJbRKRdVLfA3a1eZi7ZzCUnDiYnIznS5YiIRJWoDvBZy7awp8XL9RM1vltE5GABBbgx5kJjzFpjzHpjzH3BKmqf5xZUMWpgHya4+gX7pUVEYl6vA9wY4wQeAy4CjgOuM8YcF6zCVm1uYOmm3Vw3MR9jTLBeVkQkbgTSAp8IrLfWbrTWtgLPA5cHp6y2m5cpSQ6+ecqQ/cfKy8spKCjA4VMZyY8AAAVpSURBVHBQUFCg3eZFJKEFMgplCLDpgK+rgdMOPskYUwwUQ/fXKmn2+Ji5pIaLxh1F34wUYN9emMX7d52vrKykuLgYQCNTRCQhhfwmprW2zFpbaK0tHDBgQLe+5+1V22ho9nJN4bD9x0pKSvaH9z5ut5uSkpKg1isiEisCCfAaYNgBXw9tPxawlxZuYkjfdCaPyN1/rLKystNzuzouIhLvAgnwz4FRxpjhxpgU4Frg1UALqtm9l4/X7+A/JwzF4fj3zUun09np+V0dFxGJd73uA7fWeo0xPwDeApzAE9balYEWNGNRNdbC1ROGdjju8/k6Pb+r4yIi8S6gqfTW2tnA7CDVgt9veWlRNZNH5DKsf0aH51wuV6fdJS6XK1g/XkQkpkTVTMwFFTup2unmmlOHHvJcaWkpGRkdQ12rEYpIIouqAH9x4SayUpO48PjBhzxXVFSk1QhFRA4QNasRNjZ7eGPFVq4YP4T0lM5vTBYVaTVCEZF9oqYF/vryLez1+Li68NDuExEROVTUBPhLi6oZOSCT8cP6RroUEZGYEBUBvqF2D4sqd3FN4TAtXCUi0k1REeCvLK7BYeDK8UOOfLKIiABREOB+v+WVJTVMGTWAgdlpkS5HRCRmGGtt+H6YMbVAMBYvyQN2BOF1YlUiX38iXzvo+hP1+l3W2kNWAwxrgAeLMWahtbYw0nVESiJffyJfO+j6E/36DxbxLhQREekdBbiISIyK1QAvi3QBEZbI15/I1w66/kS//g5isg9cRERitwUuIpLwFOAiIjEqqgPcGHOhMWatMWa9Mea+Tp5PNca80P78fGNMQfirDI1uXPvdxphVxpjlxpj3jDFxtbPFka7/gPP+0xhjjTFxNbSsO9dvjLmm/XdgpTHmuXDXGCrd+N3PN8a8b4xZ0v77f3Ek6owK1tqo/KBtm7YNwAggBVgGHHfQOXcAf21/fC3wQqTrDuO1nw1ktD/+Xrxce3evv/28LGAuMA8ojHTdYf73HwUsAfq1fz0w0nWH8drLgO+1Pz4OqIh03ZH6iOYW+ERgvbV2o7W2FXgeuPygcy4Hnm5/PB0418THalhHvHZr7fvWWnf7l/OAeFqHtzv/9gC/AH4NNIezuDDozvV/F3jMWrsLwFq7Pcw1hkp3rt0C2e2Pc4DNYawvqkRzgA8BNh3wdXX7sU7PsdZ6gXogNyzVhVZ3rv1AtwJvhLSi8Dri9RtjTgGGWWtfD2dhYdKdf//RwGhjzCfGmHnGmAvDVl1odefa/w8w1RhTTduevD8MT2nRJ2p25JHeMcZMBQqBr0e6lnAxxjiA3wE3RbiUSEqirRvlLNrefc01xpxgrd0d0arC4zrgKWvtw8aYycCzxphx1lp/pAsLt2hugdcAww74emj7sU7PMcYk0fZ2qi4s1YVWd64dY8x5QAlwmbW2JUy1hcORrj8LGAd8YIypACYBr8bRjczu/PtXA69aaz3W2q+AL2kL9FjXnWu/FXgRwFr7GZBG2yJXCSeaA/xzYJQxZrgxJoW2m5SvHnTOq8CN7Y+vAubY9jsbMe6I126MGQ/8jbbwjpf+z30Oe/3W2nprbZ61tsBaW0DbPYDLrLULI1Nu0HXnd38mba1vjDF5tHWpbAxnkSHSnWuvAs4FMMYcS1uA14a1yigRtQHe3qf9A+AtYDXworV2pTHm58aYy9pP+weQa4xZD9wNdDncLJZ089p/C/QBXjLGLDXGHPxLHrO6ef1xq5vX/xZQZ4xZBbwP3GOtjfl3n9289h8B3zXGLAP+CdwUJw23HtNUehGRGBW1LXARETk8BbiISIxSgIuIxCgFuIhIjFKAi4jEKAW4iEiMUoCLiMSo/w+gUBhgMKgtQAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.scatter(X.ravel(), y, color='black')\n",
    "lim = plt.axis()\n",
    "y_test = PolynomialRegression(3).fit(X, y).predict(X_test)\n",
    "plt.plot(X_test.ravel(), y_test)\n",
    "plt.axis(lim)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}