{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "41049409",
   "metadata": {
    "colab_type": "text",
    "id": "view-in-github"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/Nabizeus/TOAR_Julich/blob/main/GettingStarted_Nabiz_TOAR_Temp.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "vEcmNvugBhx0",
   "metadata": {
    "id": "vEcmNvugBhx0",
    "tags": []
   },
   "source": [
    "# **Getting started with TOAR-II data**[<img style=\"float: right;\" src=\"2022-02-Logo-TOAR-DataBase.png\" alt=\"\" width=\"200\"/>](2022-02-Logo-TOAR-DataBase.png)\n",
    "(by Niklas Selke and Sabine Schröder)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "mFEn7BZOBhx2",
   "metadata": {
    "id": "mFEn7BZOBhx2"
   },
   "source": [
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "n0quNf4HBhx2",
   "metadata": {
    "id": "n0quNf4HBhx2"
   },
   "source": [
    "**This Python notebook follows the examples from the [TOAR Quick Start Guide](https://toar-data.fz-juelich.de/sphinx/TOAR_UG_Vol02_Quick_Start/build/html/examples.html).**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "z-SpWqenBhx3",
   "metadata": {
    "id": "z-SpWqenBhx3"
   },
   "source": [
    "## **Set up kernel (only needed the very first time)**\n",
    "When you execute the notebook for the very first time please go to the next cell and press **y** to enable the cell.  \n",
    "After that, press **Ctrl+Enter** to execute the cell (takes about a minute).  \n",
    "Finally, after execution has finished, disable the cell again by pressing **r**. We do not need this cell again.\n",
    "\n",
    "For all notebooks please choose the **TOARWorkshop** kernel in the top right corner which is now available."
   ]
  },
  {
   "cell_type": "raw",
   "id": "tM9KfjksBhx4",
   "metadata": {
    "id": "tM9KfjksBhx4"
   },
   "source": [
    "!bash setup_kernel.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0g2X4BsEBhx4",
   "metadata": {
    "id": "0g2X4BsEBhx4"
   },
   "source": [
    "## Import Python packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "_LRYLOlRBhx5",
   "metadata": {
    "id": "_LRYLOlRBhx5"
   },
   "outputs": [],
   "source": [
    "from io import StringIO\n",
    "import json\n",
    "import pprint as pp\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "import pandas as pd\n",
    "import requests"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "Lzub1iLhBhx6",
   "metadata": {
    "id": "Lzub1iLhBhx6"
   },
   "source": [
    "**[StringIO](https://docs.python.org/3/library/io.html#io.StringIO)**: creates a file-like object from a simple string  \n",
    "**[json](https://docs.python.org/3/library/json.html)**: module for processing JSON formatted files and strings  \n",
    "**[pprint](https://docs.python.org/3/library/pprint.html)**: module for formatted print outputs (see also [pprint.ipynb](pprint.ipynb))  \n",
    "**[matplotlib](https://matplotlib.org/)**: package for plotting  \n",
    "**[pandas](https://pandas.pydata.org/)**: package for data analysis and manipulation  \n",
    "**[requests](https://requests.readthedocs.io)**: package for sending HTTPS requests"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "n0cZVzzkBhx7",
   "metadata": {
    "id": "n0cZVzzkBhx7"
   },
   "source": [
    "### 1. Find stations by country\n",
    "\n",
    "The TOAR-II database stores all countries in [ISO-3166 ALPHA-2 codes](https://en.wikipedia.org/wiki/ISO_3166-1#Current_codes).  \n",
    "To search for all stations of a specific country, you use the argument _country_ and its ALPHA-2 code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "oLxrqtW0Bhx8",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 520
    },
    "id": "oLxrqtW0Bhx8",
    "outputId": "7be28a8e-cc31-4e47-baaf-0c42db4f3698",
    "tags": []
   },
   "outputs": [
    {
     "ename": "ReadTimeout",
     "evalue": "HTTPSConnectionPool(host='toar-data.fz-juelich.de', port=443): Read timed out. (read timeout=10)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mtimeout\u001b[0m                                   Traceback (most recent call last)",
      "File \u001b[0;32m~/anaconda3/lib/python3.8/site-packages/urllib3/connectionpool.py:466\u001b[0m, in \u001b[0;36mHTTPConnectionPool._make_request\u001b[0;34m(self, conn, method, url, timeout, chunked, **httplib_request_kw)\u001b[0m\n\u001b[1;32m    462\u001b[0m         \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mBaseException\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[1;32m    463\u001b[0m             \u001b[38;5;66;03m# Remove the TypeError from the exception chain in\u001b[39;00m\n\u001b[1;32m    464\u001b[0m             \u001b[38;5;66;03m# Python 3 (including for exceptions like SystemExit).\u001b[39;00m\n\u001b[1;32m    465\u001b[0m             \u001b[38;5;66;03m# Otherwise it looks like a bug in the code.\u001b[39;00m\n\u001b[0;32m--> 466\u001b[0m             \u001b[43msix\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mraise_from\u001b[49m\u001b[43m(\u001b[49m\u001b[43me\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;28;43;01mNone\u001b[39;49;00m\u001b[43m)\u001b[49m\n\u001b[1;32m    467\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m (SocketTimeout, BaseSSLError, SocketError) \u001b[38;5;28;01mas\u001b[39;00m e:\n",
      "File \u001b[0;32m<string>:3\u001b[0m, in \u001b[0;36mraise_from\u001b[0;34m(value, from_value)\u001b[0m\n",
      "File \u001b[0;32m~/anaconda3/lib/python3.8/site-packages/urllib3/connectionpool.py:461\u001b[0m, in \u001b[0;36mHTTPConnectionPool._make_request\u001b[0;34m(self, conn, method, url, timeout, chunked, **httplib_request_kw)\u001b[0m\n\u001b[1;32m    460\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m--> 461\u001b[0m     httplib_response \u001b[38;5;241m=\u001b[39m \u001b[43mconn\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mgetresponse\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    462\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mBaseException\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[1;32m    463\u001b[0m     \u001b[38;5;66;03m# Remove the TypeError from the exception chain in\u001b[39;00m\n\u001b[1;32m    464\u001b[0m     \u001b[38;5;66;03m# Python 3 (including for exceptions like SystemExit).\u001b[39;00m\n\u001b[1;32m    465\u001b[0m     \u001b[38;5;66;03m# Otherwise it looks like a bug in the code.\u001b[39;00m\n",
      "File \u001b[0;32m~/anaconda3/lib/python3.8/http/client.py:1348\u001b[0m, in \u001b[0;36mHTTPConnection.getresponse\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m   1347\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m-> 1348\u001b[0m     \u001b[43mresponse\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mbegin\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m   1349\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mConnectionError\u001b[39;00m:\n",
      "File \u001b[0;32m~/anaconda3/lib/python3.8/http/client.py:316\u001b[0m, in \u001b[0;36mHTTPResponse.begin\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    315\u001b[0m \u001b[38;5;28;01mwhile\u001b[39;00m \u001b[38;5;28;01mTrue\u001b[39;00m:\n\u001b[0;32m--> 316\u001b[0m     version, status, reason \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_read_status\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    317\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m status \u001b[38;5;241m!=\u001b[39m CONTINUE:\n",
      "File \u001b[0;32m~/anaconda3/lib/python3.8/http/client.py:277\u001b[0m, in \u001b[0;36mHTTPResponse._read_status\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    276\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21m_read_status\u001b[39m(\u001b[38;5;28mself\u001b[39m):\n\u001b[0;32m--> 277\u001b[0m     line \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mstr\u001b[39m(\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mfp\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mreadline\u001b[49m\u001b[43m(\u001b[49m\u001b[43m_MAXLINE\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m+\u001b[39;49m\u001b[43m \u001b[49m\u001b[38;5;241;43m1\u001b[39;49m\u001b[43m)\u001b[49m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124miso-8859-1\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[1;32m    278\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mlen\u001b[39m(line) \u001b[38;5;241m>\u001b[39m _MAXLINE:\n",
      "File \u001b[0;32m~/anaconda3/lib/python3.8/socket.py:669\u001b[0m, in \u001b[0;36mSocketIO.readinto\u001b[0;34m(self, b)\u001b[0m\n\u001b[1;32m    668\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m--> 669\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_sock\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrecv_into\u001b[49m\u001b[43m(\u001b[49m\u001b[43mb\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    670\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m timeout:\n",
      "File \u001b[0;32m~/anaconda3/lib/python3.8/ssl.py:1274\u001b[0m, in \u001b[0;36mSSLSocket.recv_into\u001b[0;34m(self, buffer, nbytes, flags)\u001b[0m\n\u001b[1;32m   1271\u001b[0m         \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(\n\u001b[1;32m   1272\u001b[0m           \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mnon-zero flags not allowed in calls to recv_into() on \u001b[39m\u001b[38;5;132;01m%s\u001b[39;00m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;241m%\u001b[39m\n\u001b[1;32m   1273\u001b[0m           \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m\u001b[38;5;18m__class__\u001b[39m)\n\u001b[0;32m-> 1274\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mread\u001b[49m\u001b[43m(\u001b[49m\u001b[43mnbytes\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mbuffer\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m   1275\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n",
      "File \u001b[0;32m~/anaconda3/lib/python3.8/ssl.py:1132\u001b[0m, in \u001b[0;36mSSLSocket.read\u001b[0;34m(self, len, buffer)\u001b[0m\n\u001b[1;32m   1131\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m buffer \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[0;32m-> 1132\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_sslobj\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mread\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mlen\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mbuffer\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m   1133\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n",
      "\u001b[0;31mtimeout\u001b[0m: The read operation timed out",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[0;31mReadTimeoutError\u001b[0m                          Traceback (most recent call last)",
      "File \u001b[0;32m~/anaconda3/lib/python3.8/site-packages/requests/adapters.py:486\u001b[0m, in \u001b[0;36mHTTPAdapter.send\u001b[0;34m(self, request, stream, timeout, verify, cert, proxies)\u001b[0m\n\u001b[1;32m    485\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m--> 486\u001b[0m     resp \u001b[38;5;241m=\u001b[39m \u001b[43mconn\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43murlopen\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m    487\u001b[0m \u001b[43m        \u001b[49m\u001b[43mmethod\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mrequest\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mmethod\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    488\u001b[0m \u001b[43m        \u001b[49m\u001b[43murl\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43murl\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    489\u001b[0m \u001b[43m        \u001b[49m\u001b[43mbody\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mrequest\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mbody\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    490\u001b[0m \u001b[43m        \u001b[49m\u001b[43mheaders\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mrequest\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mheaders\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    491\u001b[0m \u001b[43m        \u001b[49m\u001b[43mredirect\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mFalse\u001b[39;49;00m\u001b[43m,\u001b[49m\n\u001b[1;32m    492\u001b[0m \u001b[43m        \u001b[49m\u001b[43massert_same_host\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mFalse\u001b[39;49;00m\u001b[43m,\u001b[49m\n\u001b[1;32m    493\u001b[0m \u001b[43m        \u001b[49m\u001b[43mpreload_content\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mFalse\u001b[39;49;00m\u001b[43m,\u001b[49m\n\u001b[1;32m    494\u001b[0m \u001b[43m        \u001b[49m\u001b[43mdecode_content\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mFalse\u001b[39;49;00m\u001b[43m,\u001b[49m\n\u001b[1;32m    495\u001b[0m \u001b[43m        \u001b[49m\u001b[43mretries\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mmax_retries\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    496\u001b[0m \u001b[43m        \u001b[49m\u001b[43mtimeout\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtimeout\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    497\u001b[0m \u001b[43m        \u001b[49m\u001b[43mchunked\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mchunked\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    498\u001b[0m \u001b[43m    \u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    500\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m (ProtocolError, \u001b[38;5;167;01mOSError\u001b[39;00m) \u001b[38;5;28;01mas\u001b[39;00m err:\n",
      "File \u001b[0;32m~/anaconda3/lib/python3.8/site-packages/urllib3/connectionpool.py:798\u001b[0m, in \u001b[0;36mHTTPConnectionPool.urlopen\u001b[0;34m(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)\u001b[0m\n\u001b[1;32m    796\u001b[0m     e \u001b[38;5;241m=\u001b[39m ProtocolError(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mConnection aborted.\u001b[39m\u001b[38;5;124m\"\u001b[39m, e)\n\u001b[0;32m--> 798\u001b[0m retries \u001b[38;5;241m=\u001b[39m \u001b[43mretries\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mincrement\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m    799\u001b[0m \u001b[43m    \u001b[49m\u001b[43mmethod\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43murl\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43merror\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43me\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m_pool\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m_stacktrace\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43msys\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mexc_info\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[43m[\u001b[49m\u001b[38;5;241;43m2\u001b[39;49m\u001b[43m]\u001b[49m\n\u001b[1;32m    800\u001b[0m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    801\u001b[0m retries\u001b[38;5;241m.\u001b[39msleep()\n",
      "File \u001b[0;32m~/anaconda3/lib/python3.8/site-packages/urllib3/util/retry.py:550\u001b[0m, in \u001b[0;36mRetry.increment\u001b[0;34m(self, method, url, response, error, _pool, _stacktrace)\u001b[0m\n\u001b[1;32m    549\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m read \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mFalse\u001b[39;00m \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_is_method_retryable(method):\n\u001b[0;32m--> 550\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[43msix\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mreraise\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mtype\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43merror\u001b[49m\u001b[43m)\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43merror\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m_stacktrace\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    551\u001b[0m \u001b[38;5;28;01melif\u001b[39;00m read \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n",
      "File \u001b[0;32m~/anaconda3/lib/python3.8/site-packages/urllib3/packages/six.py:770\u001b[0m, in \u001b[0;36mreraise\u001b[0;34m(tp, value, tb)\u001b[0m\n\u001b[1;32m    769\u001b[0m         \u001b[38;5;28;01mraise\u001b[39;00m value\u001b[38;5;241m.\u001b[39mwith_traceback(tb)\n\u001b[0;32m--> 770\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m value\n\u001b[1;32m    771\u001b[0m \u001b[38;5;28;01mfinally\u001b[39;00m:\n",
      "File \u001b[0;32m~/anaconda3/lib/python3.8/site-packages/urllib3/connectionpool.py:714\u001b[0m, in \u001b[0;36mHTTPConnectionPool.urlopen\u001b[0;34m(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)\u001b[0m\n\u001b[1;32m    713\u001b[0m \u001b[38;5;66;03m# Make the request on the httplib connection object.\u001b[39;00m\n\u001b[0;32m--> 714\u001b[0m httplib_response \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_make_request\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m    715\u001b[0m \u001b[43m    \u001b[49m\u001b[43mconn\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    716\u001b[0m \u001b[43m    \u001b[49m\u001b[43mmethod\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    717\u001b[0m \u001b[43m    \u001b[49m\u001b[43murl\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    718\u001b[0m \u001b[43m    \u001b[49m\u001b[43mtimeout\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtimeout_obj\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    719\u001b[0m \u001b[43m    \u001b[49m\u001b[43mbody\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mbody\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    720\u001b[0m \u001b[43m    \u001b[49m\u001b[43mheaders\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mheaders\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    721\u001b[0m \u001b[43m    \u001b[49m\u001b[43mchunked\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mchunked\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    722\u001b[0m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    724\u001b[0m \u001b[38;5;66;03m# If we're going to release the connection in ``finally:``, then\u001b[39;00m\n\u001b[1;32m    725\u001b[0m \u001b[38;5;66;03m# the response doesn't need to know about the connection. Otherwise\u001b[39;00m\n\u001b[1;32m    726\u001b[0m \u001b[38;5;66;03m# it will also try to release it and we'll have a double-release\u001b[39;00m\n\u001b[1;32m    727\u001b[0m \u001b[38;5;66;03m# mess.\u001b[39;00m\n",
      "File \u001b[0;32m~/anaconda3/lib/python3.8/site-packages/urllib3/connectionpool.py:468\u001b[0m, in \u001b[0;36mHTTPConnectionPool._make_request\u001b[0;34m(self, conn, method, url, timeout, chunked, **httplib_request_kw)\u001b[0m\n\u001b[1;32m    467\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m (SocketTimeout, BaseSSLError, SocketError) \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[0;32m--> 468\u001b[0m     \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_raise_timeout\u001b[49m\u001b[43m(\u001b[49m\u001b[43merr\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43me\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43murl\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43murl\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mtimeout_value\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mread_timeout\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    469\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m\n",
      "File \u001b[0;32m~/anaconda3/lib/python3.8/site-packages/urllib3/connectionpool.py:357\u001b[0m, in \u001b[0;36mHTTPConnectionPool._raise_timeout\u001b[0;34m(self, err, url, timeout_value)\u001b[0m\n\u001b[1;32m    356\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(err, SocketTimeout):\n\u001b[0;32m--> 357\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m ReadTimeoutError(\n\u001b[1;32m    358\u001b[0m         \u001b[38;5;28mself\u001b[39m, url, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mRead timed out. (read timeout=\u001b[39m\u001b[38;5;132;01m%s\u001b[39;00m\u001b[38;5;124m)\u001b[39m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;241m%\u001b[39m timeout_value\n\u001b[1;32m    359\u001b[0m     )\n\u001b[1;32m    361\u001b[0m \u001b[38;5;66;03m# See the above comment about EAGAIN in Python 3. In Python 2 we have\u001b[39;00m\n\u001b[1;32m    362\u001b[0m \u001b[38;5;66;03m# to specifically catch it and throw the timeout error\u001b[39;00m\n",
      "\u001b[0;31mReadTimeoutError\u001b[0m: HTTPSConnectionPool(host='toar-data.fz-juelich.de', port=443): Read timed out. (read timeout=10)",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[0;31mReadTimeout\u001b[0m                               Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[3], line 2\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[38;5;66;03m# provides all stations from Andorra(AD) and Gibraltar(GI)\u001b[39;00m\n\u001b[0;32m----> 2\u001b[0m resp \u001b[38;5;241m=\u001b[39m \u001b[43mrequests\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mget\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mhttps://toar-data.fz-juelich.de/api/v2/stationmeta/?country=DE&limit=None\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mtimeout\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m3.05\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m10\u001b[39;49m\u001b[43m)\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m      3\u001b[0m stations \u001b[38;5;241m=\u001b[39m resp\u001b[38;5;241m.\u001b[39mjson()\n\u001b[1;32m      5\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m ind \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28mrange\u001b[39m(\u001b[38;5;241m0\u001b[39m,\u001b[38;5;241m1\u001b[39m):\n",
      "File \u001b[0;32m~/anaconda3/lib/python3.8/site-packages/requests/api.py:73\u001b[0m, in \u001b[0;36mget\u001b[0;34m(url, params, **kwargs)\u001b[0m\n\u001b[1;32m     62\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mget\u001b[39m(url, params\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mNone\u001b[39;00m, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs):\n\u001b[1;32m     63\u001b[0m \u001b[38;5;250m    \u001b[39m\u001b[38;5;124mr\u001b[39m\u001b[38;5;124;03m\"\"\"Sends a GET request.\u001b[39;00m\n\u001b[1;32m     64\u001b[0m \n\u001b[1;32m     65\u001b[0m \u001b[38;5;124;03m    :param url: URL for the new :class:`Request` object.\u001b[39;00m\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m     70\u001b[0m \u001b[38;5;124;03m    :rtype: requests.Response\u001b[39;00m\n\u001b[1;32m     71\u001b[0m \u001b[38;5;124;03m    \"\"\"\u001b[39;00m\n\u001b[0;32m---> 73\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mrequest\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mget\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43murl\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mparams\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mparams\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/anaconda3/lib/python3.8/site-packages/requests/api.py:59\u001b[0m, in \u001b[0;36mrequest\u001b[0;34m(method, url, **kwargs)\u001b[0m\n\u001b[1;32m     55\u001b[0m \u001b[38;5;66;03m# By using the 'with' statement we are sure the session is closed, thus we\u001b[39;00m\n\u001b[1;32m     56\u001b[0m \u001b[38;5;66;03m# avoid leaving sockets open which can trigger a ResourceWarning in some\u001b[39;00m\n\u001b[1;32m     57\u001b[0m \u001b[38;5;66;03m# cases, and look like a memory leak in others.\u001b[39;00m\n\u001b[1;32m     58\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m sessions\u001b[38;5;241m.\u001b[39mSession() \u001b[38;5;28;01mas\u001b[39;00m session:\n\u001b[0;32m---> 59\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43msession\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrequest\u001b[49m\u001b[43m(\u001b[49m\u001b[43mmethod\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mmethod\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43murl\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43murl\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/anaconda3/lib/python3.8/site-packages/requests/sessions.py:589\u001b[0m, in \u001b[0;36mSession.request\u001b[0;34m(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)\u001b[0m\n\u001b[1;32m    584\u001b[0m send_kwargs \u001b[38;5;241m=\u001b[39m {\n\u001b[1;32m    585\u001b[0m     \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mtimeout\u001b[39m\u001b[38;5;124m\"\u001b[39m: timeout,\n\u001b[1;32m    586\u001b[0m     \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mallow_redirects\u001b[39m\u001b[38;5;124m\"\u001b[39m: allow_redirects,\n\u001b[1;32m    587\u001b[0m }\n\u001b[1;32m    588\u001b[0m send_kwargs\u001b[38;5;241m.\u001b[39mupdate(settings)\n\u001b[0;32m--> 589\u001b[0m resp \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43msend\u001b[49m\u001b[43m(\u001b[49m\u001b[43mprep\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43msend_kwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    591\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m resp\n",
      "File \u001b[0;32m~/anaconda3/lib/python3.8/site-packages/requests/sessions.py:703\u001b[0m, in \u001b[0;36mSession.send\u001b[0;34m(self, request, **kwargs)\u001b[0m\n\u001b[1;32m    700\u001b[0m start \u001b[38;5;241m=\u001b[39m preferred_clock()\n\u001b[1;32m    702\u001b[0m \u001b[38;5;66;03m# Send the request\u001b[39;00m\n\u001b[0;32m--> 703\u001b[0m r \u001b[38;5;241m=\u001b[39m \u001b[43madapter\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43msend\u001b[49m\u001b[43m(\u001b[49m\u001b[43mrequest\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    705\u001b[0m \u001b[38;5;66;03m# Total elapsed time of the request (approximately)\u001b[39;00m\n\u001b[1;32m    706\u001b[0m elapsed \u001b[38;5;241m=\u001b[39m preferred_clock() \u001b[38;5;241m-\u001b[39m start\n",
      "File \u001b[0;32m~/anaconda3/lib/python3.8/site-packages/requests/adapters.py:532\u001b[0m, in \u001b[0;36mHTTPAdapter.send\u001b[0;34m(self, request, stream, timeout, verify, cert, proxies)\u001b[0m\n\u001b[1;32m    530\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m SSLError(e, request\u001b[38;5;241m=\u001b[39mrequest)\n\u001b[1;32m    531\u001b[0m \u001b[38;5;28;01melif\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(e, ReadTimeoutError):\n\u001b[0;32m--> 532\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m ReadTimeout(e, request\u001b[38;5;241m=\u001b[39mrequest)\n\u001b[1;32m    533\u001b[0m \u001b[38;5;28;01melif\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(e, _InvalidHeader):\n\u001b[1;32m    534\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m InvalidHeader(e, request\u001b[38;5;241m=\u001b[39mrequest)\n",
      "\u001b[0;31mReadTimeout\u001b[0m: HTTPSConnectionPool(host='toar-data.fz-juelich.de', port=443): Read timed out. (read timeout=10)"
     ]
    }
   ],
   "source": [
    "# provides all stations from Andorra(AD) and Gibraltar(GI)\n",
    "resp = requests.get(\"https://toar-data.fz-juelich.de/api/v2/stationmeta/?country=DE&limit=None\", timeout=(3.05, 10))\n",
    "stations = resp.json()\n",
    "\n",
    "for ind in range(0,1):\n",
    "    print(stations[ind]['id'])\n",
    "    print(stations[ind]['codes'])\n",
    "    print(stations[ind]['name'])\n",
    "# Access nested variables\n",
    "for item in data:\n",
    "    name = item[1]\n",
    "    #age = item[\"age\"]\n",
    "    #city = item[\"address\"][\"city\"]\n",
    "    #state = item[\"address\"][\"state\"]\n",
    "    #print(f\"Name: {id}\")\n",
    "    #print(f\"Name: {name}, Age: {age}, City: {city}, State: {state}\")\n",
    "print(stations)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "BJipB0RzBhx9",
   "metadata": {
    "id": "BJipB0RzBhx9"
   },
   "source": [
    "### 2. Find all time series of one station\n",
    "\n",
    "From the last command, we found the station Escaldes-Engordany (Andorra). Its station code is AD0942A.  \n",
    "Now we want to extract the metadata of all time series for this specific station that are stored in the TOAR database."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "I4umIwMBBhx-",
   "metadata": {
    "id": "I4umIwMBBhx-",
    "tags": []
   },
   "outputs": [],
   "source": [
    "resp = requests.get(\"https://toar-data.fz-juelich.de/api/v2/search/?codes=DEUB001&variable_id=21&limit=None\", timeout=(3.05, 5))\n",
    "metadata = resp.json()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "ddqXFEp-Bhx_",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ddqXFEp-Bhx_",
    "outputId": "b745c3a0-5381-4a5c-9f77-8f46afe7bf0a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "18400\n",
      "94503\n"
     ]
    }
   ],
   "source": [
    "for i in range(0,2):\n",
    "    print(metadata[i]['id'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ZJ6jYuucBhx_",
   "metadata": {
    "id": "ZJ6jYuucBhx_"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "7OJIsgOEBhx_",
   "metadata": {
    "id": "7OJIsgOEBhx_"
   },
   "source": [
    "### 3. Download data of a specific time series\n",
    "\n",
    "From the previous results, we choose the time series of ozone from the resource_provider (roles) European Environment Agency (EEA) with id 24896.  \n",
    "We want to download all available data in csv format, which is possible by providing the argument _format=csv_.  \n",
    "This means that data is no longer provided in JSON format (and we cannot use json() to access it).  \n",
    "We want to read the data into a pandas DataFrame.  \n",
    "(_Data from the TOAR-II database will always be delivered with its related metadata!_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "id": "pXjwuCcfBhyA",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "pXjwuCcfBhyA",
    "outputId": "e2332f47-66e8-47e8-e819-ad3471e298a0"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<Response [200]>\n"
     ]
    }
   ],
   "source": [
    "\n",
    "URL = \"https://toar-data.fz-juelich.de/api/v2/data/timeseries/18400?format=csv&daterange=2010-05-01T00:00:00,2010-05-05T08:00:00\"\n",
    "#URL = \"https://toar-data.fz-juelich.de/api/v2/search/?bounding_box=49,7,50,8&variable_id=21&limit=None\"\n",
    "\n",
    "resp = requests.get(URL)\n",
    "pp.pprint(resp)\n",
    "# save to file\n",
    "with open(\"StationDEUB001_timeseries_id18400_01MAY0000_2010_04MAY2359_2010_air_temperature.csv\", \"w\") as file:\n",
    "    file.write(resp.text)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "U7LEMXVvBhyA",
   "metadata": {
    "id": "U7LEMXVvBhyA"
   },
   "source": [
    "(For details on how we load the metadata here see [loading_metadata_from_data_download.ipynb](loading_metadata_from_data_download.ipynb))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "id": "api2oRmSBhyB",
   "metadata": {
    "id": "api2oRmSBhyB"
   },
   "outputs": [],
   "source": [
    "# read metadata from the top of the response\n",
    "#metadata = json.loads(\"\\n\".join([line[1:] for line in resp.text.split(\"\\n\") if line.startswith(\"#\")]))\n",
    "#(\"metadata:\")\n",
    "#pp.pprint(metadata)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 189,
   "id": "lsZgZoffBhyB",
   "metadata": {
    "id": "lsZgZoffBhyB"
   },
   "outputs": [],
   "source": [
    "# read the actual data\n",
    "#csv_file = \"StationDEUB001_timeseries_id94503_JAN_2010_FEB_2010_air_temperature.csv\"\n",
    "data = pd.read_csv(StringIO(resp.text), comment=\"#\", index_col=\"datetime\", parse_dates=True, infer_datetime_format=True)\n",
    "#print(f\"data:\\n{data}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "I93tCC0NBhyC",
   "metadata": {
    "id": "I93tCC0NBhyC"
   },
   "source": [
    "### 4. Plot the time series"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 190,
   "id": "Wta1SEKnBhyC",
   "metadata": {
    "id": "Wta1SEKnBhyC"
   },
   "outputs": [],
   "source": [
    "\n",
    "import numpy as np\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.linear_model import LinearRegression\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 191,
   "id": "PANg5gsyBhyC",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 868
    },
    "id": "PANg5gsyBhyC",
    "outputId": "82411902-1399-4195-d71e-751a0bef7306"
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 56420 characters]\n",
      "text/plain": [
       "<Figure size 2000x1000 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data.reset_index().plot(x=\"datetime\", y=\"value\", kind=\"scatter\", title=\"Plot of time series StationDEUB001_timeseries_id18400_01MAY0000_2010_05MAY0700_2010_air_temperature\", figsize=(20, 10));"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "GfhIqQAcBhyC",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 237
    },
    "id": "GfhIqQAcBhyC",
    "outputId": "6f51c572-23f7-4531-9315-e96a835fe830"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-cfb704c5-40ef-40e1-a4e7-0129c7d93436\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>value</th>\n",
       "      <th>flags</th>\n",
       "      <th>version</th>\n",
       "      <th>timeseries_id</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>datetime</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2010-05-01 00:00:00+00:00</th>\n",
       "      <td>8.20</td>\n",
       "      <td>7</td>\n",
       "      <td>000001.000000.00000000000000</td>\n",
       "      <td>18400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-05-01 01:00:00+00:00</th>\n",
       "      <td>8.25</td>\n",
       "      <td>7</td>\n",
       "      <td>000001.000000.00000000000000</td>\n",
       "      <td>18400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-05-01 02:00:00+00:00</th>\n",
       "      <td>8.60</td>\n",
       "      <td>7</td>\n",
       "      <td>000001.000000.00000000000000</td>\n",
       "      <td>18400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-05-01 03:00:00+00:00</th>\n",
       "      <td>8.80</td>\n",
       "      <td>7</td>\n",
       "      <td>000001.000000.00000000000000</td>\n",
       "      <td>18400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-05-01 04:00:00+00:00</th>\n",
       "      <td>8.90</td>\n",
       "      <td>7</td>\n",
       "      <td>000001.000000.00000000000000</td>\n",
       "      <td>18400</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-cfb704c5-40ef-40e1-a4e7-0129c7d93436')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-cfb704c5-40ef-40e1-a4e7-0129c7d93436 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-cfb704c5-40ef-40e1-a4e7-0129c7d93436');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "                           value  flags                       version  \\\n",
       "datetime                                                                \n",
       "2010-05-01 00:00:00+00:00   8.20      7  000001.000000.00000000000000   \n",
       "2010-05-01 01:00:00+00:00   8.25      7  000001.000000.00000000000000   \n",
       "2010-05-01 02:00:00+00:00   8.60      7  000001.000000.00000000000000   \n",
       "2010-05-01 03:00:00+00:00   8.80      7  000001.000000.00000000000000   \n",
       "2010-05-01 04:00:00+00:00   8.90      7  000001.000000.00000000000000   \n",
       "\n",
       "                           timeseries_id  \n",
       "datetime                                  \n",
       "2010-05-01 00:00:00+00:00          18400  \n",
       "2010-05-01 01:00:00+00:00          18400  \n",
       "2010-05-01 02:00:00+00:00          18400  \n",
       "2010-05-01 03:00:00+00:00          18400  \n",
       "2010-05-01 04:00:00+00:00          18400  "
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "Zge53dcfBhyD",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Zge53dcfBhyD",
    "outputId": "2ee7e125-3b72-4fd5-91ed-9f7c11de2640"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n"
     ]
    }
   ],
   "source": [
    "print(type(data))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "id": "Ebn7ION3BhyD",
   "metadata": {
    "id": "Ebn7ION3BhyD"
   },
   "outputs": [],
   "source": [
    "data.reset_index(inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "onZlQ_YSBhyD",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "onZlQ_YSBhyD",
    "outputId": "9e64eb39-54b9-4a7a-f25c-c06b30fdc73b"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-5bb0e4e3-be14-41b4-8043-d609f44fbbea\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>datetime</th>\n",
       "      <th>value</th>\n",
       "      <th>flags</th>\n",
       "      <th>version</th>\n",
       "      <th>timeseries_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2010-05-01 00:00:00+00:00</td>\n",
       "      <td>8.20</td>\n",
       "      <td>7</td>\n",
       "      <td>000001.000000.00000000000000</td>\n",
       "      <td>18400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2010-05-01 01:00:00+00:00</td>\n",
       "      <td>8.25</td>\n",
       "      <td>7</td>\n",
       "      <td>000001.000000.00000000000000</td>\n",
       "      <td>18400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2010-05-01 02:00:00+00:00</td>\n",
       "      <td>8.60</td>\n",
       "      <td>7</td>\n",
       "      <td>000001.000000.00000000000000</td>\n",
       "      <td>18400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2010-05-01 03:00:00+00:00</td>\n",
       "      <td>8.80</td>\n",
       "      <td>7</td>\n",
       "      <td>000001.000000.00000000000000</td>\n",
       "      <td>18400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2010-05-01 04:00:00+00:00</td>\n",
       "      <td>8.90</td>\n",
       "      <td>7</td>\n",
       "      <td>000001.000000.00000000000000</td>\n",
       "      <td>18400</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-5bb0e4e3-be14-41b4-8043-d609f44fbbea')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-5bb0e4e3-be14-41b4-8043-d609f44fbbea button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-5bb0e4e3-be14-41b4-8043-d609f44fbbea');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "                   datetime  value  flags                       version  \\\n",
       "0 2010-05-01 00:00:00+00:00   8.20      7  000001.000000.00000000000000   \n",
       "1 2010-05-01 01:00:00+00:00   8.25      7  000001.000000.00000000000000   \n",
       "2 2010-05-01 02:00:00+00:00   8.60      7  000001.000000.00000000000000   \n",
       "3 2010-05-01 03:00:00+00:00   8.80      7  000001.000000.00000000000000   \n",
       "4 2010-05-01 04:00:00+00:00   8.90      7  000001.000000.00000000000000   \n",
       "\n",
       "   timeseries_id  \n",
       "0          18400  \n",
       "1          18400  \n",
       "2          18400  \n",
       "3          18400  \n",
       "4          18400  "
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 193,
   "id": "iteUqCMDBhyE",
   "metadata": {
    "id": "iteUqCMDBhyE"
   },
   "outputs": [],
   "source": [
    "data['datetime'] = pd.to_datetime(data['datetime'])\n",
    "data = data.sort_values('datetime')\n",
    "data = data[['datetime', 'value']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "B-qhpsmGBhyE",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "B-qhpsmGBhyE",
    "outputId": "52898f24-55ac-4eff-b040-26949a533861"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-11b93b6e-7b04-463f-aff1-cf26e69e51cc\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>datetime</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2010-05-01 00:00:00+00:00</td>\n",
       "      <td>8.20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2010-05-01 01:00:00+00:00</td>\n",
       "      <td>8.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2010-05-01 02:00:00+00:00</td>\n",
       "      <td>8.60</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2010-05-01 03:00:00+00:00</td>\n",
       "      <td>8.80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2010-05-01 04:00:00+00:00</td>\n",
       "      <td>8.90</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-11b93b6e-7b04-463f-aff1-cf26e69e51cc')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-11b93b6e-7b04-463f-aff1-cf26e69e51cc button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-11b93b6e-7b04-463f-aff1-cf26e69e51cc');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "                   datetime  value\n",
       "0 2010-05-01 00:00:00+00:00   8.20\n",
       "1 2010-05-01 01:00:00+00:00   8.25\n",
       "2 2010-05-01 02:00:00+00:00   8.60\n",
       "3 2010-05-01 03:00:00+00:00   8.80\n",
       "4 2010-05-01 04:00:00+00:00   8.90"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "id": "ojSTcymfBhyF",
   "metadata": {
    "id": "ojSTcymfBhyF"
   },
   "outputs": [],
   "source": [
    "train_data, test_data = train_test_split(data, test_size=0.2, shuffle=False)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "id": "MyOneNFGBhyF",
   "metadata": {
    "id": "MyOneNFGBhyF"
   },
   "outputs": [],
   "source": [
    "X_train = train_data['datetime'].astype(np.int64) // 10**9  # Convert datetime to epoch timestamp\n",
    "y_train = train_data['value']\n",
    "X_test = test_data['datetime'].astype(np.int64) // 10**9\n",
    "y_test = test_data['value']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "Yn8xAvC0BhyF",
   "metadata": {
    "id": "Yn8xAvC0BhyF"
   },
   "source": [
    "\n",
    "# Linear Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "id": "s6MnQ7GsBhyF",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 75
    },
    "id": "s6MnQ7GsBhyF",
    "outputId": "28d02015-6c6e-4e8f-92f1-4c03d05c6938"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-2 {color: black;background-color: white;}#sk-container-id-2 pre{padding: 0;}#sk-container-id-2 div.sk-toggleable {background-color: white;}#sk-container-id-2 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-2 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-2 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-2 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-2 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-2 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-2 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-2 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-2 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-2 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-2 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-2 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-2 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-2 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-2 div.sk-item {position: relative;z-index: 1;}#sk-container-id-2 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-2 div.sk-item::before, #sk-container-id-2 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-2 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-2 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-2 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-2 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-2 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-2 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-2 div.sk-label-container {text-align: center;}#sk-container-id-2 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-2 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-2\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>LinearRegression()</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-2\" type=\"checkbox\" checked><label for=\"sk-estimator-id-2\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">LinearRegression</label><div class=\"sk-toggleable__content\"><pre>LinearRegression()</pre></div></div></div></div></div>"
      ],
      "text/plain": [
       "LinearRegression()"
      ]
     },
     "execution_count": 134,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = LinearRegression()\n",
    "model.fit(X_train.values.reshape(-1, 1), y_train)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "id": "TKmBhTQ7BhyG",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "TKmBhTQ7BhyG",
    "outputId": "c2676c37-f31b-45e9-e388-8e84305f7bfe"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<bound method LinearRegression.fit of LinearRegression()>\n"
     ]
    }
   ],
   "source": [
    "print(model.fit)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "id": "XqcNNzM5BhyG",
   "metadata": {
    "id": "XqcNNzM5BhyG"
   },
   "outputs": [],
   "source": [
    "#next_timestamps = X_test[-1:].values.reshape(-1, 1) + np.array([3600, 7200, 10800])  # Add 1, 2, 3 hours to the last timestamp in the test set\n",
    "\n",
    "predicted_values = model.predict(X_test.values.reshape(-1,1))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "id": "OIaK_8CrBhyG",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "OIaK_8CrBhyG",
    "outputId": "8e9a20d9-d8a0-4e57-832c-71900e2bb34c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[6.71721835 6.71210394 6.70698954 6.70187513 6.69676073 6.69164632\n",
      " 6.68653192 6.68141751 6.6763031  6.6711887  6.66607429 6.66095989\n",
      " 6.65584548 6.65073107 6.64561667 6.64050226 6.63538786 6.63027345\n",
      " 6.62515904 6.62004464 6.61493023 6.60981583 6.60470142 6.59958702\n",
      " 6.59447261 6.5893582  6.5842438  6.57912939 6.57401499 6.56890058\n",
      " 6.56378617 6.55867177 6.55355736 6.54844296 6.54332855 6.53821415\n",
      " 6.53309974 6.52798533 6.52287093 6.51775652 6.51264212 6.50752771\n",
      " 6.5024133  6.4972989  6.49218449 6.48707009 6.48195568 6.47684127\n",
      " 6.47172687 6.46661246 6.46149806 6.45638365 6.45126925 6.44615484\n",
      " 6.44104043 6.43592603 6.43081162 6.42569722 6.42058281 6.4154684\n",
      " 6.410354   6.40523959 6.40012519 6.39501078 6.38989637 6.38478197\n",
      " 6.37966756 6.37455316 6.36943875 6.36432435 6.35920994 6.35409553]\n"
     ]
    }
   ],
   "source": [
    "print(predicted_values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "id": "0RFodschBhyH",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "0RFodschBhyH",
    "outputId": "de78565a-2cfd-4bb2-d182-8ba395fa294d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MAE: 1.3029813130070689\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import mean_absolute_error\n",
    "y_test = test_data['value']\n",
    "actual_values = y_test.values.reshape(-1,1)\n",
    "# Assuming you have the actual values in a variable named 'actual_values'\n",
    "mae = mean_absolute_error(actual_values, predicted_values)\n",
    "print(\"MAE:\", mae)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "id": "vD0Dq1AtBhyH",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "vD0Dq1AtBhyH",
    "outputId": "801781ba-edf3-47c3-9d86-9c58493914b7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MSE: 2.116641192871284\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import mean_squared_error\n",
    "\n",
    "mse = mean_squared_error(actual_values, predicted_values)\n",
    "print(\"MSE:\", mse)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "id": "SKRZgcqKBhyH",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "SKRZgcqKBhyH",
    "outputId": "8e8d5097-adaa-42ea-97e7-7a875a708835"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE: 1.4548681015374842\n"
     ]
    }
   ],
   "source": [
    "rmse = np.sqrt(mse)\n",
    "print(\"RMSE:\", rmse)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "id": "4Rx4mg1PBhyH",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "4Rx4mg1PBhyH",
    "outputId": "3192a184-7f77-4a7d-b181-7e193d2717d7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "R2 Score: -3.765968528024489\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import r2_score\n",
    "\n",
    "r2 = r2_score(actual_values, predicted_values)\n",
    "print(\"R2 Score:\", r2)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 194,
   "id": "BGBSQuxoBhyW",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 504
    },
    "id": "BGBSQuxoBhyW",
    "outputId": "7b415366-333a-4941-86cb-aaad9ce5e421"
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 43264 characters]\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data.plot(x='datetime',y='value')\n",
    "\n",
    "plt.xlabel('Datetime')\n",
    "plt.ylabel('Value')\n",
    "plt.title('Time Series Data')\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fqxxbkA_BhyW",
   "metadata": {
    "id": "fqxxbkA_BhyW"
   },
   "source": [
    "# ARIMA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "id": "OTgA4uoRBhyW",
   "metadata": {
    "id": "OTgA4uoRBhyW"
   },
   "outputs": [],
   "source": [
    "from statsmodels.tsa.arima.model import ARIMA\n",
    "model = ARIMA(data['value'], order=(1, 3, 1))  # Replace p, d, q with appropriate values\n",
    "model_fit = model.fit()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "id": "_obhIKYJBhyX",
   "metadata": {
    "id": "_obhIKYJBhyX"
   },
   "outputs": [],
   "source": [
    "forecast = model_fit.forecast(steps=3)  # Replace 3 with the number of steps you want to forecast\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "id": "-v8Yy6qeBhyX",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "-v8Yy6qeBhyX",
    "outputId": "205588dd-ff63-4a9a-8539-f4cc94402f0e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "y_test series: 84     7.20\n",
      "85     7.30\n",
      "86     7.30\n",
      "87     7.25\n",
      "88     7.15\n",
      "89     7.05\n",
      "90     6.85\n",
      "91     6.55\n",
      "92     6.25\n",
      "93     6.05\n",
      "94     5.95\n",
      "95     5.90\n",
      "96     5.85\n",
      "97     5.80\n",
      "98     5.80\n",
      "99     5.80\n",
      "100    5.90\n",
      "101    6.25\n",
      "102    6.85\n",
      "103    7.50\n",
      "104    7.90\n",
      "Name: value, dtype: float64\n",
      "forecast series: 105    8.393880\n",
      "106    8.858647\n",
      "107    9.338290\n",
      "Name: predicted_mean, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "print('y_test series:', y_test)\n",
    "print('forecast series:', forecast)\n",
    "# Concatenate the forecasted values DataFrame with the test series DataFrame\n",
    "combined_data = pd.concat([y_test, forecast])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "id": "MvGcxpjvBhyY",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 472
    },
    "id": "MvGcxpjvBhyY",
    "outputId": "66dc6fbe-b443-43af-ae3d-c681b0677acc"
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 42000 characters]\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot the combined_data\n",
    "plt.plot(y_train.index, y_train,color='gray',label='Train')\n",
    "plt.plot(y_test.index, y_test,label='Test')\n",
    "plt.plot(forecast.index, forecast,label='Forecast')\n",
    "plt.xlabel('Index')\n",
    "plt.ylabel('Value')\n",
    "plt.title('ARIMA model  01MAY - 05MAY, forecast: 09:00-11:00 ')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ijuUGRC6BhyY",
   "metadata": {
    "id": "ijuUGRC6BhyY"
   },
   "source": [
    "# RNN method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 248,
   "id": "RAICmpPyBhyZ",
   "metadata": {
    "id": "RAICmpPyBhyZ"
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import MinMaxScaler\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import LSTM, Dense, Dropout\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 249,
   "id": "4jpTxYb3KL_W",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "4jpTxYb3KL_W",
    "outputId": "c758f544-7a83-4fc0-bfd0-0493fb722bae"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-1d433891-3e71-462f-bb5e-248eee9e9cfb\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>datetime</th>\n",
       "      <th>value</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2010-05-01 00:00:00+00:00</td>\n",
       "      <td>8.20</td>\n",
       "      <td>00:00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2010-05-01 01:00:00+00:00</td>\n",
       "      <td>8.25</td>\n",
       "      <td>01:00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2010-05-01 02:00:00+00:00</td>\n",
       "      <td>8.60</td>\n",
       "      <td>02:00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2010-05-01 03:00:00+00:00</td>\n",
       "      <td>8.80</td>\n",
       "      <td>03:00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2010-05-01 04:00:00+00:00</td>\n",
       "      <td>8.90</td>\n",
       "      <td>04:00:00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-1d433891-3e71-462f-bb5e-248eee9e9cfb')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-1d433891-3e71-462f-bb5e-248eee9e9cfb button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-1d433891-3e71-462f-bb5e-248eee9e9cfb');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "                   datetime  value      time\n",
       "0 2010-05-01 00:00:00+00:00   8.20  00:00:00\n",
       "1 2010-05-01 01:00:00+00:00   8.25  01:00:00\n",
       "2 2010-05-01 02:00:00+00:00   8.60  02:00:00\n",
       "3 2010-05-01 03:00:00+00:00   8.80  03:00:00\n",
       "4 2010-05-01 04:00:00+00:00   8.90  04:00:00"
      ]
     },
     "execution_count": 249,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 250,
   "id": "rK3KbSreBhyZ",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "rK3KbSreBhyZ",
    "outputId": "59c236c8-c17e-4891-b293-3f6b3b779c7d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "values[:4] [8.2  8.25 8.6  8.8 ]\n",
      "scaler MinMaxScaler()\n",
      "normalized_values[:4] [[0.47524752]\n",
      " [0.48514851]\n",
      " [0.55445545]\n",
      " [0.59405941]]\n"
     ]
    }
   ],
   "source": [
    "# Assuming you have the time series data stored in a DataFrame called 'data'\n",
    "\n",
    "# Extract the values from the 'value' column as a NumPy array\n",
    "values = data['value'].values\n",
    "print('values[:4]',values[:4])\n",
    "\n",
    "# Normalize the values to a range between 0 and 1\n",
    "scaler = MinMaxScaler()\n",
    "print('scaler',scaler)\n",
    "normalized_values = scaler.fit_transform(values.reshape(-1, 1))\n",
    "print('normalized_values[:4]',normalized_values[:4])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 251,
   "id": "uDqCaG2DDdSw",
   "metadata": {
    "id": "uDqCaG2DDdSw"
   },
   "outputs": [],
   "source": [
    "# Define the size of the training set\n",
    "train_size = int(len(normalized_values) * 0.8)  # 80% for training\n",
    "\n",
    "# Split the data into training and testing sets\n",
    "train_data = normalized_values[:train_size]\n",
    "test_data = normalized_values[train_size:]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 262,
   "id": "8HNiLercDi8g",
   "metadata": {
    "id": "8HNiLercDi8g"
   },
   "outputs": [],
   "source": [
    "# Define the number of previous time steps to consider for each prediction\n",
    "time_steps = 5\n",
    "\n",
    "# Create input and output sequences for training and testing\n",
    "def create_sequences(data, time_steps):\n",
    "    X = []\n",
    "    y = []\n",
    "    for i in range(len(data) - time_steps):\n",
    "     \n",
    "        X.append(data[i:i + time_steps])\n",
    "        y.append(data[i + time_steps])\n",
    "      \n",
    "    return np.array(X), np.array(y)\n",
    "\n",
    "# Create sequences for training and testing\n",
    "X_train, y_train = create_sequences(train_data, time_steps)\n",
    "X_test, y_test = create_sequences(test_data, time_steps)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 263,
   "id": "pEI9G4L3nJcn",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "pEI9G4L3nJcn",
    "outputId": "a8706403-dc42-4e6b-ada7-67295ef3f938"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "data[1:21                      datetime  value      time\n",
      "0  2010-05-01 00:00:00+00:00   8.20  00:00:00\n",
      "1  2010-05-01 01:00:00+00:00   8.25  01:00:00\n",
      "2  2010-05-01 02:00:00+00:00   8.60  02:00:00\n",
      "3  2010-05-01 03:00:00+00:00   8.80  03:00:00\n",
      "4  2010-05-01 04:00:00+00:00   8.90  04:00:00\n",
      "5  2010-05-01 05:00:00+00:00   8.75  05:00:00\n",
      "6  2010-05-01 06:00:00+00:00   8.85  06:00:00\n",
      "7  2010-05-01 07:00:00+00:00   9.10  07:00:00\n",
      "8  2010-05-01 08:00:00+00:00   9.50  08:00:00\n",
      "9  2010-05-01 09:00:00+00:00   9.15  09:00:00\n",
      "10 2010-05-01 10:00:00+00:00   8.90  10:00:00\n",
      "11 2010-05-01 11:00:00+00:00   8.60  11:00:00\n",
      "12 2010-05-01 12:00:00+00:00   8.65  12:00:00\n",
      "13 2010-05-01 13:00:00+00:00   8.40  13:00:00\n",
      "14 2010-05-01 14:00:00+00:00   7.45  14:00:00\n",
      "15 2010-05-01 15:00:00+00:00   8.25  15:00:00\n",
      "16 2010-05-01 16:00:00+00:00   8.40  16:00:00\n",
      "17 2010-05-01 17:00:00+00:00   8.15  17:00:00\n",
      "18 2010-05-01 18:00:00+00:00   7.70  18:00:00\n",
      "19 2010-05-01 19:00:00+00:00   7.50  19:00:00\n",
      "20 2010-05-01 20:00:00+00:00   7.30  20:00:00\n",
      "data[1]                     datetime  value      time\n",
      "0  2010-05-01 00:00:00+00:00   8.20  00:00:00\n",
      "1  2010-05-01 01:00:00+00:00   8.25  01:00:00\n",
      "2  2010-05-01 02:00:00+00:00   8.60  02:00:00\n",
      "3  2010-05-01 03:00:00+00:00   8.80  03:00:00\n",
      "4  2010-05-01 04:00:00+00:00   8.90  04:00:00\n",
      "5  2010-05-01 05:00:00+00:00   8.75  05:00:00\n",
      "6  2010-05-01 06:00:00+00:00   8.85  06:00:00\n",
      "7  2010-05-01 07:00:00+00:00   9.10  07:00:00\n",
      "8  2010-05-01 08:00:00+00:00   9.50  08:00:00\n",
      "9  2010-05-01 09:00:00+00:00   9.15  09:00:00\n",
      "10 2010-05-01 10:00:00+00:00   8.90  10:00:00\n",
      "11 2010-05-01 11:00:00+00:00   8.60  11:00:00\n",
      "12 2010-05-01 12:00:00+00:00   8.65  12:00:00\n",
      "13 2010-05-01 13:00:00+00:00   8.40  13:00:00\n",
      "14 2010-05-01 14:00:00+00:00   7.45  14:00:00\n",
      "15 2010-05-01 15:00:00+00:00   8.25  15:00:00\n",
      "16 2010-05-01 16:00:00+00:00   8.40  16:00:00\n",
      "17 2010-05-01 17:00:00+00:00   8.15  17:00:00\n",
      "18 2010-05-01 18:00:00+00:00   7.70  18:00:00\n",
      "19 2010-05-01 19:00:00+00:00   7.50  19:00:00\n",
      "20 2010-05-01 20:00:00+00:00   7.30  20:00:00\n",
      "X_test[:10] [[[0.27722772]\n",
      "  [0.2970297 ]\n",
      "  [0.2970297 ]\n",
      "  [0.28712871]\n",
      "  [0.26732673]]\n",
      "\n",
      " [[0.2970297 ]\n",
      "  [0.2970297 ]\n",
      "  [0.28712871]\n",
      "  [0.26732673]\n",
      "  [0.24752475]]]\n",
      "y_test[:10]  [[0.24752475]\n",
      " [0.20792079]]\n",
      "len(X_test),len(y_test) 16 16\n",
      " <class 'numpy.ndarray'>\n"
     ]
    }
   ],
   "source": [
    "print('data[1:21 ',data[0:21])\n",
    "print('data[1]',data[0:21])\n",
    "print('X_test[:10]',X_test[:2])\n",
    "print('y_test[:10] ',y_test[:2])\n",
    "print('len(X_test),len(y_test)',len(X_test),len(y_test))\n",
    "print('',type(X_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 264,
   "id": "UCjeINHeDrvi",
   "metadata": {
    "id": "UCjeINHeDrvi"
   },
   "outputs": [],
   "source": [
    "# Define the RNN model architecture\n",
    "model = Sequential()\n",
    "model.add(LSTM(64, input_shape=(time_steps, 1)))\n",
    "model.add(Dropout(0.2))  # Add dropout layer with a dropout rate of 0.2\n",
    "model.add(Dense(1))\n",
    "model.compile(optimizer='adam', loss='mean_squared_error')\n",
    "\n",
    "# Train the model\n",
    "#model.fit(X_train, y_train, epochs=50, batch_size=16)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 265,
   "id": "_PuQM4aHFFHg",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "_PuQM4aHFFHg",
    "outputId": "e9a713a3-ecbf-4fab-9029-1d20b1adb2be"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "5/5 [==============================] - 3s 9ms/step - loss: 0.1769\n",
      "Epoch 2/20\n",
      "5/5 [==============================] - 0s 8ms/step - loss: 0.0980\n",
      "Epoch 3/20\n",
      "5/5 [==============================] - 0s 8ms/step - loss: 0.0467\n",
      "Epoch 4/20\n",
      "5/5 [==============================] - 0s 8ms/step - loss: 0.0215\n",
      "Epoch 5/20\n",
      "5/5 [==============================] - 0s 8ms/step - loss: 0.0299\n",
      "Epoch 6/20\n",
      "5/5 [==============================] - 0s 9ms/step - loss: 0.0314\n",
      "Epoch 7/20\n",
      "5/5 [==============================] - 0s 8ms/step - loss: 0.0233\n",
      "Epoch 8/20\n",
      "5/5 [==============================] - 0s 8ms/step - loss: 0.0231\n",
      "Epoch 9/20\n",
      "5/5 [==============================] - 0s 8ms/step - loss: 0.0266\n",
      "Epoch 10/20\n",
      "5/5 [==============================] - 0s 8ms/step - loss: 0.0222\n",
      "Epoch 11/20\n",
      "5/5 [==============================] - 0s 7ms/step - loss: 0.0234\n",
      "Epoch 12/20\n",
      "5/5 [==============================] - 0s 7ms/step - loss: 0.0239\n",
      "Epoch 13/20\n",
      "5/5 [==============================] - 0s 7ms/step - loss: 0.0259\n",
      "Epoch 14/20\n",
      "5/5 [==============================] - 0s 7ms/step - loss: 0.0226\n",
      "Epoch 15/20\n",
      "5/5 [==============================] - 0s 7ms/step - loss: 0.0189\n",
      "Epoch 16/20\n",
      "5/5 [==============================] - 0s 6ms/step - loss: 0.0221\n",
      "Epoch 17/20\n",
      "5/5 [==============================] - 0s 6ms/step - loss: 0.0211\n",
      "Epoch 18/20\n",
      "5/5 [==============================] - 0s 7ms/step - loss: 0.0229\n",
      "Epoch 19/20\n",
      "5/5 [==============================] - 0s 7ms/step - loss: 0.0218\n",
      "Epoch 20/20\n",
      "5/5 [==============================] - 0s 9ms/step - loss: 0.0196\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 31772 characters]\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Train the model and store the training history\n",
    "history = model.fit(X_train, y_train, epochs=20, batch_size=16)\n",
    "\n",
    "# Access the loss values from the history object\n",
    "loss = history.history['loss']\n",
    "\n",
    "# Plot the loss curve\n",
    "plt.plot(range(len(loss)), loss)\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Loss')\n",
    "plt.title('Training Loss vs. Epochs')\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 266,
   "id": "efrD00dVDwgs",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "efrD00dVDwgs",
    "outputId": "7e408aca-6101-4f5a-cdbe-f0f4d33dec47"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/1 [==============================] - 0s 497ms/step\n"
     ]
    }
   ],
   "source": [
    "# Generate predictions on the test data\n",
    "y_pred = model.predict(X_test)\n",
    "#print('X_test',X_test)\n",
    "#print('y_pred',y_pred)\n",
    "# Inverse transform the normalized predictions to the original scale\n",
    "y_pred = scaler.inverse_transform(y_pred)\n",
    "\n",
    "# Inverse transform the normalized test data to the original scale\n",
    "y_test = scaler.inverse_transform(y_test)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 267,
   "id": "RYfMCke5D4Vf",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "RYfMCke5D4Vf",
    "outputId": "5e9809a4-07e4-461e-f6be-a9a7306cb88e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE: 0.7591631268873975\n",
      "MAE: 0.7197793126106264\n"
     ]
    }
   ],
   "source": [
    "# Calculate evaluation metrics such as RMSE or MAE\n",
    "from sklearn.metrics import mean_squared_error, mean_absolute_error\n",
    "\n",
    "rmse = np.sqrt(mean_squared_error(y_test, y_pred))\n",
    "mae = mean_absolute_error(y_test, y_pred)\n",
    "\n",
    "print(\"RMSE:\", rmse)\n",
    "print(\"MAE:\", mae)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 268,
   "id": "pHk-lkfhD8EB",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 454
    },
    "id": "pHk-lkfhD8EB",
    "outputId": "18671c77-18a2-41f7-8e9a-e1802c22d2fa"
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 37032 characters]\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot the actual values and the predictions\n",
    "plt.plot(y_test, label='TOAR observed air_temperature data',linestyle='dotted')\n",
    "plt.plot(y_pred, label='Predicted')\n",
    "plt.xlabel('Time')\n",
    "plt.ylabel('Value')\n",
    "plt.legend()\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "id": "bmWbPtP6D-1k",
   "metadata": {
    "id": "bmWbPtP6D-1k"
   },
   "outputs": [],
   "source": [
    "# Assuming you have the time series data stored in a DataFrame called 'data'\n",
    "\n",
    "# Extract the values from the 'value' column as a NumPy array\n",
    "values = data['value'].values\n",
    "\n",
    "# Normalize the values to a range between 0 and 1\n",
    "scaler = MinMaxScaler()\n",
    "normalized_values = scaler.fit_transform(values.reshape(-1, 1))\n",
    "\n",
    "# Create input sequence for prediction\n",
    "def create_input_sequence(data, time_steps):\n",
    "    sequence = data[-time_steps:]  # Take the last 'time_steps' values as input\n",
    "    return np.array(sequence).reshape(1, -1, 1)\n",
    "\n",
    "# Create input sequence for prediction using the last 'time_steps' values\n",
    "input_sequence = create_input_sequence(normalized_values, time_steps)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "id": "Z4t3mN5yJ8qe",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Z4t3mN5yJ8qe",
    "outputId": "9cecaaa8-d92c-4672-9370-aa796c5eaca2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/1 [==============================] - 0s 35ms/step\n"
     ]
    }
   ],
   "source": [
    "# Generate the forecast for future points\n",
    "forecast = model.predict(input_sequence)\n",
    "\n",
    "# Inverse transform the normalized forecast to the original scale\n",
    "forecast = scaler.inverse_transform(forecast)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 163,
   "id": "xaVQizUKKcZ0",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "xaVQizUKKcZ0",
    "outputId": "7ff4ee0f-7185-496c-f1e0-0b1c6f926210"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-750c6273-e85c-4051-a45c-2e48b42d0c67\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>datetime</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2010-05-01 00:00:00+00:00</td>\n",
       "      <td>8.20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2010-05-01 01:00:00+00:00</td>\n",
       "      <td>8.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2010-05-01 02:00:00+00:00</td>\n",
       "      <td>8.60</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2010-05-01 03:00:00+00:00</td>\n",
       "      <td>8.80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2010-05-01 04:00:00+00:00</td>\n",
       "      <td>8.90</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-750c6273-e85c-4051-a45c-2e48b42d0c67')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-750c6273-e85c-4051-a45c-2e48b42d0c67 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-750c6273-e85c-4051-a45c-2e48b42d0c67');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "                   datetime  value\n",
       "0 2010-05-01 00:00:00+00:00   8.20\n",
       "1 2010-05-01 01:00:00+00:00   8.25\n",
       "2 2010-05-01 02:00:00+00:00   8.60\n",
       "3 2010-05-01 03:00:00+00:00   8.80\n",
       "4 2010-05-01 04:00:00+00:00   8.90"
      ]
     },
     "execution_count": 163,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.values.reshape(-1,1)\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 171,
   "id": "ndhXTGxgLjtc",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "ndhXTGxgLjtc",
    "outputId": "a66dc3d5-43aa-4561-d832-bb2644adbd40"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-18d600e5-c6f0-4b4f-825a-2c949c3cf5b7\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>datetime</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2010-05-01 00:00:00+00:00</td>\n",
       "      <td>8.20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2010-05-01 01:00:00+00:00</td>\n",
       "      <td>8.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2010-05-01 02:00:00+00:00</td>\n",
       "      <td>8.60</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2010-05-01 03:00:00+00:00</td>\n",
       "      <td>8.80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2010-05-01 04:00:00+00:00</td>\n",
       "      <td>8.90</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-18d600e5-c6f0-4b4f-825a-2c949c3cf5b7')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-18d600e5-c6f0-4b4f-825a-2c949c3cf5b7 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-18d600e5-c6f0-4b4f-825a-2c949c3cf5b7');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "                   datetime  value\n",
       "0 2010-05-01 00:00:00+00:00   8.20\n",
       "1 2010-05-01 01:00:00+00:00   8.25\n",
       "2 2010-05-01 02:00:00+00:00   8.60\n",
       "3 2010-05-01 03:00:00+00:00   8.80\n",
       "4 2010-05-01 04:00:00+00:00   8.90"
      ]
     },
     "execution_count": 171,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "yPF0FCJrO5_n",
   "metadata": {
    "id": "yPF0FCJrO5_n"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 270,
   "id": "sIJzQzVtP0S3",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "sIJzQzVtP0S3",
    "outputId": "bf7745cc-2f5c-42b8-c979-8f5bae5e9f96"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input_sequence [[[0.00990099]\n",
      "  [0.        ]\n",
      "  [0.        ]\n",
      "  [0.        ]\n",
      "  [0.01980198]\n",
      "  [0.08910891]\n",
      "  [0.20792079]\n",
      "  [0.33663366]\n",
      "  [0.41584158]]]\n",
      "1/1 [==============================] - 0s 20ms/step\n",
      "1/1 [==============================] - 0s 30ms/step\n",
      "1/1 [==============================] - 0s 32ms/step\n",
      "forecast_0 [[0.29442492]]\n",
      "forecast_1 [[0.29442492]]\n",
      "forecast_2 [[0.29442492]]\n",
      "forecast [[7.2868457]]\n",
      "last_date 2010-05-05 08:00:00+00:00\n",
      "future_dates DatetimeIndex(['2010-05-05 09:00:00+00:00', '2010-05-05 10:00:00+00:00'], dtype='datetime64[ns, UTC]', freq='H')\n",
      "all_dates.head() 0   2010-05-01 00:00:00+00:00\n",
      "1   2010-05-01 01:00:00+00:00\n",
      "2   2010-05-01 02:00:00+00:00\n",
      "3   2010-05-01 03:00:00+00:00\n",
      "4   2010-05-01 04:00:00+00:00\n",
      "dtype: datetime64[ns, UTC]\n",
      "all_dates.tail() 102   2010-05-05 06:00:00+00:00\n",
      "103   2010-05-05 07:00:00+00:00\n",
      "104   2010-05-05 08:00:00+00:00\n",
      "0     2010-05-05 09:00:00+00:00\n",
      "1     2010-05-05 10:00:00+00:00\n",
      "dtype: datetime64[ns, UTC]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-270-40674a23227e>:43: FutureWarning: Argument `closed` is deprecated in favor of `inclusive`.\n",
      "  future_dates = pd.date_range(last_date, freq = '1H', periods=future_steps, closed='right')\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "\n",
    "# Assuming you have the time series data stored in a DataFrame called 'data'\n",
    "\n",
    "# Extract the values from the 'value' column as a NumPy array\n",
    "values = data['value'].values\n",
    "\n",
    "# Normalize the values to a range between 0 and 1\n",
    "scaler = MinMaxScaler()\n",
    "normalized_values = scaler.fit_transform(values.reshape(-1, 1))\n",
    "\n",
    "# Create input sequence for prediction\n",
    "def create_input_sequence(data, time_steps):\n",
    "    sequence = data[-time_steps:]  # Take the last 'time_steps' values as input\n",
    "    return np.array(sequence).reshape(1, -1, 1)\n",
    "\n",
    "# Define the number of future points to forecast\n",
    "future_steps = 3\n",
    "time_steps = 3\n",
    "\n",
    "# Create input sequence for prediction using the last 'time_steps' values\n",
    "input_sequence = create_input_sequence(normalized_values, 9)\n",
    "print('input_sequence',input_sequence)\n",
    "# Generate the forecast for future points\n",
    "forecast_0 = model.predict(input_sequence)\n",
    "forecast_1 = model.predict(input_sequence)\n",
    "forecast_2 = model.predict(input_sequence)\n",
    "\n",
    "print('forecast_0',forecast_0)\n",
    "print('forecast_1',forecast_1)\n",
    "print('forecast_2',forecast_2)\n",
    "\n",
    "\n",
    "# Inverse transform the normalized forecast to the original scale\n",
    "forecast = scaler.inverse_transform(forecast_0)\n",
    "print('forecast',forecast)\n",
    "# Generate future dates for the forecasted points\n",
    "last_date = data['datetime'].iloc[-1]\n",
    "print('last_date',last_date)\n",
    "future_dates = pd.date_range(last_date, freq = '1H', periods=future_steps, closed='right')\n",
    "print('future_dates',future_dates)\n",
    "# Concatenate the original dates and the future dates\n",
    "all_dates = pd.concat([data['datetime'], pd.Series(future_dates)])\n",
    "print('all_dates.head()',all_dates.head())\n",
    "print('all_dates.tail()',all_dates.tail())\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 170,
   "id": "OEu0sbB-P8zp",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 484
    },
    "id": "OEu0sbB-P8zp",
    "outputId": "413b6716-6c8e-4778-89e8-29ec81f235e1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "values[-1] [7.9]\n",
      "forecast_extended=np.concatenate([values[-1:], forecast.flatten()]):  [7.9        6.91631413]\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 40964 characters]\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Extend the forecast array to match the length of all_dates\n",
    "forecast_extended = np.concatenate([values[-1:], forecast.flatten()])\n",
    "print('values[-1]',values[-1:])\n",
    "print('forecast_extended=np.concatenate([values[-1:], forecast.flatten()]): ',forecast_extended)\n",
    "# Plot the actual values and the forecasted values\n",
    "plt.plot(all_dates[:len(values)], values,label='Actual')\n",
    "plt.plot(all_dates[len(values):], forecast_extended,'r+', label='Forecast')\n",
    "\n",
    "plt.xlabel('Time')\n",
    "plt.ylabel('Temperature')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 235,
   "id": "m9opUhEzYeLQ",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 675
    },
    "id": "m9opUhEzYeLQ",
    "outputId": "dc4920a0-2f6d-435b-b5bc-8ff3ce424b72"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "type(values) <class 'numpy.ndarray'>\n",
      "[ 8.2   8.25  8.6   8.8   8.9   8.75  8.85  9.1   9.5   9.15  8.9   8.6\n",
      "  8.65  8.4   7.45  8.25  8.4   8.15  7.7   7.5   7.3   7.3   7.15  7.05\n",
      "  6.95  6.85  6.75  6.65  6.5   6.6   6.9   7.15  7.7   8.    8.25  8.75\n",
      "  9.15  9.2   9.    9.    8.8   8.8   8.9   8.6   8.7   8.35  7.95  8.25\n",
      "  8.2   8.35  8.    8.1   7.85  7.35  7.75  8.    8.3   8.2   8.7   9.5\n",
      "  9.8  10.6  10.5  10.85 10.85 10.7  10.25  9.5   7.8   7.85  8.25  8.\n",
      "  7.65  7.1   6.8   6.5   6.35  6.4   6.45  6.45  6.55  6.85  7.05  7.15\n",
      "  7.2   7.3   7.3   7.25  7.15  7.05  6.85  6.55  6.25  6.05  5.95  5.9\n",
      "  5.85  5.8   5.8   5.8   5.9   6.25  6.85  7.5   7.9 ]\n",
      "1/1 [==============================] - 0s 54ms/step\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-235-23fed817f36f>:35: FutureWarning: Argument `closed` is deprecated in favor of `inclusive`.\n",
      "  future_dates = pd.date_range(last_date, freq='1H',periods=future_steps, closed='right')\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 34176 characters]\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "\n",
    "# Assuming you have the time series data stored in a DataFrame called 'data'\n",
    "\n",
    "# Extract the values from the 'value' column as a NumPy array\n",
    "values = data['value'].values\n",
    "print('type(values)',type(values))\n",
    "print(values)\n",
    "# Normalize the values to a range between 0 and 1\n",
    "scaler = MinMaxScaler()\n",
    "normalized_values = scaler.fit_transform(values.reshape(-1, 1))\n",
    "\n",
    "# Create input sequence for prediction\n",
    "def create_input_sequence(data, time_steps):\n",
    "    sequence = data[-time_steps:]  # Take the last 'time_steps' values as input\n",
    "    return np.array(sequence).reshape(1, -1, 1)\n",
    "\n",
    "# Define the number of future points to forecast\n",
    "future_steps = 3\n",
    "\n",
    "# Create input sequence for prediction using the last 'time_steps' values\n",
    "input_sequence = create_input_sequence(normalized_values, time_steps)\n",
    "\n",
    "# Generate the forecast for future points\n",
    "forecast = model.predict(input_sequence)\n",
    "\n",
    "# Inverse transform the normalized forecast to the original scale\n",
    "forecast = scaler.inverse_transform(forecast)\n",
    "\n",
    "# Generate future dates for the forecasted points\n",
    "last_date = data['datetime'].iloc[-1]\n",
    "future_dates = pd.date_range(last_date, freq='1H',periods=future_steps, closed='right')\n",
    "\n",
    "# Concatenate the original dates and the future dates\n",
    "all_dates = pd.concat([data['datetime'], pd.Series(future_dates)])\n",
    "\n",
    "# Extend the forecast array to match the length of all_dates\n",
    "forecast_extended = np.concatenate([values[-1:], forecast.flatten()])\n",
    "\n",
    "# Plot the actual values and the forecasted values\n",
    "plt.plot(all_dates[:len(values)], values, label='Actual', color='blue')\n",
    "plt.plot(all_dates[len(values):], forecast_extended, label='Forecast', color='green')\n",
    "\n",
    "plt.xlabel('Time')\n",
    "plt.ylabel('Value')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 269,
   "id": "dUBFCM2KaAOp",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "dUBFCM2KaAOp",
    "outputId": "4f947f4d-2546-4d27-9bca-0673c340c372"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2010-05-05 08:00:00+00:00\n",
      "DatetimeIndex(['2010-05-05 09:00:00+00:00', '2010-05-05 10:00:00+00:00',\n",
      "               '2010-05-05 11:00:00+00:00'],\n",
      "              dtype='datetime64[ns, UTC]', freq='H')\n",
      "<class 'pandas.core.indexes.datetimes.DatetimeIndex'>\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "input_sequence [[[0.01980198]\n",
      "  [0.00990099]\n",
      "  [0.        ]\n",
      "  [0.        ]\n",
      "  [0.        ]\n",
      "  [0.01980198]\n",
      "  [0.08910891]\n",
      "  [0.20792079]\n",
      "  [0.33663366]\n",
      "  [0.41584158]]]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-269-a51bd15a9cb2>:24: FutureWarning: Argument `closed` is deprecated in favor of `inclusive`.\n",
      "  future_dates = pd.date_range(last_date, freq='1H',periods=4, closed='right')\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/1 [==============================] - 0s 403ms/step\n",
      "forecast_norm [[0.29481745]]\n",
      "forecast from 20 last steps [[7.2888284]]\n",
      "[[7.2888284]]\n",
      "input_sequence [[[0.00990099]\n",
      "  [0.        ]\n",
      "  [0.        ]\n",
      "  [0.        ]\n",
      "  [0.01980198]\n",
      "  [0.08910891]\n",
      "  [0.20792079]\n",
      "  [0.33663366]\n",
      "  [0.41584158]\n",
      "  [0.2948175 ]]]\n",
      "1/1 [==============================] - 0s 22ms/step\n",
      "forecast_norm [[0.33458707]]\n",
      "forecast from 20 last steps [[7.489665]]\n",
      "input_sequence [[[0.        ]\n",
      "  [0.        ]\n",
      "  [0.        ]\n",
      "  [0.01980198]\n",
      "  [0.08910891]\n",
      "  [0.20792079]\n",
      "  [0.33663366]\n",
      "  [0.41584158]\n",
      "  [0.2948175 ]\n",
      "  [0.33458713]]]\n",
      "1/1 [==============================] - 0s 20ms/step\n",
      "forecast_norm [[0.37149787]]\n",
      "forecast from last steps [[7.6760645]]\n",
      "df                      datetime  value      time\n",
      "0   2010-05-01 00:00:00+00:00   8.20  00:00:00\n",
      "1   2010-05-01 01:00:00+00:00   8.25  01:00:00\n",
      "2   2010-05-01 02:00:00+00:00   8.60  02:00:00\n",
      "3   2010-05-01 03:00:00+00:00   8.80  03:00:00\n",
      "4   2010-05-01 04:00:00+00:00   8.90  04:00:00\n",
      "..                        ...    ...       ...\n",
      "100 2010-05-05 04:00:00+00:00   5.90  04:00:00\n",
      "101 2010-05-05 05:00:00+00:00   6.25  05:00:00\n",
      "102 2010-05-05 06:00:00+00:00   6.85  06:00:00\n",
      "103 2010-05-05 07:00:00+00:00   7.50  07:00:00\n",
      "104 2010-05-05 08:00:00+00:00   7.90  08:00:00\n",
      "\n",
      "[105 rows x 3 columns]\n",
      "len(df) 105\n",
      "type(df) <class 'pandas.core.frame.DataFrame'>\n",
      "all_dates 0     2010-05-01 00:00:00+00:00\n",
      "1     2010-05-01 01:00:00+00:00\n",
      "2     2010-05-01 02:00:00+00:00\n",
      "3     2010-05-01 03:00:00+00:00\n",
      "4     2010-05-01 04:00:00+00:00\n",
      "                 ...           \n",
      "103   2010-05-05 07:00:00+00:00\n",
      "104   2010-05-05 08:00:00+00:00\n",
      "0     2010-05-05 09:00:00+00:00\n",
      "1     2010-05-05 10:00:00+00:00\n",
      "2     2010-05-05 11:00:00+00:00\n",
      "Length: 108, dtype: datetime64[ns, UTC]\n",
      "len(all_dates) 108\n",
      "type(all_dates) <class 'pandas.core.series.Series'>\n",
      "forecast_extended [7.2888284 7.489665  7.6760645]\n",
      "len(forecast_extended) 3\n",
      "type(forecast_extended) <class 'numpy.ndarray'>\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 52624 characters]\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Sample data\n",
    "#data = {\n",
    "#    'datetime': ['2010-05-01 00:00:00+00:00', '2010-05-01 01:00:00+00:00', '2010-05-01 02:00:00+00:00', '2010-05-01 03:00:00+00:00', '2010-05-01 04:00:00+00:00'],\n",
    "#    'value': [8.20, 8.25, 8.60, 8.80, 8.90]\n",
    "#}\n",
    "\n",
    "# Create a DataFrame from the data\n",
    "\n",
    "df = pd.DataFrame(data)\n",
    "\n",
    "# Convert the 'datetime' column to datetime data type\n",
    "df['datetime'] = pd.to_datetime(df['datetime'])\n",
    "\n",
    "# Extract the time from the 'datetime' column\n",
    "df['time'] = df['datetime'].dt.time\n",
    "\n",
    "\n",
    "# Generate future dates for the forecasted points\n",
    "last_date = df['datetime'].iloc[-1]\n",
    "print(last_date)\n",
    "future_dates = pd.date_range(last_date, freq='1H',periods=4, closed='right')\n",
    "print(future_dates)\n",
    "print(type(future_dates))\n",
    "print(type(df))\n",
    "# Concatenate the original dates and the future dates\n",
    "all_dates = pd.concat([df['datetime'], pd.Series(future_dates)])\n",
    "\n",
    "\n",
    "\n",
    "# 1st Normalize the values to a range between 0 and 1\n",
    "scaler = MinMaxScaler()\n",
    "normalized_values = scaler.fit_transform(values.reshape(-1, 1))\n",
    "# Create input sequence for prediction using the last 'time_steps' values\n",
    "input_sequence = create_input_sequence(normalized_values, 10)\n",
    "print('input_sequence',input_sequence)\n",
    "# Generate the forecast for future points\n",
    "forecast_norm = model.predict(input_sequence)\n",
    "print('forecast_norm',forecast_norm)\n",
    "# Inverse transform the normalized forecast to the original scale\n",
    "forecast_1 = scaler.inverse_transform(forecast_norm)\n",
    "print('forecast from 20 last steps',forecast_1)\n",
    "\n",
    "\n",
    "# 2nd Normalize the values to a range between 0 and 1\n",
    "print(forecast_1)\n",
    "values_new = np.concatenate((values,forecast_1[0]))\n",
    "scaler = MinMaxScaler()\n",
    "normalized_values = scaler.fit_transform(values_new.reshape(-1, 1))\n",
    "# Create input sequence for prediction using the last 'time_steps' values\n",
    "input_sequence = create_input_sequence(normalized_values, 10)\n",
    "print('input_sequence',input_sequence)\n",
    "# Generate the forecast for future points\n",
    "forecast_norm = model.predict(input_sequence)\n",
    "print('forecast_norm',forecast_norm)\n",
    "# Inverse transform the normalized forecast to the original scale\n",
    "forecast_2 = scaler.inverse_transform(forecast_norm)\n",
    "print('forecast from 20 last steps',forecast_2)\n",
    "\n",
    "# 3rd Normalize the values to a range between 0 and 1\n",
    "values_3 = np.concatenate((values_new,forecast_2[0]))\n",
    "scaler = MinMaxScaler()\n",
    "normalized_values = scaler.fit_transform(values_3.reshape(-1, 1))\n",
    "# Create input sequence for prediction using the last 'time_steps' values\n",
    "input_sequence = create_input_sequence(normalized_values, 10)\n",
    "print('input_sequence',input_sequence)\n",
    "# Generate the forecast for future points\n",
    "forecast_norm = model.predict(input_sequence)\n",
    "print('forecast_norm',forecast_norm)\n",
    "# Inverse transform the normalized forecast to the original scale\n",
    "forecast_3 = scaler.inverse_transform(forecast_norm)\n",
    "print('forecast from last steps',forecast_3)\n",
    "\n",
    "\n",
    "\n",
    "# Extend the forecast array to match the length of all_dates\n",
    "forecast_extended = np.concatenate([forecast_1.flatten(),forecast_2.flatten(),forecast_3.flatten()])\n",
    "\n",
    "print('df',df)\n",
    "print('len(df)',len(df))\n",
    "print('type(df)',type(df))\n",
    "print('all_dates',all_dates)\n",
    "print('len(all_dates)',len(all_dates))\n",
    "print('type(all_dates)',type(all_dates))\n",
    "print('forecast_extended',forecast_extended)\n",
    "print('len(forecast_extended)',len(forecast_extended))\n",
    "print('type(forecast_extended)',type(forecast_extended))\n",
    "# Plotting\n",
    "plt.plot(df['datetime'], df['value'],label='Historical T data')\n",
    "plt.plot(future_dates, forecast_extended,'r+',label='Forecast')\n",
    "plt.xlabel('Datetime')\n",
    "plt.ylabel('Value')\n",
    "plt.title('Forecast of air_temperature for 3 next hours 09:00-11:00')\n",
    "plt.xticks(rotation=90)\n",
    "plt.grid(True)\n",
    "plt.show()\n"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "include_colab_link": true,
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
