5

Author: TacNayn

.DS_Store

@@ -1,6 +1,6 @@
 ---
 title: "Introduction to SQL"
-teaching: 30
+teaching: 20
 exercises: 5
 questions:
 - "What is a relational database and why should I use it?"
@@ -20,12 +20,8 @@ keypoints:
 
 _Note: this should have been done by participants before the start of the workshop._
 
-We use [SQLite Manager](https://addons.mozilla.org/en-us/firefox/addon/sqlite-manager/)
-and the 
-[Portal Project dataset](https://figshare.com/articles/Portal_Project_Teaching_Database/1314459)
-throughout this lesson. See [Setup](/sql-ecology-lesson/setup/) for
-instructions on how to download the data, and also how to install and open
-SQLite Manager.
+See [Setup](../setup/) for
+instructions on how to download the data, and also how to install jupyter notebook  
 
 ## What is SQL?
 
@@ -36,7 +32,7 @@ These queries can allow you to perform a number of actions such as: insert, upda
 
 > ## Think about this situation
 >
-> Imagine if you are a owner of a convenience store, and you are trying to record your soda purchase record <br>
+> Imagine if you are a owner of a convenience store, and you are trying to record your soda purchase. <br>
 > In each invoice, it contains the following information: <br>
 > Invoice id, Date, Category, Soda name, Volume, Cost, Retail Price, Vendor, Number of bottle purchased <br>
 > How would you store the data?  
@@ -46,19 +42,20 @@ These queries can allow you to perform a number of actions such as: insert, upda
 If you store all these invoice information in one Excel file, 
 What problem could raise from this approach?  
 <b>Data redundency:</b> <br>
-![alt text](../img/00_1.png)
+![alt text](../img/00_1.png){:width="700px"}
 <br><br>
 Imagine if you consistantly purchased some Big Dog Cola from LCDM Beverage vendor every day for 5 days, 
 Notice these columns: Category, Soda_name, Volume, Cost, Retail_Price, Vendor, Vendor number
 With tranditional file approach, you have to record exact same information in these columns 5 times.  
 <b>Data inconsistancy:</b> <br>
-![alt text](../img/00_2.png)
+![alt text](../img/00_2.png){:width="700px"}
 <br><br>
 Imagine if the vendor changed its phone number. Then multiple changes has to be made. 
 There are only 5 rows so it might be easy to change everything. If the data size gets large, mistakes are likely to occur.  
 
 If you thought about storing these information in few different Excel files, <b>great idea! You are on the right track </b><br>
-However, if you want information from all files at the same time, how do you combine them? If each file contains thoudans of rows, Ah...  
+However, if you want information from all files at the same time, how do you combine them? If each file contains thoudans of rows, Ah... <br>
+The largest table that Excel can handle is 1,048,576 * 16,384. Excel can be very slow when working with large number of data. Moreover, in the real word, you can easily get 1 million rows of data...   
 ![alt text](../img/tuxue.png){:height="100px" width="100px"}
 
 ## Goals
@@ -103,6 +100,7 @@ Using a relational database serves several purposes.
 * It's fast, even for large amounts of data.
 * It improves quality control of data entry (type constraints and use of forms in MS Access, Filemaker, Oracle Application Express etc.)
 * The concepts of relational database querying are core to understanding how to do similar things using programming languages such as R or Python.
+* Many big companies are using Relational Database to store financial data. As business students, we can learn how to pool useful data by ourselves. 
 
 ## Database Management Systems (DBMS)
 
@@ -111,47 +109,12 @@ relational data. We're going to use SQLite today, but basically everything we
 teach you will apply to the other database systems as well (e.g. MySQL,
 PostgreSQL, MS Access, MS SQL Server, Oracle Database and Filemaker Pro). The 
 only things that will differ are the details of exactly how to import and 
-export data and the [details of data types](#datatypediffs).
+export data and the data types.  
 
 Look at the [popularity of database](https://db-engines.com/en/ranking) <br>
 ![alt text](../img/dbms.png){:height="170px"} <br>
 Top 4 are all Relational Database Management Systems (RDBMS). More and more companies choose to use relational database. 
 
-## Relational databases
-
-Let's look at a pre-existing database, the `soda.db`
-file that we downloaded during
-[Setup](/sql-business/setup/). Clicking on the "open file" icon, then
-find that file and clicking on it will open the database.
-
-You can see the tables in the database by looking at the left hand side of the
-screen under Tables, where each table corresponds to one of the `csv` files 
-we were exploring earlier.  To see the contents of any table, click on it, and
-then click the “Browse and Search” tab in the right panel.  This will 
-give us a view that we're used to - just a copy of the table.  Hopefully this 
-helps to show that a database is, in some sense, just a collection of tables, 
-where there's some value in the tables that allows them to be connected to each 
-other (the "related" part of "relational database").  
-
-The leftmost tab, "Structure", provides some metadata about each table.  It 
-describes the columns, often called *fields*. (The rows of a database table 
-are called *records*.)  If you scroll down in the Structure view, you'll 
-see a list of fields, their labels, and their data *type*.  Each field contains 
-one variety or type of data, often numbers or text.  You can see in the 
-`surveys` table that most fields contain numbers (integers) while the `species` 
-table is nearly all text.  
-
-The "Execute SQL" tab is blank now - this is where we'll be typing our queries 
-to retrieve information from the database tables.  
-
-To summarize: 
-
-* Relational databases store data in tables with fields (columns) and records
-  (rows)
-* Data in tables has types, and all values in a field have
-  the same type ([list of data types](#datatypes))
-* Queries let us look up data or make calculations based on columns
-
 ## Database Design
 
 * Every row-column combination contains a single *atomic* value, i.e., not
@@ -162,3 +125,11 @@ To summarize:
     * Needs an identifier in common between tables – shared column - to
        reconnect (known as a *foreign key*).
 
+To summarize: 
+
+* Relational databases store data in tables with fields (columns) and records
+  (rows)
+* Each row is uniquely identified by a Primary Key  
+* Queries let us look up data or make calculations based on columns
+
+

_episodes/00-sql-introduction.md

@@ -1,7 +1,7 @@
 ---
 title: "Getting to know your data"
 teaching: 30
-exercises: 5
+exercises: 0
 questions:
 - "How to connect to database with python?"
 - "What is in the database?"
@@ -22,7 +22,7 @@ keypoints:
 
 _Note: this should have been done by participants before the start of the workshop._
 
-See [Setup](/sql-ecology-lesson/setup/) for
+See [Setup](../setup/) for
 instructions on how to download the data, and also how to install and open SQLite Manager.
 
 ## Import

_episodes/01-sql-intro2.md _episodes/01-sql-data.md

@@ -191,9 +191,48 @@ To truly be alphabetical, we might want to order by genus then species.
 > ## Challenge
 >
 > - Write a query that returns Item_Description, Bottle_Cost, volume and retail price
-> of the soda, sorted firstly with retail price in ascending order, then with volume in descending order.  
+> of the soda, sorted firstly with retail price in descending order, then with volume in ascending order.  
 {: .challenge}
 
+## Dealing with dates  
+Firstly, we take a look at the `invoice_info` table  
+```
+SELECT * FROM invoice_info;
+```
+The date in sqlite3 can be stored as multiple string formats: 
+```
+    YYYY-MM-DD
+    YYYY-MM-DD HH:MM
+    YYYY-MM-DD HH:MM:SS
+    YYYY-MM-DD HH:MM:SS.SSS
+    YYYY-MM-DDTHH:MM
+    YYYY-MM-DDTHH:MM:SS
+    YYYY-MM-DDTHH:MM:SS.SSS
+    HH:MM
+    HH:MM:SS
+    HH:MM:SS.SSS
+    now
+    DDDDDDDDDD 
+```
+In our database, the date were stored as "YYYY-MM-DD" format. We can extract the year/month/date with `strftime` function. For example: 
+```
+Select *, strftime('%Y', Date) AS YEAR
+FROM invoice_info; 
+```
+For month and day, simply replace `%Y` with `%m` or `%d`. You can also do `%Y-%m` to get the year and month. 
+You can get all invoice after 2017 by: 
+```
+SELECT * FROM invoice_info
+WHERE Date >= "2017-01-01";
+```
+You can get a invoice from a range of time by:  
+```
+SELECT * FROM invoice_info
+WHERE Date BETWEEN "2017-01-01" AND "2017-02-27"; 
+```
+Note that `BETWEEN` is inclusive, that is, invoices at 2017-01-01 and 2017-02-27 will be returned  
+If you are interested at more cool things you can do with dates, heres the [Documentation](https://www.sqlite.org/lang_datefunc.html)
+
 ## Order of execution
 
 Another note for ordering. We don’t actually have to display a column to sort by
@@ -221,7 +260,7 @@ we recommend to put each clause on its own line.
 > ## Challenge
 >
 > - Let's try to combine what we've learned so far in a single
-> query.  Using the **item_info** table write a query to display the three date fields,
+> query.  Using the **item_info** table write a query to display the three data fields,
 > `Item_Description`, `Bottle_Volume_ml` and Retail Price for 6 packs (give it an alias `Six_Pack_Price`), for
 > all sodas that are usually sold in 6 packs, ordered firstly by `Six_Pack_Price`, then alphabetically by the `Item_Description`.
 > - Write the query as a single line, then put each clause on its own line, and

_episodes/02-sql-basic-queries.md

@@ -96,8 +96,8 @@ of these groups (`HAVING`).
 
 > ## Challenge
 >
-> What sodas were sold more than 100000 bottles from 2012 to 2017?  
-> In another word, write a query that returns item_id in invoice_info table
+> What sodas were sold more than 100000 bottles in the whole database?  
+> In another word, write a query that returns item_id and total bottles sold in invoice_info table
 > Where the total bottles sold is more than 100000
 {: .challenge}