Skip to content

Commit

Permalink
Merge pull request palexandremello#6 from spestana/master
Browse files Browse the repository at this point in the history 
Updates to ABI_Downloader to handle ABI products that don't have a "channel"
  • Loading branch information
@palexandremello
palexandremello authored Jan 15, 2021
2 parents f355238 + 37f2838 commit a0a8e31
Show file tree
Hide file tree
Showing 3 changed files with 952 additions and 51 deletions.
44 changes: 28 additions & 16 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,41 +3,50 @@
[![Downloads](https://pepy.tech/badge/goespy)](https://pepy.tech/project/goespy)
[![Version](https://img.shields.io/pypi/v/goespy.svg)](https://pypi.org/project/goespy/)

It's a Python package to acess and get the dataset from GOES satellite next generation on Amazon Web Service(AWS)
This Python package provides functions to download [GOES (GOES-16 & GOES-17) products from Amazon Web Services (AWS)](https://registry.opendata.aws/noaa-goes/).

## HOWTO-Install
## Installing goes-py

### 1. Source code:
### Option 1: Installing from source code:

If you want to build this package, just clone this repository:

```bash
$ git clone https://github.com/palexandremello/goes-py.git

$ cd goes-py

$ python setup.py install
```
But if you don't want to build the cloned repository, just use the pip on terminal.

### 2. Pip Install:
### Option 2: Installing with pip:

If you don't want to build from the cloned source code repository, just use pip in a terminal.

Acess a terminal and use the command **pip**:
Access a terminal and use the command **pip**:
```bash
$ pip install goespy
```
### 3. Update the package:

## Updating goes-py

### Option 1: Install again from source code:

The goespy will have more new version in the future, so when this release comes. you need upgrade your package, so use this command on terminal. (new release 0.2v)
Build the package again from the new version of the source code to update goes-py.

### Option 2: Update with pip:

The goes-py package will have new versions in the future, you will need upgrade your package to use these future releases. Use this command in a terminal to update goes-py. (new release 0.2v)

```bash
$ pip install --upgrade goespy
```
Or if you want build again the new source code

## Examples how to use:
## Examples how to use:

This package has two main function, can be used to get dataset from GOES:
This package has two main functions to download GOES products from AWS:

### 1. HOW TO get from ABI-sensors:
### 1. HOW TO download ABI products:

First open a file with the filename **firstexample.py** and put the next command on header's script.
Expand All @@ -59,7 +68,7 @@ month: month string
day: day string
hour: hour string, but it's need be UTC coordinate not local time
product: "ABI-sensors" for this example we will use FullDisk L2
channel: channels of your choose, can be C01 at C16**
channel: Required only for "ABI-L1b-Rad" and "ABI-L2-CMIP" products (can be C01 through C16). Channel is ignored for other ABI products.
```
Below do the initialization for these variable in your firstexample.py :

Expand Down Expand Up @@ -89,14 +98,17 @@ python firstexample.py

After the download to be finishes, check your destination path (**Linux and mac OS X users**) and your dataset will be in a directory with the same name from the satellite used on bucket variable. In this case: **goes16**.

### 2. HOW TO get from GLM total lightning:
### 2. HOW TO download GLM total lightning product:

```py
from goespy import GLM_Downloader
```

For the GLM use an especific example on the **examples/** in the source directory.
For the GLM use see the example in **examples/** in the source directory.

## Contributors:
Centro de Pesquisas e Previsões Meteorológicas Prof. Darci Pegoraro Casarin (<a href="https://wp.ufpel.edu.br/cppmet/">CPMet</a>) for the place and computers necessary to work on this project

* [Paulo Alexandre Mello](https://github.com/palexandremello)
* Centro de Pesquisas e Previsões Meteorológicas Prof. Darci Pegoraro Casarin (<a href="https://wp.ufpel.edu.br/cppmet/">CPMet</a>) for the place and computers necessary to work on this project
* [Steven Pestana](https://github.com/spestana/)

865 changes: 865 additions & 0 deletions examples/plotting_data/goespy_download_plot_example.ipynb
View file

Large diffs are not rendered by default.

94 changes: 59 additions & 35 deletions goespy/Downloader.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

def ABI_Downloader(home, bucket, year, month, day, hour, product, channel):
""" ABI_Downloader(home, bucket,year,month,day,hour,product,channel): All these variables are strings.
The first argument is the Bucket it's the reposity where has the contents from the satellite, example:
The second argument is the Bucket it's the reposity where has the contents from the satellite, example:
home = string, set directory to download ABI products
bucket='noaa-goes16'
Expand All @@ -13,7 +13,7 @@ def ABI_Downloader(home, bucket, year, month, day, hour, product, channel):
day = can be List or a single string for day date: example = ['10','20','30'] or "20"
hour = can be List or a single string to hour, and need be UTC coordinate time date: example = ['06','12','18'] or "06"
product = can be a List or a single string for ABI sensors products from GOES satellite next-generation example: ["ABI-L2-CMIPF"] or "ABI-L1b-RadF"
channel = can be a List or a single string for the channels from ABI sensors. Example = ['01','02'] or "13"
channel = Required only for "ABI-L1b-Rad" and "ABI-L2-CMIP" products. Can be a List or a single string for the channels from ABI sensors. Example = ['01','02'] or "13" (channel is ignored for other ABI products)
"""
from goespy.utils import __isAList
from goespy.utils import ProgressPercentage
Expand All @@ -32,7 +32,7 @@ def ABI_Downloader(home, bucket, year, month, day, hour, product, channel):
year, month, day, product, hour,channel, julianDay=julianDay)

## for loop to all variable year (it's a list var)
for i in year:
for y in year:

## same think above
for mth in month:
Expand All @@ -51,39 +51,63 @@ def ABI_Downloader(home, bucket, year, month, day, hour, product, channel):
objs = goes16.objects.filter(
Delimiter='/',
Prefix="{0}/{1}/{2}/{3}/".format(
prod, i, julianDay[days], nindex))
for object in objs:
prod, y, julianDay[days], nindex))

for obj in objs:
## the keys it's a "path"+"filename" in the bucket, solution
### we need only the filename, that's why used the rsplit function.
filename = object.key.rsplit('/', 1)[1]
for ch in channel:
## when the filename has the same Channel from the user channel variable
## call the function from download, but before it's done somes check files and directory
if filename.partition(ch)[1] == ch:

filename = obj.key.rsplit('/', 1)[1]

if ( (prod[:-1] == "ABI-L1b-Rad") or (prod[-1] == "ABI-L2-CMIP") ):
for ch in channel:
## when the filename has the same Channel from the user channel variable
## call the function from download, but before it's done somes check files and directory
if filename.partition(ch)[1] == ch:

# creating the new directory where we will put the dataset from the bucket
# creating the new directory where we will put the dataset from the bucket

path = checkData.createPathGoesData(home, bucket,
i, mth, day[days], prod, nindex, ch)

#checking if the file exist on the new directory and your size
if checkData.checkFiles(path, filename):

if checkData.checkSize(
path, filename, object.size):

pass
path = checkData.createPathGoesData(home, bucket,
y, mth, day[days], prod, nindex, ch)

#checking if the file exist on the new directory and your size
if checkData.checkFiles(path, filename):

if checkData.checkSize(
path, filename, obj.size):
pass

else:

# Downloading the file with the boto3
goes16.download_file(
obj.key, path + filename,Callback=ProgressPercentage(filename,obj.size))
else:

# Downloading the file with the boto3
goes16.download_file(
object.key, path + filename,Callback=ProgressPercentage(filename,object.size))
else:
obj.key, path + filename,Callback=ProgressPercentage(filename,obj.size))
else:
# creating the new directory where we will put the dataset from the bucket

path = checkData.createPathGoesData(home, bucket,
y, mth, day[days], prod, nindex)

#checking if the file exist on the new directory and your size
if checkData.checkFiles(path, filename):

if checkData.checkSize(
path, filename, obj.size):
pass

else:

# Downloading the file with the boto3
goes16.download_file(
object.key, path + filename,Callback=ProgressPercentage(filename,object.size))
goes16.download_file(obj.key, path + filename,Callback=ProgressPercentage(filename,obj.size))
else:

# Downloading the file with the boto3
goes16.download_file(obj.key, path + filename,Callback=ProgressPercentage(filename,obj.size))
days += 1

return 0
Expand Down Expand Up @@ -115,7 +139,7 @@ def GLM_Downloader(home, bucket, year, month, day, hour):
year, month, day, product, hour, julianDay = __isAList(
year, month, day, product, hour, julianDay=julianDay)

for i in year:
for y in year:
for mth in month:
while days <= len(day) - 1 and days <= len(julianDay) - 1:
for prod in product:
Expand All @@ -126,31 +150,31 @@ def GLM_Downloader(home, bucket, year, month, day, hour):
objs = goes16.objects.filter(
Delimiter='/',
Prefix="{0}/{1}/{2}/{3}/".format(
prod, i, julianDay[days], nindex))
prod, y, julianDay[days], nindex))

#print("{0}/{1}/{2}/{3}/".format(prod,i,julianDay[days],nindex))
for object in objs:
#print("{0}/{1}/{2}/{3}/".format(prod,y,julianDay[days],nindex))
for obj in objs:

filename = object.key.rsplit('/', 1)[1]
filename = obj.key.rsplit('/', 1)[1]
## creating the directory where will put the dataset from the bucket
pathFile = checkData.createPathGoesData(
home, bucket, i, mth, day[days], prod, nindex)
home, bucket, y, mth, day[days], prod, nindex)

# checking if the data exist and your size!!!
if checkData.checkFiles(pathFile, filename):
if checkData.checkSize(pathFile, filename,
object.size):
obj.size):
pass

else:

# Downloading the file with the boto3
goes16.download_file(object.key,pathFile+filename,Callback=ProgressPercentage(filename,object.size))
goes16.download_file(obj.key,pathFile+filename,Callback=ProgressPercentage(filename,obj.size))
print('\n')
else:

# Downloading the file with the boto3
goes16.download_file(object.key,pathFile+filename,Callback=ProgressPercentage(filename,object.size))
goes16.download_file(obj.key,pathFile+filename,Callback=ProgressPercentage(filename,obj.size))
print('\n')

days += 1
Expand Down

0 comments on commit a0a8e31

Please sign in to comment.