Merge pull request amazon-archives#24 from zhongnansu/test

Provide user option to toggle to use aws sigV4 authentication

Author: abbashus

README.md

@@ -3,52 +3,59 @@
 ODFE: Open Distro for Elasticsearch 
 
 ODFE SQL CLI is a stand alone Python application and can be launched by a wake word `odfesql`. It serves as a support only for 
-[Open Distro SQL plugin for Elasticsearch](https://opendistro.github.io/for-elasticsearch-docs/docs/sql/). You can run 
-it on any OS we support, and connect to any valid remote endpoint without installing Elasticsearch.
+[Open Distro SQL plugin for Elasticsearch](https://opendistro.github.io/for-elasticsearch-docs/docs/sql/). User must have ODFE SQL
+plugin installed to the Elasticsearch instance to connect to. Usr can run this CLI on any OS we support, and connect to any valid
+remote Elasticsearch endpoint.
 
 
 ## Installation
 - `pip install odfesql` 
-- odfe sql cli is compatible with Python 3, because Python 2 is no longer maintained since 01/01/2020 https://pythonclock.org/ 
+- ODFE SQL CLI is only compatible with Python 3, because Python 2 is no longer maintained since 01/01/2020 https://pythonclock.org/ 
 
 
 ## Configuration
-- A config file is automatically created at `~/.config/escli/config` at first launch. 
-See the file itself for a description of all available options.
-
+- A config file is automatically created at `~/.config/odfesql-cli/config` at first launch (for MacOS and Linux). 
+Check out [clirc](./odfesql_cli/conf/clirc) for details of all available configurations.
+- It can be configured and will be auto-loaded next time.
 
 ## Features
 - Multiline input
-- Auto-completion with index suggestion
+- Input Auto-completion with index suggestion
+- Syntax highlighting
 - Formatted output
     - Tabular format
     - Fields name with color
-    - Enable horizontal display (by default) and vertical display when output is too wide
+    - Enable horizontal display (by default) and vertical display when output is too wide, for better visualization
     - Pagination for long output
-- Syntax highlighting
-- Connect to Elasticsearch node/cluster with/without security on either **ES localhost, Open Distro ES, or AWS Elasticsearch Domain**.
-Refer to [test plan](./tests/test_plan.md) on how to connect to different instance with/without security
-- Load Config file
-- Run single query from Command Line with parameters
-    - *endpoint:* no need to specify a parameter, anything follow by wake word `odfesql` should be the endpoint. 
-    By default, it’s http://localhost:9200
-    - *--help:* help page for options and params
-    - *-q:* follow by a single query user wants to run.
+
+- Connect to Elasticsearch with/without security enabled on either **Elasticsarch, Elasticsarch OSS, or AWS Elasticsearch Domain**.
+- Load config file
+
+## Basic Usage
+- ![](./screenshots/usage.gif)
+- The CLI supports all types of query that ODFE SQL supports. Refer to [ODFE SQL basic usage](https://github.com/opendistro-for-elasticsearch/sql#basic-usage)
+    
+- Configurable connection properties
+    - *endpoint:* no need to specify an option, anything follow by launch word `odfesql` is regarded as the endpoint. 
+    If user doesn't provide an endpoint, by default it will try to connect to http://localhost:9200
+    - *-u/-w:* username and password. User needs to provide credentials when connecting to:
+        - Elasticsearch with X-pack security enabled
+        - Elasticsearch OSS with [Open distro Security Plugin](https://opendistro.github.io/for-elasticsearch-docs/docs/install/plugins/) installed
+        - IP-based AWS Elasticsearch domain with [Fine Grained Access Control](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/fgac.html) enabled
+
+    - *--aws-auth:* turn on to use AWS sigV4 authentication. It can be configured by AWS CLI `aws configure` command. ODFE SQL
+    CLI will try to retrieve this config to connect.
+
+- Run single query from command line with options
+    - *--help:* help page for options
+    - *-q:* follow by a single query
     - *-f:* support *jdbc/raw* format output
     - *-v:* display data vertically
-    - *-u:* username to connect to Elasticsearch 
-    - *-w:* password for username
     - *-e:* translate sql to DSL
 
-- Run the CLI with parameters
+- Run the CLI with options
     - *-p*: always use pager to display output
-    - *--esclirc*: provide path of config file to load.
-
-
-
-## Basic Usage
-- The CLI supports all types of query that ODFE SQL supports. See [ODFE SQL basic usage](https://github.com/opendistro-for-elasticsearch/sql#basic-usage)
-- ![](./screenshots/usage.gif)
+    - *--clirc*: provide path of config file to load
 
 
 

development_guide.md

@@ -33,7 +33,7 @@ any endpoint, it uses http://localhost:9200 by default.
 ### Workflow
 
 1. Update version number
-    1. Modify the version number in `_init_.py` under `escli` package. It will be used by `setup.py` for release.
+    1. Modify the version number in `_init_.py` under `odfesql_cli` package. It will be used by `setup.py` for release.
 2. Create/Update `setup.py` (if needed)
     1. For more details refer to https://packaging.python.org/tutorials/packaging-projects/#creating-setup-py 
 3. Update README.md, Legal and copyright files(if needed)
@@ -42,6 +42,7 @@ any endpoint, it uses http://localhost:9200 by default.
 4. Generate distribution archives
     1. Make sure you have the latest versions of `setuptools` and `wheel` installed:  `pip install --user --upgrade setuptools wheel`
     2. Run this command from the same directory where `setup.py` is located: `python3 setup.py sdist bdist_wheel`
+    3. Check artifacts under `sql-cli/dist/`, there should be a tar file and a whi file with correct version. Remove other deprecated artifacts.
 5. Upload the distribution archives to TestPyPI
     1. Register an account on [testPyPI](https://test.pypi.org/)
     2. `pip install --user —upgrade twine `

escli/__init__.py odfesql_cli/__init__.py

@@ -20,22 +20,22 @@
 # lines. End of line (return) is considered as the end of the statement.
 multi_line = True
 
-# If multi_line_mode is set to "escli", in multi-line mode, [Enter] will execute
+# If multi_line_mode is set to "odfesql_cli", in multi-line mode, [Enter] will execute
 # the current input if the input ends in a semicolon.
 # If multi_line_mode is set to "safe", in multi-line mode, [Enter] will always
 # insert a newline, and [Esc] [Enter] or [Alt]-[Enter] must be used to execute
 # a command.
-multi_line_mode = escli
+multi_line_mode = odfesql_cli
 
 # log_file location.
-# In Unix/Linux: ~/.conf/escli/log
-# In Windows: %USERPROFILE%\AppData\Local\dbcli\escli\log
+# In Unix/Linux: ~/.conf/odfesql-cli/log
+# In Windows: %USERPROFILE%\AppData\Local\dbcli\odfesql-cli\log
 # %USERPROFILE% is typically C:\Users\{username}
 log_file = default
 
 # history_file location.
-# In Unix/Linux: ~/.conf/escli/history
-# In Windows: %USERPROFILE%\AppData\Local\dbcli\escli\history
+# In Unix/Linux: ~/.conf/odfesql-cli/history
+# In Windows: %USERPROFILE%\AppData\Local\dbcli\odfesql-cli\history
 # %USERPROFILE% is typically C:\Users\{username}
 history_file = default
 

escli/conf/__init__.py odfesql_cli/conf/__init__.py

@@ -24,12 +24,12 @@
 def config_location():
     """Return absolute conf file path according to different OS."""
     if "XDG_CONFIG_HOME" in os.environ:
-        return "%s/escli/" % expanduser(os.environ["XDG_CONFIG_HOME"])
+        return "%s/odfesql-cli/" % expanduser(os.environ["XDG_CONFIG_HOME"])
     elif platform.system() == "Windows":
         # USERPROFILE is typically C:\Users\{username}
-        return "%s\\AppData\\Local\\dbcli\\escli\\" % os.getenv("USERPROFILE")
+        return "%s\\AppData\\Local\\dbcli\\odfesql-cli\\" % os.getenv("USERPROFILE")
     else:
-        return expanduser("~/.config/escli/")
+        return expanduser("~/.config/odfesql-cli/")
 
 
 def _load_config(user_config, default_config=None):
@@ -66,20 +66,20 @@ def _write_default_config(source, destination, overwrite=False):
 
 
 # https://stackoverflow.com/questions/40193112/python-setuptools-distribute-configuration-files-to-os-specific-directories
-def get_config(esclirc_file=None):
+def get_config(clirc_file=None):
     """
-    Get config for escli.
+    Get config for odfesql_cli.
 
     This config comes from either existing config in the OS, or create a config file in the OS, and write default config
     including in the package to it.
     """
-    from escli.conf import __file__ as package_root
+    from odfesql_cli.conf import __file__ as package_root
 
     package_root = os.path.dirname(package_root)
 
-    esclirc_file = esclirc_file or "%sconfig" % config_location()
-    default_config = os.path.join(package_root, "esclirc")
+    clirc_file = clirc_file or "%sconfig" % config_location()
+    default_config = os.path.join(package_root, "clirc")
 
-    _write_default_config(default_config, esclirc_file)
+    _write_default_config(default_config, clirc_file)
 
-    return _load_config(esclirc_file, default_config)
+    return _load_config(clirc_file, default_config)

escli/conf/esclirc odfesql_cli/conf/clirc

@@ -19,16 +19,16 @@
 from prompt_toolkit.application import get_app
 
 
-def es_is_multiline(escli):
+def es_is_multiline(odfesql_cli):
     """Return function that returns boolean to enable/unable multiline mode."""
 
     @Condition
     def cond():
         doc = get_app().layout.get_buffer_by_name(DEFAULT_BUFFER).document
 
-        if not escli.multi_line:
+        if not odfesql_cli.multi_line:
             return False
-        if escli.multiline_mode == "safe":
+        if odfesql_cli.multiline_mode == "safe":
             return True
         else:
             return not _multiline_exception(doc.text)

escli/config.py odfesql_cli/config.py

@@ -30,7 +30,7 @@ class ESConnection:
     as well as send user's SQL query to Elasticsearch.
     """
 
-    def __init__(self, endpoint=None, http_auth=None):
+    def __init__(self, endpoint=None, http_auth=None, use_aws_authentication=False):
         """Initialize an ESConnection instance.
 
         Set up client and get indices list.
@@ -46,6 +46,7 @@ def __init__(self, endpoint=None, http_auth=None):
         self.indices_list = []
         self.endpoint = endpoint
         self.http_auth = http_auth
+        self.use_aws_authentication = use_aws_authentication
 
     def get_indices(self):
         if self.client:
@@ -61,7 +62,7 @@ def get_aes_client(self):
         if credentials is not None:
             self.aws_auth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service)
         else:
-            click.secho(message="Can not retrieve credentials, check your aws config", fg="red")
+            click.secho(message="Can not retrieve your AWS credentials, check your AWS config", fg="red")
 
         aes_client = Elasticsearch(
             hosts=[{"host": str(self.endpoint), "port": 443}],
@@ -96,9 +97,8 @@ def set_connection(self, is_reconnect=False):
         if self.http_auth:
             es_client = self.get_open_distro_client()
 
-        elif str(self.endpoint).endswith("es.amazonaws.com"):
+        elif self.use_aws_authentication:
             es_client = self.get_aes_client()
-
         else:
             es_client = Elasticsearch([self.endpoint], verify_certs=True)
 

escli/esbuffer.py odfesql_cli/esbuffer.py

@@ -20,7 +20,7 @@
 from .config import config_location
 from .esconnection import ESConnection
 from .utils import OutputSettings
-from .essqlcli import ESSqlCli
+from .odfesql_cli import OdfeSqlCli
 from .formatter import Formatter
 
 click.disable_unicode_literals_warning = True
@@ -31,10 +31,10 @@
 @click.option("-q", "--query", "query", type=click.STRING, help="Run single query in non-interactive mode")
 @click.option("-e", "--explain", "explain", is_flag=True, help="Explain SQL to ES DSL")
 @click.option(
-    "--esclirc",
+    "--clirc",
     default=config_location() + "config",
-    envvar="ESCLIRC",
-    help="Location of esclirc file.",
+    envvar="CLIRC",
+    help="Location of clirc file.",
     type=click.Path(dir_okay=False),
 )
 @click.option(
@@ -64,7 +64,15 @@
     help="Always use pager to display output. If not specified, smart pager mode will be used according to the \
          length/width of output",
 )
-def cli(endpoint, query, explain, esclirc, result_format, is_vertical, username, password, always_use_pager):
+@click.option(
+    "--aws-auth",
+    "use_aws_authentication",
+    is_flag=True,
+    default=False,
+    help="Use AWS sigV4 to connect to AWS ELasticsearch domain",
+)
+def cli(endpoint, query, explain, clirc, result_format, is_vertical, username, password, always_use_pager,
+        use_aws_authentication):
     """
     Provide endpoint for Elasticsearch client.
     By default, it uses http://localhost:9200 to connect.
@@ -79,7 +87,7 @@ def cli(endpoint, query, explain, esclirc, result_format, is_vertical, username,
 
     # handle single query without more interaction with user
     if query:
-        es_executor = ESConnection(endpoint, http_auth)
+        es_executor = ESConnection(endpoint, http_auth, use_aws_authentication)
         es_executor.set_connection()
         if explain:
             output = es_executor.execute_query(query, explain=True, use_console=False)
@@ -95,9 +103,9 @@ def cli(endpoint, query, explain, esclirc, result_format, is_vertical, username,
         sys.exit(0)
 
     # use console to interact with user
-    escli = ESSqlCli(esclirc_file=esclirc, always_use_pager=always_use_pager)
-    escli.connect(endpoint, http_auth)
-    escli.run_cli()
+    odfesql_cli = OdfeSqlCli(clirc_file=clirc, always_use_pager=always_use_pager, use_aws_authentication=use_aws_authentication)
+    odfesql_cli.connect(endpoint, http_auth)
+    odfesql_cli.run_cli()
 
 
 if __name__ == "__main__":

escli/esconnection.py odfesql_cli/esconnection.py

@@ -44,25 +44,26 @@
 click.disable_unicode_literals_warning = True
 
 
-class ESSqlCli:
-    """ESSqlCli instance is used to build and run the ODFE SQL CLI."""
+class OdfeSqlCli:
+    """OdfeSqlCli instance is used to build and run the ODFE SQL CLI."""
 
-    def __init__(self, esclirc_file=None, always_use_pager=False):
+    def __init__(self, clirc_file=None, always_use_pager=False, use_aws_authentication=False):
         # Load conf file
-        config = self.config = get_config(esclirc_file)
+        config = self.config = get_config(clirc_file)
         literal = self.literal = self._get_literals()
 
         self.prompt_app = None
         self.es_executor = None
         self.always_use_pager = always_use_pager
+        self.use_aws_authentication = use_aws_authentication
         self.keywords_list = literal["keywords"]
         self.functions_list = literal["functions"]
         self.syntax_style = config["main"]["syntax_style"]
         self.cli_style = config["colors"]
         self.table_format = config["main"]["table_format"]
         self.multiline_continuation_char = config["main"]["multiline_continuation_char"]
         self.multi_line = config["main"].as_bool("multi_line")
-        self.multiline_mode = config["main"].get("multi_line_mode", "escli")
+        self.multiline_mode = config["main"].get("multi_line_mode", "odfesql_cli")
         self.null_string = config["main"].get("null_string", "null")
         self.style_output = style_factory_output(self.syntax_style, self.cli_style)
 
@@ -164,7 +165,7 @@ def echo_via_pager(self, text, color=None):
             click.echo(text, color=color)
 
     def connect(self, endpoint, http_auth=None):
-        self.es_executor = ESConnection(endpoint, http_auth)
+        self.es_executor = ESConnection(endpoint, http_auth, self.use_aws_authentication)
         self.es_executor.set_connection()
 
     def _get_literals(self):
@@ -173,7 +174,7 @@ def _get_literals(self):
 
         :return: a dict that is parsed from esliterals.json
         """
-        from escli.esliterals import __file__ as package_root
+        from odfesql_cli.esliterals import __file__ as package_root
 
         package_root = os.path.dirname(package_root)
 

escli/esliterals/__init__.py odfesql_cli/esliterals/__init__.py

@@ -30,7 +30,7 @@
 
 _version_re = re.compile(r"__version__\s+=\s+(.*)")
 
-with open("escli/__init__.py", "rb") as f:
+with open("odfesql_cli/__init__.py", "rb") as f:
     version = str(
         ast.literal_eval(_version_re.search(f.read().decode("utf-8")).group(1))
     )
@@ -48,12 +48,12 @@
     license="Apache 2.0",
     url="https://opendistro.github.io/for-elasticsearch-docs/",
     packages=find_packages(),
-    package_data={"escli": ["conf/esclirc", "esliterals/esliterals.json"]},
+    package_data={"odfesql_cli": ["conf/clirc", "esliterals/esliterals.json"]},
     description=description,
     long_description=long_description,
     long_description_content_type="text/markdown",
     install_requires=install_requirements,
-    entry_points={"console_scripts": ["odfesql=escli.main:cli"]},
+    entry_points={"console_scripts": ["odfesql=odfesql_cli.main:cli"]},
     classifiers=[
         "Intended Audience :: Developers",
         "License :: OSI Approved :: Apache Software License",
@@ -72,4 +72,5 @@
         "Topic :: Software Development",
         "Topic :: Software Development :: Libraries :: Python Modules",
     ],
+    python_requires='>=3.0'
 )

escli/esliterals/esliterals.json odfesql_cli/esliterals/esliterals.json

@@ -34,10 +34,10 @@ def connection():
 
 @pytest.fixture(scope="function")
 def default_config_location():
-    from escli.conf import __file__ as package_root
+    from odfesql_cli.conf import __file__ as package_root
 
     package_root = os.path.dirname(package_root)
-    default_config = os.path.join(package_root, "esclirc")
+    default_config = os.path.join(package_root, "clirc")
 
     yield default_config
 

escli/esstyle.py odfesql_cli/esstyle.py

@@ -16,7 +16,7 @@
 import stat
 import pytest
 
-from escli.config import ensure_dir_exists
+from odfesql_cli.config import ensure_dir_exists
 
 
 class TestConfig: