EOEPCA Reference Implementation - System
Explore the docs »
View Demo
·
Report Bug
·
Request Feature
- Table of Contents
- About The Project
- Getting Started
- System Documentation
- Building Blocks
- Implemented Standards
- Releases
- Issues
- License
- Contact
- Acknowledgements
EO Exploitation Platform Common Architecture (EOEPCA)
Earth Observation (EO) data has quickly evolved into an indispensable resource, directly facilitating solutions for society's most pressing challenges. This intensifying influx of data, oftentimes distributed across multiple independent platforms, presents a significant challenge for end-users in efficiently accessing and collaborating on critical geospatial tasks. Nevertheless, these platforms are more commonly collocated with cloud computing resources and applications such that users are now able to perform geospatial analysis tasks remotely. Working in the cloud bypasses traditional download, storage and performance limitations, however the distributed nature of these platform networks introduces complexities in the free and collective access to this remote geospatial data.
Our vision with EOEPCA then is for greater interoperability between such platforms, towards an open network of resources, whilst enabling current and future users to easily collaborate on geospatial analysis tasks at source. To this end we are helping to establish a consensus of best practice for EO Exploitation Platforms, based on open standards. Supporting that, we are developing a reference implementation of building blocks, as free open source software.
The goal of the EOEPCA's “Common Architecture” is therefore to define and agree the technical interfaces for the future exploitation of Earth Observation data in a distributed environment. The Common Architecture will provide the interfaces to facilitate the federation of different EO resources into a “Network of EO Resources”. The “Common Architecture” will be defined using open interfaces that link the different Resource Servers (building blocks) so that a user can efficiently access and consume the disparate services of the “Network of EO Resources”.
This repository represents the system integration of the building blocks that comprise the Reference Implementation of the Common Architecture.
The system is designed for deployment to cloud infrastructure orchestrated by a Kubernetes cluster. We include here the automation required to provision, deploy and test the emerging EOEPCA system.
The EOEPCA system deployment comprises several steps. Instructions are provided for both cloud deployment, and local deployment for development purposes.
For the latest release (v1.4) ensure that the correct version of this README is followed.
The first step is to fork this repository into your GitHub account. Use of fork (rather than clone) is recommended to support our GitOps approach to deployment with Flux Continuous Delivery, which requires write access to your git repository for deployment configurations.
Having forked, clone the repository to your local platform...
$ git clone [email protected]:<user>/eoepca.git
$ cd eoepca
$ git checkout v1.4
NOTE that this clones the specific tag that is well tested. The develop branch should alternatively be used for the latest development.
| Step | Cloud (OpenStack) | Local Developer |
|---|---|---|
| Infrastructure | CREODIAS | n/a (local developer platform) |
| Kubernetes Cluster | Rancher Kubernetes Engine | Minikube |
| EOEPCA System Deployment ( flux) |
EOEPCA GitOps | EOEPCA GitOps |
| EOEPCA System Deployment (Deployment Guide) |
Deployment Guide | Deployment Guide |
| Acceptance Test | Run Test Suite | Run Test Suite |
NOTE that, with release v1.4, the number of system components has been expanded to the point where it is more difficult to make a full system deployment in minikube, due to the required resource demands. Nevertheless, it is possible to make a minikube deployment to a single node with sufficient resources (8 cpu, 32GB) - as illustrated by the Deployment Guide.
NOTE also that the Deployment Guide provides a more detailed description of the deployment and configuration of the components, supported by some shell scripts that deploy the components directly using helm (rather than using flux GitOps). The Deployment Guide represents a more informative introduction, and the supporting scripts assume minikube out-of-the-box.
To ease development/testing, the EOEPCA deployment is configured to use host/service names that embed IP-addresses - which avoids the need to configure public nameservers, (as would be necessary for a production deployment). Our services are exposed through Kubernetes ingress rules that use name-based routing, and so simple IP-addresses are insufficient. Therefore, we exploit the services of nip.io that provides dynamic DNS in which the hostname->IP-adress mapping is embedded in the hostname.
Thus, we use host/service names of the form <service-name>.<public-ip>.nip.io, where the <public-ip> is the public-facing IP-address of the deployment (delimited with dashes -). For cloud deployment the public IP is that of the cloud load-balancer, or for minikube it is the minikube ip - for example workspace.192-168-49-2.nip.io.
NOTE: nip.io supports either dots . or dashes - to delimit the IP-address in the DNS name. We use dashes, which seem to work better with LetsEncrypt rate limits.
NOTES:
We also maintain deployments for development and test, under the domainsdevelop.eoepca.organddemo.eoepca.org.
Our public endpoint address is baked into our deployment configuration - in particular the Kubernetes Ingress resources. To re-use our deployment configuration these Ingress values must be updated to suit your deployment environment.
The Deployment Guide provides the best system-level introduction to the system.
The following is the original documentation that was produced at the start of the project.
It is included here for context...
The following provides links to the resources for each building block - including documentaiton and source repositories...
- ADES - ZOO-Project DRU (Application Deployment & Execution Service) - [repository] [docs]:
- Helm chart
- eoepca-proc-service-template - Cookie-cutter template for Application Package execution in Kubernetes
- zoo-calrissian-runner - Python library used by the
eoepca-proc-service-templateto aid orchestration of CWL application packages running in Kubernetes via Calrissian - pycalrissian - Python library used by
zoo-calrissian-runnerto aid interfacing with Calrissian and Kubernetes
- Application Hub - [repository] [docs]:
- ApplicationHub Applications:
- Processor Development Environment [repository]
- JupyterLab [repository]
- Remote Desktop [repository]
- Remote Desktop with QGIS [repository]
- Remote Desktop with SNAP [repository]
- Dashboard with Streamlit [repository]
- ApplicationHub Applications:
- Resource Catalogue - [repository] - [docs]
- Data Access - [repository] - [docs]
- Registration API - [repository] - [docs]
- Workspace API - [repository] - [docs]
- Minio Bucket API - repository - docs
- Identity & Access Management...
- Keycloak Solution (NEW)
- Gluu Solution (deprecated)
The EOEPCA project aligns with OGC standards and best practices including:
- OGC API: Coverages, Maps, Records, Features, Processing, Tiles
- OGC Web Services: WFS, WMTS, WCS, WMS, WPS
For full documentation on how EOEPCA implements these standards, visit the EOEPCA repository. For further details on the OGC standards visit the official API and Web Services pages.
EOEPCA system releases are made to provide integrated deployments of the developed building blocks. The release history is as follows:
| Date | Release |
|---|---|
| 28/02/2024 | Release 1.4 |
| 25/09/2023 | Release 1.3 |
| 20/12/2022 | Release 1.2 |
| 31/05/2022 | Release 1.1 |
| 24/12/2021 | Release 1.0 |
| 23/07/2021 | Release 0.9 |
| 10/03/2021 | Release 0.3 |
| 23/11/2020 | Release 0.2 |
| 13/08/2020 | Release 0.1.2 |
| 06/08/2020 | Release 0.1.1 |
| 22/06/2020 | Release 0.1 |
See the open issues for a list of proposed features (and known issues).
The EOEPCA SYSTEM is distributed under the OSI approved Apache 2.0 Licence. See LICENSE for more information.
Building-blocks and their sub-components are individually licensed. See their respective source repositories for details.
Project Link: Project Home (https://eoepca.org/)
- README.md is based on this template by Othneil Drew.