Load/calculate some data, fill out a simple python object, host the results!
As part of our music recommendation engine efforts at MetaBrainz, we're creating a ton of data sets that are in various states of being useful. In order to be able to evaluate this data and to start using it in our recommendation tools we needed a quick way to turn data into a hosted API, so this project was born.
Install the hoster, ideally in a virtual env:
pip install -e git+ssh://[email protected]/metabrainz/data-set-hoster.git#egg=datasethoster
Then run this program and go to http://localhost:8000 !
from datasethoster import Query
from datasethoster.main import create_app, register_query
class ExampleQuery(Query):
def names(self):
return ("example", "Useless arithmetic table example")
def introduction(self):
return """This is the introduction, which could provide more useful info that this introduction does."""
def inputs(self):
return ['number', 'num_lines']
def outputs(self):
return ['number', 'multiplied']
def fetch(self, args, offset=-1, limit=-1):
data = []
for arg in args:
for i in range(int(arg['num_lines'])):
data.append({ 'number': str(i),
'multiplied': str(i * int(arg['number']))
})
return data
register_query(ExampleQuery())
if __name__ == "__main__":
app = create_app()
app.run(host="0.0.0.0", port=8000, debug=True)The resulting web page should look something like this:
There is a fully functioning example in the example directory.
This project also provides two other endpoints other than the web endpoint:
The example above can be fetched using this URL:
http://localhost:8000/example/json?number=1&num_lines=2
and the JSON returned will be:
[
{
"multiplied": "0",
"number": "0"
},
{
"multiplied": "1",
"number": "1"
}
]
However, we do not recommend using this endpoint for more than testing, since the arguments for the GET call can make the URL quite long, which can give problems with proxies that don't support long enough URLs.
It is best to use the POST JSON endpoint.
The example above can be fetched by posting the following JSON
[
{
"num_lines": "2",
"number": "1"
}
]
to this URL:
http://localhost:8000/example/json
and the JSON returned will be:
[
{
"multiplied": "0",
"number": "0"
},
{
"multiplied": "1",
"number": "1"
}
]
The JSON POST endpoint (and only it) support pagination. You can add the count
and offset parameters to the URL to control the number and offset
of items returned:
http://localhost:8000/example/json?count=3&offset=2
To host your own data, install the libary as shown above. Then create an object derived from the Query object and override the following functions:
- names: Return a tuple of the slug (short name) and descrption of this data set. The name should be a simple identifier that provides the dataset a unique location. We recommend using nothing but alphanumeric characters and -. The description should give short a human readable description of what your data set does.
- introduction: Return a longer description of what the data set is and why your user might want to use it.
- inputs: Return a list of required input names. These should be identifiers as well with no spaces. If an input is enclosed in [] it denotes that a list is required as input to this endpoint. Otherwise the input will be a simple value.
- outputs: Return a list of outputs from this functions. These too should be identifier names with no spaces and if they are enclosed in [] it also denotes a list.
- fetch: This is the function where the work is carried out. Given the passed in parameters, the function should carry out more error checking on the arguments and then fetch the data needed. This function should return a list of dicts with keys named exactly after each of the outputs. Alternatively, return a tuple of (data, summary), and the summary will be displayed as HTML above the results. Use this to return additional information about the queries, such timing or debug information. For the POST method endpoint the arguments count and offset may be provided -- for the web view and GET JSON views, offset and count are not handled.
Once a query object has been defined, it needs to be registered by calling register_query, passing an instance of the object. Finally you'll need to run the app -- the example code shows how to run the app in the development mode. Please note: Do not use this mode for production -- instead use a proper UWSGI container.
If you use Docker, there is a Dockerfile provided in the example directory that will host this project correctly. See below for more details.
Use the Dockerfile in examples as a template and follow the instructions on how to adapt the template for your own use. Then build and start the container:
docker build -t metabrainz/datasethoster -f example/Dockerfile .
docker run -d -p 80:80 metabrainz/datasethoster
If you need to connect to another container, you should probably connect the new container to the existing network. To connect to an instance of musicbrainz-docker, using an all around more useful invocation for testing is:
docker run -it --rm --name datasethoster-test --network musicbrainzdocker_default -p 4200:80 metabrainz/datasethoster