newslynx-core

NewsLynx Core is an expandable open-source toolkit for building modular content analytics workflows. It provides a fully RESTful API as well as a comprehensive python client and command line interface.

NewsLynx Core was built to power newslynx-app but is capable of powering a diverse range of potential applications, as well, including:

• A Mention.net-like pipeline for your personal or company blog.
• A Flexible timeseries store for content metrics which will automatically summarize and compare your data, as well as enable the additional of custom, computed metrics.
• A framework for configuring, scheduling, and monitoring arbitrary python jobs via API.
• A content-extraction API.

Installation

For most applications, refer to our installation guide. If you'd like to setup a development environment, following the instructions below for MacOS X. If you'd like to spin up a Virtual Machine, check out our automation guide.

Dependencies

Postgres

NOTE We recommend using Postgres APP. However, if you prefer the brew distribution, make sure to install it with plpythonu.

$ brew install postgresql --build-from-source --with-python

(Re)create a postgresql database

# If you already have a database called `newslynx`, delete it first
$ dropdb newslynx 
$ createdb newslynx

Redis

Install redis:

$ brew install redis

Open another tab in your shell and run:

$ redis-server

Installation / Initialization

NOTE we recommend that you install newslynx in a virtual environment.

First clone this repository and move into it's root directory

$ git clone https://github.com/newslynx/newslynx-core.git
$ cd newslynx-core

OPTIONAL Set your configuration. If you don't do this, we will fallback on the app's default configuration file.

Initialize the database, super user, and install default sous chefs, tags, and recipes.

Run this command when working with newslynx-app

$ make app_install

Initialize a bare install.

[Expert mode] don't install the app's default sous chefs, tags, or recipes.

$ make bare_install 

Start the server

• In debug mode: newslynx debug
• Debug mode with errors: newslynx debug --raise-errors
• Production gunicorn server: bin/run

Start the task workers

$ bin/start_workers

Stop the task workers

$ bin/stop_workers

Start the cron daemon

$ newslynx cron

Testing

newslynx is a complicated applcation, to simplify testing...

Makefile

This repository contains a Makefile which provides access to the following commands:

• make clean

  • clean out cruft generated by pip.

• make clean_sc

  • clean out cruft generated by pip in ~/.newslynx/sous-chefs

• make install

  • installs newslynx
  • will first uninstall newslynx if a version already exists.

• make app_install

  • Installs newslynx, initializes the Super User, and adds the Recipes, Tags, and Sous Chefs needed to run newslynx-app

• make bare_install

  • Installs newslynx and initializes the Super User

• make test_install

  • removes all files under ~/.newslynx
  • runs make app_install
  • runs pip install -r test-requirements.txt

• distribute

  • creates a new version of newslynx on PyPI

Docs

• Configuration
• First steps
• Understanding Sous Chefs
• Understanding Recipes
• Understanding Metrics
• Understanding Tags
• Understanding Content
• Understanding Events
• Full API Reference
• Command Line Interface
• Write your own Sous Chef

Application Structure

License

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.