Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Shorten ReadMe #237

Open
wants to merge 4 commits into
base: gh/PaliC/78/base
Choose a base branch
from
Open

Shorten ReadMe #237

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
0542033
Shorten ReadMe
PaliC Oct 27, 2022
66f7048
Update on "Shorten ReadMe"
PaliC Oct 31, 2022
90a8066
Update on "Shorten ReadMe"
PaliC Oct 31, 2022
c91ce75
Update on "Shorten ReadMe"
PaliC Oct 31, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
169 changes: 5 additions & 164 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@ Requirements:

> ℹ️ `torch::deploy` is ready for use in production environments, but is in Beta and may have some rough edges that we're continuously working on improving. We're always interested in hearing feedback and usecases that you might have. Feel free to reach out!

## The Easy Path to Installation

## Installation

### Building via Docker
Expand Down Expand Up @@ -183,170 +185,9 @@ cd build
./test_deploy
```

## Examples

See the [examples directory](./examples) for complete examples.

### Packaging a model `for multipy::runtime`

``multipy::runtime`` can load and run Python models that are packaged with
``torch.package``. You can learn more about ``torch.package`` in the ``torch.package`` [documentation](https://pytorch.org/docs/stable/package.html#tutorials).

For now, let's create a simple model that we can load and run in ``multipy::runtime``.

```python
from torch.package import PackageExporter
import torchvision

# Instantiate some model
model = torchvision.models.resnet.resnet18()

# Package and export it.
with PackageExporter("my_package.pt") as e:
e.intern("torchvision.**")
e.extern("numpy.**")
e.extern("sys")
e.extern("PIL.*")
e.extern("typing_extensions")
e.save_pickle("model", "model.pkl", model)
```

Note that since "numpy", "sys", "PIL" were marked as "extern", `torch.package` will
look for these dependencies on the system that loads this package. They will not be packaged
with the model.

Now, there should be a file named ``my_package.pt`` in your working directory.

<br>

### Load the model in C++
```cpp
#include <multipy/runtime/deploy.h>
#include <multipy/runtime/path_environment.h>
#include <torch/script.h>
#include <torch/torch.h>

#include <iostream>
#include <memory>

int main(int argc, const char* argv[]) {
if (argc != 2) {
std::cerr << "usage: example-app <path-to-exported-script-module>\n";
return -1;
}

// Start an interpreter manager governing 4 embedded interpreters.
std::shared_ptr<multipy::runtime::Environment> env =
std::make_shared<multipy::runtime::PathEnvironment>(
std::getenv("PATH_TO_EXTERN_PYTHON_PACKAGES") // Ensure to set this environment variable (e.g. /home/user/anaconda3/envs/multipy-example/lib/python3.8/site-packages)
);
multipy::runtime::InterpreterManager manager(4, env);

try {
// Load the model from the multipy.package.
multipy::runtime::Package package = manager.loadPackage(argv[1]);
multipy::runtime::ReplicatedObj model = package.loadPickle("model", "model.pkl");
} catch (const c10::Error& e) {
std::cerr << "error loading the model\n";
std::cerr << e.msg();
return -1;
}

std::cout << "ok\n";
}

```

This small program introduces many of the core concepts of ``multipy::runtime``.

An ``InterpreterManager`` abstracts over a collection of independent Python
interpreters, allowing you to load balance across them when running your code.

``PathEnvironment`` enables you to specify the location of Python
packages on your system which are external, but necessary, for your model.

Using the ``InterpreterManager::loadPackage`` method, you can load a
``multipy.package`` from disk and make it available to all interpreters.

``Package::loadPickle`` allows you to retrieve specific Python objects
from the package, like the ResNet model we saved earlier.

Finally, the model itself is a ``ReplicatedObj``. This is an abstract handle to
an object that is replicated across multiple interpreters. When you interact
with a ``ReplicatedObj`` (for example, by calling ``forward``), it will select
an free interpreter to execute that interaction.

<br>

### Build and execute the C++ example

Assuming the above C++ program was stored in a file called, `example-app.cpp`, a
minimal `CMakeLists.txt` file would look like:

```cmake
cmake_minimum_required(VERSION 3.12 FATAL_ERROR)
project(multipy_tutorial)

set(MULTIPY_PATH ".." CACHE PATH "The repo where multipy is built or the PYTHONPATH")

# include the multipy utils to help link against
include(${MULTIPY_PATH}/multipy/runtime/utils.cmake)

# add headers from multipy
include_directories(${MULTIPY_PATH})

# link the multipy prebuilt binary
add_library(multipy_internal STATIC IMPORTED)
set_target_properties(multipy_internal
PROPERTIES
IMPORTED_LOCATION
${MULTIPY_PATH}/multipy/runtime/build/libtorch_deploy.a)
caffe2_interface_library(multipy_internal multipy)

add_executable(example-app example-app.cpp)
target_link_libraries(example-app PUBLIC "-Wl,--no-as-needed -rdynamic" dl pthread util multipy c10 torch_cpu)
```

Currently, it is necessary to build ``multipy::runtime`` as a static library.
In order to correctly link to a static library, the utility ``caffe2_interface_library``
is used to appropriately set and unset ``--whole-archive`` flag.

Furthermore, the ``-rdynamic`` flag is needed when linking to the executable
to ensure that symbols are exported to the dynamic table, making them accessible
to the deploy interpreters (which are dynamically loaded).

**Updating LIBRARY_PATH and LD_LIBRARY_PATH**

In order to locate dependencies provided by PyTorch (e.g. `libshm`), we need to update the `LIBRARY_PATH` and `LD_LIBRARY_PATH` environment variables to include the path to PyTorch's C++ libraries. If you installed PyTorch using pip or conda, this path is usually in the site-packages. An example of this is provided below.

```bash
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/home/user/anaconda3/envs/multipy-example/lib/python3.8/site-packages/torch/lib"
export LIBRARY_PATH="$LIBRARY_PATH:/home/user/anaconda3/envs/multipy-example/lib/python3.8/site-packages/torch/lib"
```

The last step is configuring and building the project. Assuming that our code
directory is laid out like this:

```
example-app/
CMakeLists.txt
example-app.cpp
```


We can now run the following commands to build the application from within the
``example-app/`` folder:

```bash
cmake -S . -B build -DMULTIPY_PATH="/home/user/repos/multipy" # the parent directory of multipy (i.e. the git repo)
cmake --build build --config Release -j
```

Now we can run our app:

```bash
./example-app /path/to/my_package.pt
```
## Getting Started with `torch::deploy`
Once you have `torch::deploy` built, check out our [tutorials](https://pytorch.org/multipy/latest/tutorials/tutorial_root.html) and
[API documentation](https://pytorch.org/multipy/latest/api/library_root.html).

## Contributing

Expand Down
7 changes: 3 additions & 4 deletions docs/source/index.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,10 @@
``torch::deploy`` [Beta]
=====================

``torch::deploy`` is a system that allows you to load multiple python interpreters which execute PyTorch models, and run them in a single C++ process. Effectively, it allows people to multithread their pytorch models.
For more information on how torch::deploy works please see the related `arXiv paper <https://arxiv.org/pdf/2104.00254.pdf>`_. We plan to further generalize ``torch::deploy`` into a more generic system, ``multipy::runtime``,
which is more suitable for arbitrary python programs rather than just pytorch applications.
``torch::deploy`` (MultiPy for non-PyTorch use cases) is a C++ library that enables you to run eager mode PyTorch models in production without any modifications to your model to support tracing. ``torch::deploy`` provides a way to run using multiple independent Python interpreters in a single process without a shared global interpreter lock (GIL).
For more information on how ``torch::deploy`` works please see the related `arXiv paper <https://arxiv.org/pdf/2104.00254.pdf>`_.

The most up to date installation instructions for ``torch::deploy`` can be found in our `README <https://github.com/pytorch/multipy>`__.

Documentation
---------------
Expand All @@ -15,7 +15,6 @@ Documentation
:maxdepth: 2
:caption: Usage

setup.md
tutorials/tutorial_root
api/library_root

Expand Down
186 changes: 0 additions & 186 deletions docs/source/setup.rst
View file
Open in desktop

This file was deleted.