Skip to content

Commit 123dbf4

Browse files
ulrichkrulrichkr
authored
feat: add jumpstart text generation default payload notebook (#4576)
* feat: add jumpstart text generation default payload notebook * chore: black * chore: address comment
1 parent ca517c6 commit 123dbf4

File tree

2 files changed

+263
-0
lines changed

2 files changed

+263
-0
lines changed

‎introduction_to_amazon_algorithms/jumpstart-foundation-models/README.md

+1
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@
44

55
These examples provide quick walkthroughs to get you up and running with Amazon SageMaker JumpStart Foundation Models.
66

7+
- [JumpStart Text Generation Inference](jumpstart-text-generation-inference.ipynb) provides a single notebook for all JumpStart text generation models using default payloads retrieved from the JumpStartModel.
78
- [Llama-2 Text Generation](llama-2-text-completion.ipynb) and [Llama-2 Chat Completion](llama-2-chat-completion.ipynb) demonstrate text generation using Meta's Llama-2 pretrained and fine-tuned chat models, respectfully.
89
- [Llama Guard Text Moderation](llama-guard-text-moderation.ipynb) demonstrates using a Llama Guard model to moderate a conversation in conjunction with a Llama-2 model.
910
- [Text2Text Generation Flan T5 UL2](text2text-generation-flan-t5-ul2.ipynb) demonstrates Text2Text generation using state-of-the-art pretrained Flan T5 models from [Hugging Face](https://huggingface.co/docs/transformers/model_doc/flan-t5) which take an input text containing the task and returns the output of the accomplished task.

‎introduction_to_amazon_algorithms/jumpstart-foundation-models/jumpstart-text-generation-inference.ipynb

+262
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,262 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"id": "e78dc97c",
6+
"metadata": {},
7+
"source": [
8+
"# Introduction to SageMaker JumpStart - Text Generation"
9+
]
10+
},
11+
{
12+
"cell_type": "markdown",
13+
"id": "b4e567a4",
14+
"metadata": {},
15+
"source": [
16+
"---\n",
17+
"\n",
18+
"This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.\n",
19+
"\n",
20+
"![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/introduction_to_amazon_algorithms|jumpstart-foundation-models|jumpstart-text-generation-inference.ipynb)\n",
21+
"\n",
22+
"---"
23+
]
24+
},
25+
{
26+
"cell_type": "markdown",
27+
"id": "2d4e36b9",
28+
"metadata": {},
29+
"source": [
30+
"In this demo notebook, we demonstrate how to use the SageMaker Python SDK to deploy a SageMaker JumpStart text generation model and invoke the endpoint."
31+
]
32+
},
33+
{
34+
"cell_type": "markdown",
35+
"id": "0af77efb",
36+
"metadata": {},
37+
"source": [
38+
"## Setup\n",
39+
"First, upgrade to the latest sagemaker SDK to ensure all available models are deployable."
40+
]
41+
},
42+
{
43+
"cell_type": "code",
44+
"execution_count": null,
45+
"id": "9b05b931-992e-4526-978d-f03196874a3b",
46+
"metadata": {},
47+
"outputs": [],
48+
"source": [
49+
"%pip install --quiet --upgrade sagemaker jmespath"
50+
]
51+
},
52+
{
53+
"cell_type": "markdown",
54+
"id": "5dbb6b35",
55+
"metadata": {},
56+
"source": [
57+
"Select the desired model to deploy. The provided dropdown filters all text generation models available in SageMaker JumpStart."
58+
]
59+
},
60+
{
61+
"cell_type": "code",
62+
"execution_count": null,
63+
"id": "f625a488",
64+
"metadata": {},
65+
"outputs": [],
66+
"source": [
67+
"from ipywidgets import Dropdown\n",
68+
"from sagemaker.jumpstart.notebook_utils import list_jumpstart_models\n",
69+
"\n",
70+
"\n",
71+
"dropdown = Dropdown(\n",
72+
" options=list_jumpstart_models(\"search_keywords includes Text Generation\"),\n",
73+
" value=\"huggingface-llm-mistral-7b-instruct\",\n",
74+
" description=\"Select a JumpStart text generation model:\",\n",
75+
" style={\"description_width\": \"initial\"},\n",
76+
" layout={\"width\": \"max-content\"},\n",
77+
")\n",
78+
"display(dropdown)"
79+
]
80+
},
81+
{
82+
"cell_type": "code",
83+
"execution_count": null,
84+
"id": "8a40df34",
85+
"metadata": {
86+
"jumpStartAlterations": [
87+
"modelIdOnly"
88+
],
89+
"tags": []
90+
},
91+
"outputs": [],
92+
"source": [
93+
"model_id = dropdown.value\n",
94+
"model_version = \"*\""
95+
]
96+
},
97+
{
98+
"cell_type": "markdown",
99+
"id": "3c0a0388",
100+
"metadata": {},
101+
"source": [
102+
"## Deploy model\n",
103+
"\n",
104+
"Create a `JumpStartModel` object, which initializes default model configurations conditioned on the selected instance type. JumpStart already sets a default instance type, but you can deploy the model on other instance types by passing `instance_type` to the `JumpStartModel` class."
105+
]
106+
},
107+
{
108+
"cell_type": "code",
109+
"execution_count": null,
110+
"id": "85a2a8e5-789f-4041-9927-221257126653",
111+
"metadata": {
112+
"tags": []
113+
},
114+
"outputs": [],
115+
"source": [
116+
"from sagemaker.jumpstart.model import JumpStartModel\n",
117+
"\n",
118+
"\n",
119+
"model = JumpStartModel(model_id=model_id, model_version=model_version)"
120+
]
121+
},
122+
{
123+
"cell_type": "markdown",
124+
"id": "1259ad4f",
125+
"metadata": {},
126+
"source": [
127+
"You can now deploy the model using SageMaker JumpStart. If the selected model is gated, you will need to accept the end-user license agreement (EULA) prior to deployment. This is accomplished by providing the `accept_eula=True` argument to the `deploy` method. The deployment might take few minutes. "
128+
]
129+
},
130+
{
131+
"cell_type": "code",
132+
"execution_count": null,
133+
"id": "5f3b42ed",
134+
"metadata": {},
135+
"outputs": [],
136+
"source": [
137+
"predictor = model.deploy()"
138+
]
139+
},
140+
{
141+
"cell_type": "markdown",
142+
"id": "f10cf06c",
143+
"metadata": {},
144+
"source": [
145+
"## Invoke the endpoint\n",
146+
"\n",
147+
"This section demonstrates how to invoke the endpoint using example payloads that are retrieved programmatically from the `JumpStartModel` object. You can replace these example payloads with your own payloads."
148+
]
149+
},
150+
{
151+
"cell_type": "code",
152+
"execution_count": null,
153+
"id": "cbb364d8",
154+
"metadata": {},
155+
"outputs": [],
156+
"source": [
157+
"example_payloads = model.retrieve_all_examples()"
158+
]
159+
},
160+
{
161+
"cell_type": "code",
162+
"execution_count": null,
163+
"id": "bf5899c8",
164+
"metadata": {},
165+
"outputs": [],
166+
"source": [
167+
"import jmespath\n",
168+
"\n",
169+
"\n",
170+
"for payload in example_payloads:\n",
171+
" response = predictor.predict(payload.body)\n",
172+
" generated_text = jmespath.search(payload.raw_payload[\"output_keys\"][\"generated_text\"], response)\n",
173+
" print(\"Input:\\n\", payload.body[payload.prompt_key])\n",
174+
" print(\"Output:\\n\", generated_text.strip())\n",
175+
" print(\"\\n===============\\n\")"
176+
]
177+
},
178+
{
179+
"cell_type": "markdown",
180+
"id": "f6a53e57",
181+
"metadata": {},
182+
"source": [
183+
"## Clean up the endpoint\n",
184+
"Don't forget to clean up resources when finished to avoid unnecessary charges."
185+
]
186+
},
187+
{
188+
"cell_type": "code",
189+
"execution_count": null,
190+
"id": "0e19a2bb",
191+
"metadata": {},
192+
"outputs": [],
193+
"source": [
194+
"predictor.delete_predictor()"
195+
]
196+
},
197+
{
198+
"cell_type": "markdown",
199+
"id": "c4998b86",
200+
"metadata": {},
201+
"source": [
202+
"## Notebook CI Test Results\n",
203+
"\n",
204+
"This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
205+
"\n",
206+
"\n",
207+
"![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|jumpstart-text-generation-inference.ipynb)\n",
208+
"\n",
209+
"![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/introduction_to_amazon_algorithms|jumpstart-foundation-models|jumpstart-text-generation-inference.ipynb)\n",
210+
"\n",
211+
"![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|jumpstart-text-generation-inference.ipynb)\n",
212+
"\n",
213+
"![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|jumpstart-text-generation-inference.ipynb)\n",
214+
"\n",
215+
"![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|jumpstart-text-generation-inference.ipynb)\n",
216+
"\n",
217+
"![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|jumpstart-text-generation-inference.ipynb)\n",
218+
"\n",
219+
"![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/introduction_to_amazon_algorithms|jumpstart-foundation-models|jumpstart-text-generation-inference.ipynb)\n",
220+
"\n",
221+
"![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/introduction_to_amazon_algorithms|jumpstart-foundation-models|jumpstart-text-generation-inference.ipynb)\n",
222+
"\n",
223+
"![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|jumpstart-text-generation-inference.ipynb)\n",
224+
"\n",
225+
"![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|jumpstart-text-generation-inference.ipynb)\n",
226+
"\n",
227+
"![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|jumpstart-text-generation-inference.ipynb)\n",
228+
"\n",
229+
"![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/introduction_to_amazon_algorithms|jumpstart-foundation-models|jumpstart-text-generation-inference.ipynb)\n",
230+
"\n",
231+
"![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|jumpstart-text-generation-inference.ipynb)\n",
232+
"\n",
233+
"![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/introduction_to_amazon_algorithms|jumpstart-foundation-models|jumpstart-text-generation-inference.ipynb)\n",
234+
"\n",
235+
"![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|jumpstart-text-generation-inference.ipynb)\n",
236+
"\n"
237+
]
238+
}
239+
],
240+
"metadata": {
241+
"instance_type": "ml.t3.medium",
242+
"kernelspec": {
243+
"display_name": "Python 3 (Data Science 2.0)",
244+
"language": "python",
245+
"name": "python3"
246+
},
247+
"language_info": {
248+
"codemirror_mode": {
249+
"name": "ipython",
250+
"version": 3
251+
},
252+
"file_extension": ".py",
253+
"mimetype": "text/x-python",
254+
"name": "python",
255+
"nbconvert_exporter": "python",
256+
"pygments_lexer": "ipython3",
257+
"version": "3.8.13"
258+
}
259+
},
260+
"nbformat": 4,
261+
"nbformat_minor": 5
262+
}

0 commit comments

Comments
 (0)