Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TS SDK polish #41

Merged
merged 2 commits into from
Feb 28, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions 2_typescript/2_llm-prediction/completion.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,10 +62,10 @@ You can also print prediction metadata, such as the model used for generation, n
TypeScript:
language: typescript
code: |
console.info("Model used:", prediction.modelInfo.displayName);
console.info("Predicted tokens:", prediction.stats.predictedTokensCount);
console.info("Time to first token (seconds):", prediction.stats.timeToFirstTokenSec);
console.info("Stop reason:", prediction.stats.stopReason);
console.info("Model used:", completion.modelInfo.displayName);
console.info("Predicted tokens:", completion.stats.predictedTokensCount);
console.info("Time to first token (seconds):", completion.stats.timeToFirstTokenSec);
console.info("Stop reason:", completion.stats.stopReason);
```

## Example: Get an LLM to Simulate a Terminal
Expand Down
6 changes: 3 additions & 3 deletions 2_typescript/2_llm-prediction/image-input.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,15 +43,15 @@ Use the `client.files.prepareImage()` method to get a handle to the image you ca

```

If you only have the image in the form of a base64 string, you can use the `client.files.prepareImage64()` method instead.
If you only have the image in the form of a base64 string, you can use the `client.files.prepareImageBase64()` method instead.

```lms_code_snippet
variants:
Example:
language: typescript
code: |
const imageBase64 = "Your base64 string here";
const image = await client.files.prepareImage64(imageBase64);
const image = await client.files.prepareImageBase64(imageBase64);
```

We support JPEG, PNG, and WebP image formats.
Expand All @@ -65,7 +65,7 @@ Generate a prediction by passing the image to the model in the `.respond()` meth
Example:
language: typescript
code: |
const prediction = llm.respond([
const prediction = model.respond([
{ role: "user", content: "Describe this image please", images: [image] },
]);
```
8 changes: 6 additions & 2 deletions 2_typescript/2_llm-prediction/parameters.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,9 @@ The `.model()` retrieves a handle to a model that has already been loaded, or lo
const model = await client.llm.model("qwen2.5-7b-instruct", {
config: {
contextLength: 8192,
gpuOffload: 0.5,
gpu: {
ratio: 0.5,
},
},
});
```
Expand All @@ -70,7 +72,9 @@ The `.load()` method creates a new model instance and loads it with the specifie
const model = await client.llm.load("qwen2.5-7b-instruct", {
config: {
contextLength: 8192,
gpuOffload: 0.5,
gpu: {
ratio: 0.5,
},
},
});
```
Expand Down
10 changes: 1 addition & 9 deletions 2_typescript/2_llm-prediction/structured-response.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,15 +103,7 @@ const schema = {
const prediction = model.respond("Tell me about The Hobbit.", {
structured: {
type: "json",
jsonSchema: {
type: "object",
properties: {
title: { type: "string" },
author: { type: "string" },
year: { type: "integer" },
},
required: ["title", "author", "year"],
},
jsonSchema: schema,
},
});

Expand Down
12 changes: 6 additions & 6 deletions 2_typescript/2_llm-prediction/working-with-chats.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ title: Working with Chats
description: APIs for representing a chat conversation with an LLM
---

SDK methods such as `llm.respond()`, `llm.applyPromptTemplate()`, or `llm.act()`
SDK methods such as `model.respond()`, `model.applyPromptTemplate()`, or `model.act()`
takes in a chat parameter as an input. There are a few ways to represent a chat in the SDK.

## Option 1: Array of Messages
Expand All @@ -15,7 +15,7 @@ variants:
"Text-only":
language: typescript
code: |
const prediction = llm.respond([
const prediction = model.respond([
{ role: "system", content: "You are a resident AI philosopher." },
{ role: "user", content: "What is the meaning of life?" },
]);
Expand All @@ -24,7 +24,7 @@ variants:
code: |
const image = await client.files.prepareImage("/path/to/image.jpg");

const prediction = llm.respond([
const prediction = model.respond([
{ role: "system", content: "You are a state-of-art object recognition system." },
{ role: "user", content: "What is this object?", images: [image] },
]);
Expand All @@ -39,7 +39,7 @@ variants:
TypeScript:
language: typescript
code: |
const prediction = llm.respond("What is the meaning of life?");
const prediction = model.respond("What is the meaning of life?");
```

## Option 3: Using the `Chat` Helper Class
Expand All @@ -55,7 +55,7 @@ variants:
chat.append("system", "You are a resident AI philosopher.");
chat.append("user", "What is the meaning of life?");

const prediction = llm.respond(chat);
const prediction = model.respond(chat);
With Images:
language: typescript
code: |
Expand All @@ -65,7 +65,7 @@ variants:
chat.append("system", "You are a state-of-art object recognition system.");
chat.append("user", "What is this object?", { images: [image] });

const prediction = llm.respond(chat);
const prediction = model.respond(chat);
```

You can also quickly construct a `Chat` object using the `Chat.from` method.
Expand Down
8 changes: 4 additions & 4 deletions 2_typescript/4_tokenization/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,9 @@ You can tokenize a string with a loaded LLM or embedding model using the SDK. In
import { LMStudioClient } from "@lmstudio/sdk";

const client = new LMStudioClient();
const llm = await client.llm.model();
const model = await client.llm.model();

const tokens = await llm.tokenize("Hello, world!");
const tokens = await model.tokenize("Hello, world!");

console.info(tokens); // Array of token IDs.
```
Expand All @@ -34,7 +34,7 @@ If you only care about the number of tokens, you can use the `.countTokens` meth
TypeScript:
language: typescript
code: |
const tokenCount = await llm.countTokens("Hello, world!");
const tokenCount = await model.countTokens("Hello, world!");
console.info("Token count:", tokenCount);
```

Expand Down Expand Up @@ -72,7 +72,7 @@ You can determine if a given conversation fits into a model's context by doing t
// ... More messages
]);

console.info("Fits", await doesChatFitInContext(model, chat));
console.info("Fits in context:", await doesChatFitInContext(model, chat));
```

<!-- ### Context length comparisons
Expand Down
8 changes: 5 additions & 3 deletions 2_typescript/5_manage-models/loading.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,9 @@ Use `load()` to load a new instance of a model, even if one already exists. This
const client = new LMStudioClient();

const llama = await client.llm.load("llama-3.2-1b-instruct");
const another_llama = await client.llm.load("llama-3.2-1b-instruct", "second-llama");
const another_llama = await client.llm.load("llama-3.2-1b-instruct", {
identifier: "second-llama"
});
```

Learn more about the `.load()` method and the parameters it accepts in the [API Reference](../api-reference/load).
Expand All @@ -86,8 +88,8 @@ Once you no longer need a model, you can unload it by simply calling `unload()`

const client = new LMStudioClient();

const llm = await client.llm.model();
await llm.unload()
const model = await client.llm.model();
await model.unload();
```

## Set Custom Load Config Parameters
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ Context length is a special case that [has its own method](/docs/api/sdk/get-con
import { LMStudioClient } from "@lmstudio/sdk";

const client = new LMStudioClient();
const llm = await client.llm.model();
const model = await client.llm.model();

loadConfig = await llm.getLoadConfig()
loadConfig = await model.getLoadConfig()
```
2 changes: 1 addition & 1 deletion 2_typescript/6_model-info/get-context-length.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,5 +54,5 @@ You can determine if a given conversation fits into a model's context by doing t
// ... More messages
]);

console.info("Fits", await doesChatFitInContext(model, chat));
console.info("Fits in context:", await doesChatFitInContext(model, chat));
```
4 changes: 2 additions & 2 deletions 2_typescript/6_model-info/get-model-info.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ and the `path` used to [load it](/docs/api/sdk/load-model). In the below example
import { LMStudioClient } from "@lmstudio/sdk";

const client = new LMStudioClient();
const llm = await client.llm.model();
const model = await client.llm.model();

model_info = await llm.getInfo()
const model_info = await model.getInfo();
```