Skip to content

Commit d79af6d

Browse files
ajcwebdevajcwebdev
authored
Merge pull request #152 from ajcwebdev/staging
Consolidate CLI Tests, Add API Key Fields to Frontend
2 parents 13a9d62 + 5909fa3 commit d79af6d

35 files changed

+505
-998
lines changed

Diff for: ‎.gitignore

+2
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,8 @@ embeddings.json
1616
new-llm.md
1717
new-llm-*.md
1818
new-llm*.md
19+
test.md
20+
spec.log
1921

2022
setup-*.log
2123
setup.log

Diff for: ‎README.md

+6-5
Original file line numberDiff line numberDiff line change
@@ -93,13 +93,13 @@ npm run as -- --playlist "https://www.youtube.com/playlist?list=PLCVnrVv4KhXPz0S
9393
Run on a list of arbitrary URLs.
9494

9595
```bash
96-
npm run as -- --urls "content/example-urls.md"
96+
npm run as -- --urls "content/examples/example-urls.md"
9797
```
9898

9999
Run on a local audio or video file.
100100

101101
```bash
102-
npm run as -- --file "content/audio.mp3"
102+
npm run as -- --file "content/examples/audio.mp3"
103103
```
104104

105105
Run on a podcast RSS feed.
@@ -117,9 +117,10 @@ npm run as -- --video "https://www.youtube.com/watch?v=MORMZXEaONk" --ollama
117117
Use 3rd party LLM providers.
118118

119119
```bash
120-
npm run as -- --video "https://www.youtube.com/watch?v=MORMZXEaONk" --chatgpt gpt-4o-mini
121-
npm run as -- --video "https://www.youtube.com/watch?v=MORMZXEaONk" --claude CLAUDE_3_5_SONNET
122-
npm run as -- --video "https://www.youtube.com/watch?v=MORMZXEaONk" --gemini GEMINI_1_5_PRO
120+
npm run as -- --video "https://www.youtube.com/watch?v=MORMZXEaONk" --chatgpt
121+
npm run as -- --video "https://www.youtube.com/watch?v=MORMZXEaONk" --claude
122+
npm run as -- --video "https://www.youtube.com/watch?v=MORMZXEaONk" --gemini
123+
npm run as -- --video "https://www.youtube.com/watch?v=MORMZXEaONk" --deepseek
123124
npm run as -- --video "https://www.youtube.com/watch?v=MORMZXEaONk" --fireworks
124125
npm run as -- --video "https://www.youtube.com/watch?v=MORMZXEaONk" --together
125126
```

Diff for: ‎content/examples/.gitkeep

Whitespace-only changes.

Diff for: ‎content/audio.mp3 renamed to ‎content/examples/audio.mp3

File renamed without changes.

Diff for: ‎content/custom-prompt.md renamed to ‎content/examples/custom-prompt.md

File renamed without changes.

Diff for: ‎content/example-rss-feeds.md renamed to ‎content/examples/example-rss-feeds.md

File renamed without changes.

Diff for: ‎content/example-urls.md renamed to ‎content/examples/example-urls.md

File renamed without changes.

Diff for: ‎docs/cli/01-content-and-feed-inputs.md

+4-4
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ npm run as -- \
2222
Run on `audio.mp3` on the `content` directory:
2323

2424
```bash
25-
npm run as -- --file "content/audio.mp3"
25+
npm run as -- --file "content/examples/audio.mp3"
2626
```
2727

2828
## video - Process Single Video URLs
@@ -38,13 +38,13 @@ npm run as -- --video "https://www.youtube.com/watch?v=MORMZXEaONk"
3838
Run on an arbitrary list of URLs in `example-urls.md`.
3939

4040
```bash
41-
npm run as -- --urls "content/example-urls.md"
41+
npm run as -- --urls "content/examples/example-urls.md"
4242
```
4343

4444
Run on URLs file and generate JSON info file with markdown metadata of each video:
4545

4646
```bash
47-
npm run as -- --info --urls "content/example-urls.md"
47+
npm run as -- --info --urls "content/examples/example-urls.md"
4848
```
4949

5050
## playlist - Process Multiple Videos in YouTube Playlist
@@ -143,7 +143,7 @@ npm run as -- --rss "https://ajcwebdev.substack.com/feed" "https://feeds.transis
143143
Process multiple RSS feeds listed in a `.md` file:
144144

145145
```bash
146-
npm run as -- --rss "content/example-rss-feeds.md" --last 2
146+
npm run as -- --rss "content/examples/example-rss-feeds.md" --last 2
147147
```
148148

149149
Download episodes from a specific number of previous days, for example to download episodes from the last 7 days:

Diff for: ‎docs/cli/04-prompt-options.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -111,5 +111,5 @@ npm run as -- --printPrompt summary longChapters
111111
## Write a Custom Prompt
112112

113113
```bash
114-
npm run as -- --file "content/audio.mp3" --customPrompt "content/custom-prompt.md" --chatgpt
114+
npm run as -- --file "content/examples/audio.mp3" --customPrompt "content/examples/custom-prompt.md" --chatgpt
115115
```

Diff for: ‎docs/cli/07-test-suite.md

+2-53
Original file line numberDiff line numberDiff line change
@@ -1,57 +1,6 @@
11
# CLI Test Suite
22

3-
## Local Tests
4-
5-
Local services test, only uses Whisper for transcription and Ollama for LLM operations.
6-
7-
```bash
8-
npm run cli-local-test
9-
```
10-
11-
## Third Party Tests
12-
13-
Test all process commands and third party services.
14-
15-
- You'll need API keys for all services to make it through this entire command.
16-
- Mostly uses transcripts of videos around one minute long and cheaper models when possible, so the total cost of running this for any given service should be at most only a few cents.
17-
18-
```bash
19-
npm run cli-all-test
20-
```
21-
22-
Integrated LLM and transcription model test.
23-
24-
```bash
25-
npm run cli-models-test
26-
```
27-
28-
## Prompt Tests
29-
30-
Test all prompts on a single video or audio file.
31-
32-
```bash
33-
npm run cli-prompts-test
34-
```
35-
36-
## LLM Tests
37-
38-
Test all available models for a certain LLM service.
39-
40-
```bash
41-
npx tsx --test test/models/chatgpt.test.ts
42-
npx tsx --test test/models/claude.test.ts
43-
npx tsx --test test/models/deepseek.test.ts
44-
npx tsx --test test/models/gemini.test.ts
45-
npx tsx --test test/models/fireworks.test.ts
46-
npx tsx --test test/models/together.test.ts
47-
npx tsx --test test/models/ollama.test.ts
48-
```
49-
50-
## Transcription Tests
51-
52-
Test all available models for a certain transcription service.
53-
543
```bash
55-
npx tsx --test test/models/deepgram.test.ts
56-
npx tsx --test test/models/assembly.test.ts
4+
npm run test:cli
5+
npm run test:models
576
```

Diff for: ‎docs/server/03-llm-options.md

-9
Original file line numberDiff line numberDiff line change
@@ -59,15 +59,6 @@ curl --json '{
5959
}' http://localhost:3000/api/process
6060
```
6161

62-
```bash
63-
curl --json '{
64-
"type": "video",
65-
"url": "https://www.youtube.com/watch?v=MORMZXEaONk",
66-
"llm": "claude",
67-
"llmModel": "CLAUDE_3_SONNET"
68-
}' http://localhost:3000/api/process
69-
```
70-
7162
## Gemini
7263

7364
```bash

Diff for: ‎docs/server/04-prompt-options.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -260,6 +260,6 @@ curl --json '{
260260
"type": "audio",
261261
"file": "content/audio.mp3",
262262
"prompts": [],
263-
"customPrompt": "content/custom-prompt.md"
263+
"customPrompt": "content/examples/custom-prompt.md"
264264
}' http://localhost:3000/api/process
265265
```

Diff for: ‎docs/server/09-dash.md

+6-11
Original file line numberDiff line numberDiff line change
@@ -1,27 +1,22 @@
11
# Dash
22

3-
```bash
4-
curl --json '{
5-
"type": "video",
6-
"url": "https://www.youtube.com/watch?v=MORMZXEaONk",
7-
"walletAddress": "yVtCv413ByXiRxkixsj4M2LR6FyrJzVYzL",
8-
"mnemonic": "exile slab craft fade august tape length various borrow taxi bulb abuse"
9-
}' http://localhost:3000/api/process
10-
```
3+
## Create Data Contract
114

125
```bash
136
npm run create-data-contract
147
```
158

9+
## Save Content on Dash Platform
10+
1611
```bash
1712
curl --json '{
1813
"type": "video",
1914
"url": "https://www.youtube.com/watch?v=MORMZXEaONk",
2015
"llm": "chatgpt",
2116
"openaiApiKey": "",
22-
"walletAddress": "",
23-
"mnemonic": "",
24-
"identityId": "",
17+
"walletAddress": "yQHygFk4px2zxtvHk33o5YCySUWjZNqdPh",
18+
"mnemonic": "coil evidence seed guide craft thrive kangaroo height goat pilot bless visa",
19+
"identityId": "5jM4nYqQeBQ8t1hgtF8yLUhXqMiwJUcb3Yibhaqatukb",
2520
"contractId": "9c9bYUsGoRrzq9g19Vw7QDRE6wkoZy9J1r4LKrCUeJqD"
2621
}' \
2722
"http://localhost:3000/api/process"

Diff for: ‎package.json

+3-4
Original file line numberDiff line numberDiff line change
@@ -20,10 +20,9 @@
2020
"setup": "bash .github/setup/setup.sh",
2121
"check": "npx tsc",
2222
"as": "npm run tsx:base -- src/commander.ts",
23-
"cli-local-test": "npm run tsx:base -- test/cli/cli-local.test.ts",
24-
"cli-all-test": "npm run tsx:base -- test/cli/cli-all.test.ts",
25-
"cli-prompts-test": "npm run tsx:base -- test/cli/cli-prompts.test.ts",
26-
"cli-models-test": "npm run tsx:base -- test/cli/cli-models.test.ts",
23+
"test:base": "tsx --test --env-file=.env --test-reporter=spec --test-reporter-destination=spec.log --test-concurrency=1",
24+
"test:cli": "npm run test:base -- test/cli/*.test.ts",
25+
"test:models": "npm run test:base -- test/models/*.test.ts",
2726
"serve": "npm run tsx:base -- --watch src/fastify.ts",
2827
"server-local-test": "npm run tsx:base -- test/server/server-local.ts",
2928
"server-all-test": "npm run tsx:base -- test/server/server-all.ts",

Diff for: ‎test/cli/cli-all.test.ts

+40-90
Original file line numberDiff line numberDiff line change
@@ -8,129 +8,79 @@ import { join } from 'node:path'
88

99
const commands = [
1010
{
11-
// Process single local audio file.
12-
cmd: 'npm run as -- --file "content/audio.mp3"',
11+
cmd: 'npm run as -- --file "content/examples/audio.mp3" --prompt titles summary',
1312
expectedFile: 'audio-prompt.md',
14-
newName: '01-file-default.md'
13+
newName: '02-ALL-01-file-multiple-prompts.md'
1514
},
1615
{
17-
// Process local audio file with Whisper 'tiny' model.
18-
cmd: 'npm run as -- --file "content/audio.mp3" --whisper tiny',
19-
expectedFile: 'audio-prompt.md',
20-
newName: '02-file-whisper-tiny.md'
21-
},
22-
{
23-
// Process local audio file with title prompts, Whisper 'tiny' model, and Ollama.
24-
cmd: 'npm run as -- --file "content/audio.mp3" --prompt titles --whisper tiny --ollama',
25-
expectedFile: 'audio-ollama-shownotes.md',
26-
newName: '03-file-titles-prompt-whisper-tiny-ollama-shownotes.md'
27-
},
28-
{
29-
// Process local audio file with all available prompt options (except smallChapters and longChapters)
30-
cmd: 'npm run as -- --file "content/audio.mp3" --prompt titles summary',
31-
expectedFile: 'audio-prompt.md',
32-
newName: '04-file-prompts.md'
33-
},
34-
{
35-
// Process single YouTube video using AutoShow's default settings.
36-
cmd: 'npm run as -- --video "https://www.youtube.com/watch?v=MORMZXEaONk"',
37-
expectedFile: '2024-09-24-ep0-fsjam-podcast-prompt.md',
38-
newName: '05-video-default.md'
39-
},
40-
{
41-
// Process all videos in a specified YouTube playlist.
42-
cmd: 'npm run as -- --playlist "https://www.youtube.com/playlist?list=PLCVnrVv4KhXPz0SoAVu8Rc1emAdGPbSbr"',
43-
expectedFiles: [
44-
{ file: '2024-09-24-ep1-fsjam-podcast-prompt.md', newName: '06-playlist-default.md' },
45-
{ file: '2024-09-24-ep0-fsjam-podcast-prompt.md', newName: '07-playlist-default.md' }
46-
]
47-
},
48-
{
49-
// Process multiple YouTube videos from URLs listed in a file.
50-
cmd: 'npm run as -- --urls "content/example-urls.md"',
51-
expectedFiles: [
52-
{ file: '2024-09-24-ep1-fsjam-podcast-prompt.md', newName: '08-urls-default.md' },
53-
{ file: '2024-09-24-ep0-fsjam-podcast-prompt.md', newName: '09-urls-default.md' }
54-
]
55-
},
56-
{
57-
// Process podcast RSS feed from default order.
58-
cmd: 'npm run as -- --rss "https://ajcwebdev.substack.com/feed" --whisper tiny',
59-
expectedFile: '2021-05-10-thoughts-on-lambda-school-layoffs-prompt.md',
60-
newName: '10-rss-whisper-tiny.md'
61-
},
62-
{
63-
// Download JSON file with metadata for each item in the RSS feed.
64-
cmd: 'npm run as -- --rss "https://ajcwebdev.substack.com/feed" --info',
65-
expectedFile: 'ajcwebdev_info.json',
66-
newName: '11-ajcwebdev-rss-info.json',
67-
},
68-
{
69-
// process file using ChatGPT for LLM operations.
70-
cmd: 'npm run as -- --file "content/audio.mp3" --chatgpt',
16+
cmd: 'npm run as -- --file "content/examples/audio.mp3" --chatgpt',
7117
expectedFile: 'audio-chatgpt-shownotes.md',
72-
newName: '12-chatgpt-default.md'
18+
newName: '02-ALL-02-chatgpt-default.md'
7319
},
7420
{
75-
// process file using Claude for LLM operations.
76-
cmd: 'npm run as -- --file "content/audio.mp3" --claude',
21+
cmd: 'npm run as -- --file "content/examples/audio.mp3" --claude',
7722
expectedFile: 'audio-claude-shownotes.md',
78-
newName: '13-claude-default.md'
23+
newName: '02-ALL-03-claude-default.md'
7924
},
8025
{
81-
// process file using Gemini for LLM operations.
82-
cmd: 'npm run as -- --file "content/audio.mp3" --gemini',
26+
cmd: 'npm run as -- --file "content/examples/audio.mp3" --gemini',
8327
expectedFile: 'audio-gemini-shownotes.md',
84-
newName: '14-gemini-shownotes.md'
28+
newName: '02-ALL-04-gemini-shownotes.md'
8529
},
8630
{
87-
// process file using DeepSeek for LLM operations
88-
cmd: 'npm run as -- --file "content/audio.mp3" --deepseek',
31+
cmd: 'npm run as -- --file "content/examples/audio.mp3" --deepseek',
8932
expectedFile: 'audio-deepseek-shownotes.md',
90-
newName: '15-deepseek-shownotes.md'
33+
newName: '02-ALL-05-deepseek-shownotes.md'
9134
},
9235
{
93-
// process file using Fireworks for LLM operations
94-
cmd: 'npm run as -- --file "content/audio.mp3" --fireworks',
36+
cmd: 'npm run as -- --file "content/examples/audio.mp3" --fireworks',
9537
expectedFile: 'audio-fireworks-shownotes.md',
96-
newName: '16-fireworks-shownotes.md'
38+
newName: '02-ALL-06-fireworks-shownotes.md'
9739
},
9840
{
99-
// process file using Together for LLM operations
100-
cmd: 'npm run as -- --file "content/audio.mp3" --together',
41+
cmd: 'npm run as -- --file "content/examples/audio.mp3" --together',
10142
expectedFile: 'audio-together-shownotes.md',
102-
newName: '17-together-shownotes.md'
43+
newName: '02-ALL-07-together-shownotes.md'
10344
},
10445
{
105-
// process file using Deepgram for transcription
106-
cmd: 'npm run as -- --file "content/audio.mp3" --deepgram',
46+
cmd: 'npm run as -- --file "content/examples/audio.mp3" --deepgram',
10747
expectedFile: 'audio-prompt.md',
108-
newName: '18-deepgram-prompt.md'
48+
newName: '02-ALL-08-deepgram-prompt.md'
10949
},
11050
{
111-
// Process file using Deepgram and ChatGPT.
112-
cmd: 'npm run as -- --file "content/audio.mp3" --deepgram --chatgpt',
113-
expectedFile: 'audio-chatgpt-shownotes.md',
114-
newName: '19-deepgram-chatgpt-shownotes.md'
51+
cmd: 'npm run as -- --file "content/examples/audio.mp3" --assembly',
52+
expectedFile: 'audio-prompt.md',
53+
newName: '02-ALL-09-assembly-prompt.md'
11554
},
11655
{
117-
// process file using AssemblyAI for transcription
118-
cmd: 'npm run as -- --file "content/audio.mp3" --assembly',
119-
expectedFile: 'audio-prompt.md',
120-
newName: '20-assembly-prompt.md'
56+
cmd: 'npm run as -- --playlist "https://www.youtube.com/playlist?list=PLCVnrVv4KhXPz0SoAVu8Rc1emAdGPbSbr" --whisper tiny',
57+
expectedFiles: [
58+
{ file: '2024-09-24-ep1-fsjam-podcast-prompt.md', newName: '02-ALL-10-playlist-default.md' },
59+
{ file: '2024-09-24-ep0-fsjam-podcast-prompt.md', newName: '02-ALL-11-playlist-default.md' }
60+
]
12161
},
12262
{
123-
// Process file using AssemblyAI and ChatGPT.
124-
cmd: 'npm run as -- --file "content/audio.mp3" --assembly --chatgpt',
125-
expectedFile: 'audio-chatgpt-shownotes.md',
126-
newName: '21-assembly-chatgpt-shownotes.md'
127-
}
63+
cmd: 'npm run as -- --urls "content/examples/example-urls.md" --whisper tiny',
64+
expectedFiles: [
65+
{ file: '2024-09-24-ep1-fsjam-podcast-prompt.md', newName: '02-ALL-12-urls-whisper-tiny.md' },
66+
{ file: '2024-09-24-ep0-fsjam-podcast-prompt.md', newName: '02-ALL-13-urls-whisper-tiny.md' }
67+
]
68+
},
69+
{
70+
cmd: 'npm run as -- --playlist "https://www.youtube.com/playlist?list=PLCVnrVv4KhXPz0SoAVu8Rc1emAdGPbSbr" --info',
71+
expectedFile: 'fsjam_info.json',
72+
newName: '02-ALL-14-fsjam-info.json',
73+
},
74+
{
75+
cmd: 'npm run as -- --urls "content/examples/example-urls.md" --info',
76+
expectedFile: 'urls_info.json',
77+
newName: '02-ALL-15-urls-info.json',
78+
},
12879
]
12980

13081
test('AutoShow Command Tests', async (t) => {
13182
for (const [index, command] of commands.entries()) {
13283
await t.test(`should run command ${index + 1} successfully`, async () => {
133-
// Run the command
13484
execSync(command.cmd, { stdio: 'inherit' })
13585

13686
if (Array.isArray(command.expectedFiles)) {

0 commit comments

Comments
 (0)