Add vision capability for bots

[gmuffiness]

I added support for image input using GPT-4V and GPT-4o, enabling effective image interpretation.
This is an initial implementation, so I would greatly appreciate any feedback or suggestions for improvement. Thank you!

Changelog

• Two actions added, leveraging mineflayer’s screenshot functionality (as @MaxRobinsonTheGreat suggested in this issue )
  • lookAtPlayer: Allows the bot to focus on the player’s direction or viewpoint for better understanding
  • lookAtPosition: Enables the bot to focus on specific coordinates for targeted image interpretation
• Added a promptImageConvo method in src/agent/prompter.js.
• Included examples to demonstrate these new features.

Known Limitations

• Node.js Compatibility : Using a relatively new Node.js version (in my case, v23.2.0) caused installation errors with node-canvas-webgl and three packages. Switching to the LTS version (18.20.5) resolved these issues. Use nvm use 18 for compatibility.
• Minecraft Version Support : Works reliably with Minecraft versions up to 1.20.1, as specified in the Prismarine Viewer README. Rendering and execution issues may occur with versions beyond 1.20.1.

[gmuffiness]

I also made a demo video about this feature.
This video was inspired by your work, Max! I hope you enjoy it :)

https://www.youtube.com/watch?v=gPyFrBs45Es

[uukelele-scratch]

why change default port to 56069?

and why comment out init message?

[gmuffiness]

Oh, I hadn’t noticed that settings.js was changed. Thanks for pointing it out!
no particular reason, haha

[gmuffiness]

Currently, the lookAtPlayer and lookAtPosition functions in skills.js handle both 1) taking screenshots and 2) sending requests to the vision model. However, the other functions in skills.js seem to focus solely on controlling Mineflayer’s actions.

This makes me wonder if it might be better to separate these responsibilities by creating a new class, such as VisionInterpreter to handle the vision-related functionalities and use it on agent.js.

I’ll think more about whether this approach would be better. I’d appreciate any feedback or thoughts!

[Lorodn4x]

Can I test this with models from the company Mistral "pixtral-large-latest"?

[gmuffiness]

Changelog

• Extracted vision-related functionalities from skills.js into a new VisionInterpreter class to improve code organization.
• Added an allow_vision option in settings.js to toggle between vision model-based responses and !nearbyBlocks-based logic.

[gmuffiness]

Can I test this with models from the company Mistral "pixtral-large-latest"?

@Lorodn4x

Changelog

• Removed promptImageConvo and added a sendVisionRequest method for each provider (currently supports OpenAI and Mistral).

Initially, I only considered OpenAI models, but I’ve updated the implementation to support Mistral vision requests as well, as their format is slightly different. (https://docs.mistral.ai/capabilities/vision/)
Now, vision interactions are possible with the Mistral model too!
(As a side note, the pixtral-12b model seems to be slightly less effective at action selection compared to gpt-4o.)

[Vineethm0410]

You should consider adding the vision models from Groq as well. That way free users can also test out the vision capability.

[uukelele-scratch]

You should consider adding the vision models from Groq as well. That way free users can also test out the vision capability.

Also Gemini.

At this point it might be better to add a "vision_model" in profile.json

[gmuffiness]

@Vineethm0410 @uukelele-scratch
Two providers added!

[MaxRobinsonTheGreat]

this looks very promising. is it near completion?

[gmuffiness]

Yes, i think the basics are all done.
Let me know if anything needs to be modified, and I’ll update it right away.

Note: If npm install doesn’t work, try installing Node.js version 18.20.5, then run nvm use 18 before installilng.

[gmuffiness]

I noticed you added code_model to recent PR. To align with this, i'll update to set vision model in profile like @uukelele-scratch suggested.

[gmuffiness]

Changelog

• Merge the latest commit changes and add vision_model for image interpretation. If not specified, model will be used.
• Fallback to text-based description when using a non-vision model (e.g. gpt-35-turbo) or an unimplemented provider (e.g., DeepSeek) with vision features
• Update README
• Currently supports the following providers: Google, OpenAI, Anthropic, Mistral, Groq, XAI

[MaxRobinsonTheGreat]

@gmuffiness Is this ready for review?

[gmuffiness]

@MaxRobinsonTheGreat Yes!

[MaxRobinsonTheGreat]

I haven't tested yet, just have a small request and needs to merge with main

[MaxRobinsonTheGreat]

remove these comments

[gmuffiness]

I’ve removed the comments!

Also, I made a small change—sweaterdog had an issue during the installation process. So, I updated the way importing node-canvas-webgl package as like prismarine viewer library.
If you encounter any errors during installation or elsewhere, please let me know!