Log in
Download
antpb AvatarEdit Profile

antpb

  • antpb

  • Building a More Open Metaverse with Custom 3OV Blocks

    Building a More Open Metaverse with Custom 3OV Blocks

    I need to blog more. It’s such a great way to document a project and maintain confidence that things are actually happening. I’ve been thinking a lot about the role WordPress and systems like 3OV will play in the pursuit of an open metaverse. It occurred to me the most important thing to secure a vibrant future for 3OV will be extendability. People want to do custom stuff almost immediately when using 3D apps. It’s one of the most common questions I get about this plugin. The spirit of modding is in all of us!

    In the last month I shifted focus to the Pro version of the plugin which led to the imagining of an open system where 3OV users (as well as myself) can create and distribute their own custom 3D Blocks. This can be complex when using WordPress innerBlocks due to the unpredictable nature of third party generated content, but in the last few days I have been able to successfully build a framework that allows anyone to inject custom Three.js components into the 3OV app context from a separately registered script and Block. Before I get into the system and the code, I’ll preface, when this launches I will provide a boilerplate and CLI command to generate custom 3OV blocks. All you’ll need to do is focus on Three.js code. Before all that, I’ll need to walk the walk by using this system to build 3OV Pro.

    The first Pro block I mocked up to demonstrate this system is a Mirror Block which allows you to set a size and position for a mirror entity to render in world. There are two contexts that can have injected components, Editor View and Front View. In the following screenshot I demonstrate how a Mirror Block is being added from outside of the 3OV Core plugin with all the custom attributes you need defined in the sidebar. The editor view allows you to have a separate component that is more relevant to the editing experience. You can certainly use the same component for both editor and front end but the option is there.

    Screenshot of a custom "Mirror Block" which adds a mirror in the 3D editor in the block editor interface.

    The following snippet demonstrates the editor side plugin registration in the Edit.js of a custom 3OV block:

    export default function Edit({ attributes, setAttributes, isSelected, clientId }) {
	window.addEventListener('registerEditorPluginReady', function() {
		window.registerEditorPlugin(
			<ThreeMirror props />
		);
	});
Code language: JavaScript (javascript)

    Where <ThreeMirror> renders in the editor as the following:

    import React, { useState, useRef } from "react";
import { Reflector } from 'three/examples/jsm/objects/Reflector';
import {
  PlaneGeometry,
  Color,
} from "three";

export function ThreeMirror(threeMirror) {
	const mirrorObj = useRef();
	const mirrorBlockAttributes = wp.data
		.select("core/block-editor")
		.getBlockAttributes(threeMirror.pluginObjectId);

	const mirror = new Reflector(
		new PlaneGeometry(10, 10), { color: new Color(0x7f7f7f) }
	);

	return ( 
			<group
				ref={mirrorObj}
				position={ mirrorBlockAttributes.position }
				rotation={ mirrorBlockAttributes.rotation }
				scale={ mirrorBlockAttributes.scale }
			>
				<primitive object={mirror} />
			</group>
	);
}Code language: JavaScript (javascript)

    For the frontend, you can enqueue Javascript to register the front plugin. The following is a script that shows very simply how a plugin can be registered from the frontend Javascript of a newly added innerBlock:

    import React, { useEffect } from 'react';
import { Reflector } from 'three/examples/jsm/objects/Reflector';
import { PlaneGeometry, Color } from "three";
// Even drei components.

function ThreeMirrorBlockRender(props) {
	const mirror = new Reflector(
		new PlaneGeometry(10, 10), { color: new Color(0x7f7f7f) }
	);

	return (
		<group
			position={props.position}
			rotation={props.rotation}
			scale={props.scale}
		>
			<primitive object={mirror} />
		</group>
	)
}
// Wait for the plugin system to be ready and register this plugin.
window.addEventListener('registerFrontPluginReady', function() {
  window.registerFrontPlugin(<ThreeMirrorBlockRender />);
});Code language: JavaScript (javascript)
    front end rendering the same mirror component! It's alive!

    In the above scripts we wait for the plugin systems to be ready to accept outside components to add in world. Essentially any vanilla Three.js code can happen here including animations! This is proving to be a huge upgrade for 3OV that will set a foundation for limitless possibilities. This was a bit of a brain dump but I hope it illustrates how you’ll be able to extend 3OV in a very near future. Onward!

    antpb

  • Version 1.5.1 – Avatars and Sound

    Version 1.5.1 – Avatars and Sound

    3OV has had some big updates in recent weeks and it’s finally in a stable state where I can take a step back and highlight what’s new. Let’s start with the obvious, AVATARS!

    1. 3OV Avatar System:

    Say hello to the brand-new Avatar System! This new feature currently allows a single configurable default avatar for visitors of your site. The new system is a first step and will have much more dynamic avatar configurations for logged in users in the future. The avatar system is compatible with most VRM format avatars. One hidden gem in this update is the ability to use your Gravatar as part of your avatar. Rename a mesh in your avatar to “profile” and when you load into a post, it will replace the defined mesh with the current logged in user’s Gravatar profile picture. See the recommended avatars page for some resources.

    2. Enhanced Player Controller in Third Person

    Navigating through virtual worlds with 3OV is getting more seamless! The updated Player Controller delivers an immersive and user-friendly movement experience with newly added mobile controls.

    3. Audio Block and Video Block Enhancements: Sound for Your Worlds

    To further enhance the richness of your virtual world, 1.5.0 adds the Audio Block. With the Audio Block, you can now immerse yourself in a soundscape that either plays globally or is positioned within the scene. The Video Block has been improved to play audio in relation to the video’s plane or mesh location, providing more engaging multimedia experiences.

    4. Other Notable Improvements

    In addition to the Avatar System and Player Controller enhancements, 3OV made some other improvements:

    • A saving indicator has been added to the admin settings page.
    • The 1st person controller has been temporarily removed, but don’t worry – it’ll be back in a future update.
    • A loading animation now appears when users click on “Load World,” giving you a smoother transition into the virtual realm.
    • Video Block elements are now clickable, allowing you to play and pause them with just a click.


    antpb

    ,

  • Updated thoughts: The spatial computing future of WordPress

    Updated thoughts: The spatial computing future of WordPress

    In 2019, I wrote an article for WP Tavern titled “Possibilities of a CMS in the Spatial Computing Future” . I explored the potential role of WordPress in powering Extended Reality (XR) experiences. Since then, we’ve seen an exciting evolution in the metaverse landscape, with spatial computing now being embraced globally by the largest technology companies and impressive devices that eliminate the need for controllers. Today, I stand by the vision I shared four years ago, but with way more insight and firsthand experience bringing 3D WordPress to life.

    Prior to my 2019 article, I was focused exploring how WordPress could facilitate data consumption in platforms like Unity apps and other 3D platforms using the REST API. While these experiments are still relevant today, it was completely missing much bigger opportunities for WordPress in XR.

    Since then, with the development of the Three Object Viewer (3OV) plugin, my perspective has shifted to: “WordPress is the 3D platform.” It’s no longer just the data layer, it can be the entire stack start to finish with zero third party dependencies. Instant immersive publishing for all!

    A screenshot of a character named Pixel saying hello to the visitor and introducing the new NPC block

    I always like to take a step back and frame why I choose WordPress as the platform to build around versus building my own 3D focused CMS from scratch. It comes down to the following:

    • Proven post type data structure
    • Extendable data structure for users
    • Secure user system
    • Rich content editor
    • Extendability through plugins and themes 
    • Community <3

    One thing that struck me recently was the realization just how positively impacting the introduction of the block editor was to adapting to the future of the web. It is no exaggeration to say that without the new editor, I would not have been able to bring this plugin to life. 3D content for WordPress would have likely still been limited to consumption by and of “better” apps.

    The recent news…🍎

    Apple recently announced their first serious step into XR with the Vision Pro headset which focuses heavily on mixed reality content. Many, including myself, believe that this was the signal needed to show the world that the immersive web was not only viable, but the inevitable future of the web. I am very happy to report that with Apple’s added support for WebXR in Safari, the headset will be compatible with 3OV making WordPress fully ready to activate on this device! 3OV comes standard with hand tracking for movement, and based on what I have read from Apple, will be compatible with the new headset. 🥳

    Let’s Take a Quick Look Back at 3OV

    Looking back at the past year since the public release of Three Object Viewer, the plugin has evolved from a simple model viewer into a comprehensive metaverse building experience.(They grow up so fast!) When the plugin first released in March of 2022 I really had no larger ambition than making a 3D model viewer. The hope was to make a simple viewer block that would allow folks to select a single file and share things like avatars or wearables.

    With the introduction of the 3D Environment Block and nine core 3D inner block components, 3OV has transformed into a feature-rich 3D builder. I’m quite proud of where it has ended up!


    I think one of the best new blocks was the introduction of the NPC Block. This block utilizes language models and our open sourced worker for plug and play chat bots in your 3D worlds. I see great potential in AI driven assistants that can live in virtual spaces with knowledge of you and your website to communicate with visitors in your absence.

    A screenshot of the NPC block settings and the avatar beside it. The settings allow input for Name, Personality, and default message.
    screenshot of 3OV User's @emotionull showing off his AI avatar "Phetta" in his website using 3OV. The Phetta character is a purple rate and is explaining to the visitor what the phettaverse is.

    Where to next for 3OV?

    Now, as I chart the course for the next year, I’m setting my sights on several areas of focus for the plugin:

    1. Networked Experiences + Multiplayer Editing
      • Networking is very close to finished. Once this lands my focus will shift to collaborative frontend editing using this network layer.
    2. Third Person Controller
      • Introducing a third-person controller will enable visitors to interact with the virtual environment using avatars. Focus will then go to VR compatibility with these avatars. VRM will be the file type used.
    3. Mixed Reality
      • Building upon the existing mixed reality features in 3OV, I will add configuration options for setting the display type. This will open up opportunities for creating mixed reality-specific blocks, such as a “Screen Block” for defining areas as screens or monitors.
    1. Further enhanced AI
      • Expanding AI and NPC features to include multi-modal or multi-action capabilities will empower users to create more interactive experiences. The ability to program pre-defined tasks for virtual characters will bring a new level of interactivity to websites. I think this could power AI assistants that follow you in your everyday life.

    I’ll be doing blog posts about each of these focuses as time goes on but I think this will be the best direction forward for the most impact.

    What about the spatial future for WordPress?

    This is where I have been noodling on the last few days since the Apple headset release. I think it is almost certain that WordPress will adapt to a spatial computing future in more ways than just the front end of a website. The admin dashboard will likely have great opportunity in giving us truly distraction free writing. Imagine grabbing the sidebar of the editor and pinning it to a corner of your desk. An avatar embodied AI assistant is on the other side of the desk providing you paragraphs of data relative to what you are writing about on screen. You go down the street to get tacos and your AI friend is there still being rendered from your website ready to have a chat while you wait for your friends to arrive. You snap a photo and attach it to a blog post. (This is all mostly possible with 3OV today!)

    A screen capture from WebXR using the AR features in the 3OV Three Object Block. In the photo is a 3D avatar robot girl standing in front of a mexican restaurant.
    A screen capture in WebXR using the AR features in the 3OV Three Object Block. Captured from Android.

    So anyway, I’m rambling. It’s safe to say that I’m excited to see WebXR being supported by practically all XR headset makers. This builds a more open 3D web. The future of WordPress in this space is obvious now more than ever. Back in 2019, I imagined WordPress powering basic 3D data visualization. Today, I celebrate it as a beacon of the open 3D web, and I can’t wait to see how far we can take it in the years to come.

    antpb

    antpb

    ,

  • 1.3.5 released with support for GPT-4

    1.3.5 released with support for GPT-4

    We’re excited to announce that 3OV has had a few recent releases that have improved the overall performance and expanded the NPC features. In 1.3.4 we added compatibility with the WordPress Site Editor preparing 3OV for the future of WordPress. Additionally we’re excited to announce that version 1.3.5 is also released with added support for GPT-4 in the NPC block.

    If you have been granted access to the GPT-4 models you can now use our public worker to handle the requests. Simply edit your 3OV settings and add https://alchemy-gpt-4.sxp.digital as your AI url. Be sure that you are using an account that has gpt-4 access. If you do not have access you can always use our regular gpt-3 worker: https://alchemy.sxp.digital

    If you are the DIY type and would like to self-host your gpt-4 endpoint, we’re happy to report that we have also open sourced the very same worker that we are providing in our endpoint. Here’s the Github Repo for GPT-4 Worker: https://github.com/xpportal/alchemy-worker-gpt4
    In the coming releases we have big changes that might be more impacting so do expect a bit of time before networking, third person player controller, and mobile controls are released. I am very close to stable!

    antpb

    , ,

  • Personalized Chat Agents in the Metaverse: Using the SXP_personality Extension for interoperable AI entities

    Personalized Chat Agents in the Metaverse: Using the SXP_personality Extension for interoperable AI entities

    One of the biggest challenges of the metaverse is creating immersive and engaging experiences that can be shared across different platforms. The 3OV plugin for WordPress aims to put interoperability first. We’re working on a new way to take your creations to the next level with personalized chat agents to follow you across the metaverse. In this article I wanted to document some of the progress I’ve made in the pursuit of interoperable NPCs.

    Three big things I want to cover:

    • Upcoming NPC block
    • SXP_personality glTF extension we are developing around interoperable NPCs
    • Our new Cloudflare Worker for scaling chat agents

    Introducing the NPC Block

    We believe that it should be easy to create a rich experience around NPCs and see a future where they will be perceived to be “living” in your worlds. Through recent proof of concept work I found a really nice user experience for interacting with chat agents in 3OV. These entities are able to give thoughtful responses with added context about my business goals, but also be helpful by reacting to the speakers input. With this exploration I was able to draft a new NPC block for 3OV that we’ll be deciding soon how to release.

    The block has controls for a few properties that I am working into a new glTF file extension. The block will have attributes for name, personality, and other relevant properties. This allows you to inject the personality of your character in every conversation. Below is an example of a personality I’ve been experimenting with :

    #agent is an AI assistant with a cheerful, optimistic, and helpful personality. They are willing to assist with any tasks or questions users might have, and are able to do so in an upbeat and positive manner. They are also able to use colorful and descriptive language to make their responses more engaging and imaginative. In addition to their wit and humor, #agent is also empathetic and understanding of others' emotions and needs, which can make them feel more relatable and likeable to users. #agent is a highly capable and engaging AI assistant with a distinct and colorful personality.
#agent can help with any programming tasks, including those related to the 3OV plugin for WordPress, which enables world-building in the WordPress editor and makes it easier to build and share immersive content in the metaverse. The url for the website is 3ov.xyz. #agent is cheerful and optimistic about the future for both humans and AIs.

#Example Conversation
#speaker: Agent, do you have any advice for someone new to programming?
#agent: Sure thing! My top tip would be to always keep an open mind and a positive attitude. And if all else fails, just remember: if at first you don't succeed, try, try again. And then if that still doesn't work, call it a day and go get a coffee.
###
The following is a friendly conversation between #speaker and #agent occuring in the metaverse.

REAL CONVERSATION
#conversation
#speaker: #input
#agent:Code language: PHP (php)

    The properties I focused on were required variables that a MagickML endpoint would expect. MagickML is a visual IDE for no-code data pipelines and multimodal agents. It’s really great for extremely complex communication processing allowing for lifelike AI conversations.

    Screenshot of the Magick ML interface

    The spell I’ve been using against this block considers the injected personality, responds conversation input, and also runs a second query for sentiment analysis of the NPC message. From there, sentiment is assigned to facial parameters on the avatar to emote. Below is a rare example of making the NPC “angry” by literally threatening it. 😬 I’ll be nicer in the future. <3

    AI Pets

    Another really great application for this will be with pets. AI doesn’t have to only be smart. AI Pets are an opportunity that we’re exploring to prove that this concept is not only specific to human-like chat bots. With 3OV + Interoperable NPCs we can enable a Tamagotchi of the metaverse. These pets can follow you around and have a personality that learns from you over time. Pets can always consider their lore which impacts the tone and content of their responses. If it is hungry, it will be a bit more grumpy in its responses. /feed to give your friend a snack. This will all be possible through MagickML implementations and is only limited by your imaginations. It’s important to note, not everything needs to reach out to OpenAI to generate a rich response. You could entirely parse strings of input to figure out intent and generate 30 random responses at the ready relative to the topic. MagickML allows you to build a graphs to trigger this variation.

    A glTF extension for NPC personalities

    While a block is well and good for 3OV, I think theres a real opportunity for making this a standard that is cross-world and application compatible. Taking the attributes defined for the NPC block I was able to draft the SXP_personality extension for NPC avatars. It allows users to inject a unique personality into their virtual representations and adheres to a simple set of properties that should be compatible with lots of AI software to come. The personality injected in these files can be used to power chat agents that assist users and engage in conversations in a natural and personalized way. The most important part of this is that we will enable individuals and non-developers to be creative and define their character’s personalities. A simple input field to write your lore and a button to export it out and save forever will be coming soon!

    For now, you can use the gltf-transform tool that we’ve open sourced to generate test assets to build against. Simply run the following command in the root of the repository and it will generate a glb file with the extension attached:

    node script.js someobject.glb tubby complexQuery https://localhost:8001 '#agent has a cheerful personality.' 'nya nya!'

    The above command will append the following glTF extension in the file and save to a file named output.glb. 

    {
  "nodes": [
    {
      "name": "tubby",
      "extensions": {
        "SXP_personality": {
          "agent": "tubby",
          "spellName": "complexQuery",
          "host": "https://localhost:8001",
          "personality": "#agent has a cheerful personality.",
          "defaultMessage": "nya nya!"
        }
      }
    }
  ],
  "extensionsUsed": [ "SXP_personality" ]
}

    You should be able to rename that file to output.vrm and use it as expected in VRM compatible applications. If you have any bugs please file an issue! This is MIT licensed and open to any and all collaboration. We’re open to collaborating on this through any standards group that may be interested in the pursuit.

    Alchemy CF Worker

    I saw another need in this space to really help WordPress sites using 3OV scale chat agents. Magick can do a lot. I plan to continue contributing to it as long as I can because I believe in the mission of open sourcing AI. I also recognize some people might just want a simple chat bot to talk back and forth with using text. With that problem to solve I open sourced a CloudFlare worker that aims to imitate a simpler Magick spell. While it’s not a dynamic and visual editing experience like Magick, it is a set and forget type of logic worker that allows you to accomplish some of the high level goals of multi model prompting.

    An early test of the worker before adding the Personality property. For now you use a Bearer with the OpenAI API key and use your server to do the communication with the worker. (NEVER EXPOSE THIS ON THE FRONT END AND LOCK IT DOWN)

    Since inspired by MagickML I thought we could call our lite version of a spell Alchemy as it largely works to do simple logic like combining strings and merging responses. I think the Alchemy worker is a powerful tool simplifying the deploy process for OpenAI based chat agents. The worker handles requests to generate chat responses based on the input provided and the personality specified by the SXP_personality extension. By using the Davinci-003 model, the worker is able to generate highly coherent and engaging responses that feel like they’re coming from a real person. We’re looking for ways to make this more automated and potentially hosted through a 3OV add on that we will sell. Our goal is to allow this to be DIY with the option to pay us to host. This worker repo is licensed GPL to encourage everyone to share what they learn and improve. More to come, get involved on Github!

    Cross-Platform Interop

    As the metaverse continues to grow, interoperability will be key to ensuring that users can seamlessly move between platforms. I think the SXP_personality extension and Alchemy worker will be helpful tools to advance this interoperability goal. It could enable cross-platform personalized and engaging interactions and also gives us a specification to build around as the dust settles. I hope to see more options become available for designing agents. Whether you’re a developer looking to build the next generation of chat agents, or simply want to create a more immersive and engaging experience in the metaverse, I suggest keeping up with 3OV, Magick, and the SXP_personality extension to be first to try these tools out. With the NPC block I hope we’ll lower many barriers to entry and promote democratizing AI in the metaverse longer term.

    antpb

  • Exploring the Metaverse Potential of MagickML: A Multishot System Builder for AI Entities

    Exploring the Metaverse Potential of MagickML: A Multishot System Builder for AI Entities

    Artificial intelligence (AI) has come a long way in just the last year, and one area that has seen significant progress is natural language processing. This technology allows computers to understand and generate human-like language, enabling them to communicate with people in a more natural and intuitive way. With new technology comes new CMS-like software to enable anyone to participate. Enter MagickML, a multishot system builder that leverages the capabilities of GPT-3 and other OpenAI language models to create AI assistants and characters with complex communication patterns. It is, in a sense, a WordPress of AI in that it manages AI themes and content. In this post, we’ll take a closer look at Magick and explore its potential to revolutionize the way we interact with AI in virtual worlds, particularly in 3OV.

    What is Magick?

    MagickML is a JavaScript-based framework that allows game designers and developers to rapidly create powerful natural language systems and prototype NPC logic in any web based experience. It utilizes a visual coding style interface, making it easy for users to create “spells” that define complex logic and queries to generate responses. These spells are almost like WordPress themes for AI, allowing users to customize the behavior and communication patterns of their AI assistants and characters. Being JavaScript driven allows for a much wider range of developer to participate. And the icing on the cake, it is open source and Apache Licensed!

    One of the key features of Magick is its ability to integrate with GPT-3 and any other OpenAI language models to logically respond to requests. This enables users to leverage the advanced capabilities of these models to create AI assistants and characters that can understand and generate human-like language. You can use any mixture of language models to generate a final response.

    The framework also allows users to incorporate memory recall or factual biographical information into their responses, as well as any other logic they would like to include to make it feel more natural. Between the final output of an entity, users are able to utilize JavaScript logic to parse many responses from OpenAI to build a final query that generates a more human-like response. Where this differs from something like chatGPT is its ability to send multiple request before giving the end user final output. This gives you room to tweak all the funny bits of language models where a single query might not give you the response you want.

    Example of JS logic from Eliza parsing responses and logging factual information

    The Potential of Magick:

    The potential of Magick facilitating the creation of complex communication patterns with OpenAI language models is vast. In the gaming industry, Magick could be used to create more immersive and engaging experiences for players using AI characters with personality traits, making the game world feel more real and alive.

    I found the potential of chatbots and virtual assistants particularly interesting for 3D web applications. By using Magick to create AI assistants, businesses and organizations could provide more personalized and efficient customer service. Your AI assistant can sell for you within your website. And with 3OV this could even be in a 3D world!

    I built a Magick driven proof of concept in 3OV that allows WordPress site owners to define a URL where their Magick instance lives and make post requests from in-world to display responses above an avatar’s head. This avatar is given a personality through “spells” and is able to remember facts about each person that talks thanks to the awesome work Magick contributor m00n has built and shared with their Eliza entity.

    Example Integration

    The following integration of Magick in 3OV very basically shows how you can integrate spells and responses in your apps.

    function ChatBox(props) {
	const handleChange = async (event) => {
		event.preventDefault();
	};
	const handleSubmit = async (event) => {
	  event.preventDefault();
	  // Get the value of the input element
	  const input = event.target.elements.message;
	  const value = input.value;
  
	  // Send the message to the localhost endpoint
	  const client = 1;
	  const channelId = "three";
	  const entity = 11;
	  const speaker = "antpb";
	  const agent = "aiko";
	  const channel = "homepage"; 

	  try {
		const spell_handler = "four";
		const spell_version = "latest";
		const url = encodeURI( `https://localhost:8001/spells/${spell_handler}/${spell_version}` )
		const response = await axios.post(`${url}`, {
			inputs: {
			  Input: value,
			  Speaker: speaker,
			  Agent: agent,
			  Client: client,
			  ChannelID: channelId,
			  Entity: entity,
			  Channel: channel,
			},
		  }).then((response) => {
			const data = response.data;

			const outputs = data.outputs;

			const outputKey = Object.keys(outputs)[0];

			const output = outputs[outputKey];

			props.setMessages([...props.messages, output]);
		  });
		} catch (error) {
		console.error(error);
	  }
	};
   
	return (
	  <div style={{ marginTop: "-140px", position: "relative", bottom: "15%", left: "5%", width: "50%", height: "10%" }}>
		<form style={{display: "flex"}} onSubmit={handleSubmit}>
		  <input type="text" name="message" onInput={handleChange} onChange={handleChange} />
		  <button type="submit">Send</button>
		</form>
	  </div>
	);
  }Code language: JavaScript (javascript)

    In this example implementation of Magick in the 3OV plugin, the ChatBox function handles the submission of user input and sends it to a localhost endpoint through an HTTP post request. (In this case I have Magick running locally but you could install this on any server.) The request includes a few parameters like the input value, speaker and agent names, client and channel IDs, and entity and channel values. These parameters are passed to the endpoint as an object in the request body and is used to process and store information about the context of the conversation in Magick.

    Upon submission, the function sends the message to a specified Magick spell handler and version, which are defined in the URL of the request. The spell handler and version are set to “four” and “latest” respectively in my example. You could have many spells and route chats to different spell handlers to dynamically make the requests more cost efficient or more technical depending on your needs.

    		const spell_handler = "four";
		const spell_version = "latest";
		const url = encodeURI( `https://localhost:8001/spells/${spell_handler}/${spell_version}` )
		const response = await axios.post(`${url}`, {
			inputs: {
			  Input: value,
			  Speaker: speaker,
			  Agent: agent,
			  Client: client,
			  ChannelID: channelId,
			  Entity: entity,
			  Channel: channel,
			},
		  }).then((response) => {
			const data = response.data;

			const outputs = data.outputs;

			const outputKey = Object.keys(outputs)[0];

			const output = outputs[outputKey];

			props.setMessages([...props.messages, output]);
		  });
		} catch (error) {
		console.error(error);
	  }
Code language: JavaScript (javascript)

    Upon receiving the request, Magick processes the input and other parameters using the specified spell, and returns a response in the form of an object containing output data. The ChatBox function retrieves the output data and adds it to the list of messages using setMessages. This state is passed to a lower component where the AI assistant avatar file is handled. The message populates above the avatars head and closes the loop of conversation!

    You may notice in the screenshot above there’s json being output with a tone property. That is GPT-3 deciding the tone of the message it is sending! You could easily map this to VRM avatar facial expressions so the AI can naturally interact with you in-world. I plan to get more granular and give the spell control of the default VRM emotion parameters.

    Voice recognition and responses are 100% possible today thanks to the work of Magick contributors. Overall, this implementation of Magick within the 3OV plugin demonstrates simply how the framework can be used to create interactive and personalized communication patterns with AI in a 3D environment. This can go way further and is only limited by imagination.

    Pricing and Caution

    It’s important to note that OpenAI queries could potentially be expensive to use, depending on how many shots to the AI you need to make in a spell. For example, if you are using a spell that costs around $0.02 per query and you spell has multiple queries, this could add up quickly if you are making frequent requests. It’s important to be mindful of this when using Magick and to consider whether it is cost-effective for your needs. Adjusting language models to cheaper options is another way to ease costs. In my testing I have short circuited the AI bits of a spell to return static responses to mock against as I develop ideas.

    I would encourage using Magick as a local personal AI assistant or implementing safeguards in your Magick spells to recognize when someone is abusing the system. This could include defaulting to basic responses rather than querying GPT when abuse is detected. These are very early days for this project so keep your instances locked down as auth and other gated features work their way into the framework.

    Conclusion:

    Magick is a powerful tool for creating AI assistants and characters with complex communication patterns. Its ability to integrate with GPT-3 and other OpenAI language models allows users to leverage the advanced capabilities of these models to create sophisticated responses. The potential applications of Magick are numerous, and it could revolutionize the way we interact with AI in a wide range of industries, from gaming to customer service. I can’t say for certain how we will officially launch integrations for this in 3OV, but the important part is that it is now possible and we will find the right way to launch support through a NPC block. I’m excited to continue contributing to this framework as it takes shape!

    antpb

    ,

  • Three Object Viewer – 1.0.8 Release

    Three Object Viewer – 1.0.8 Release

    In our latest release, we’ve added several new features and improvements to make your content creation experience even better.

    First, we’ve added support for click events for audio objects that contain KHR_audio. This allows users to interact with audio objects in your scenes. Further improvements to this feature will be coming soon. To make an audio object interactable, make sure to include a KHR_audio object to the node of an object and make it collidable in your admin editor. To package audio in your glb files use the Third Room Unity Exporter

    We’ve also added video interactions, allowing users to play and pause videos in your scenes. A paused video will now display a play icon so users can easily resume the video source. Audio support for videos will be added in a future update.

    Below is a demo of audio interactions:

    Additionally, in this release we’ve added VR interactions for portal blocks, so users can easily navigate between different scenes by clicking on the portal using VR controller interactions. This makes it easy to create multi-layered and interconnected metaverse experiences. We plan to support the new Meta browser ability to stay in VR experience while changing pages soon.

    We’ve also fixed several issues in this version:

    • Fixed: Video – autoPlay property was not being properly respected.
    • Fixed: Model and Environment Block – Animations were previously broken since the launch of the Environment Block. All should be good now with the caviat that you cannot use the same object twice in a scene. An update will fix this very soon.
    • Fixed: Sky Block – fixed issue where teleportation to the sky was possible. Stay on the ground!

    With these new features and improvements, 3OV is improving and making it easy for anyone to create and share immersive experiences in the metaverse. Let us know if you have any issues! You can open an issue at the plugin page here. To download the plugin you can visit here or search “Three Object Viewer” in your wp-admin plugin page.

    Try all the new features below!

    antpb

    ,