Hello friends, welcome back to my blog, and happy Friday.  Today I want to continue our journey into Generative AI Chat Copilot development.  I had planned to write a different article today, but I think this topic is more appropriate, to give an overview of the different building blocks in developing a Chat Copilot.  It’s possible that some of this information applies past a simple Chat/Copilot example, but since that’s all I’ve really been researching, I wanted to make that the focus and I’ll explore other use cases where Generative AI can be useful (and how to develop those use cases) in the future.  If you missed my first blog post on this topic: [https://dylanyoung.dev/insights/create-a-local-chat-assistant-with-openai-and-microsoft-chat-copilot/](https://dylanyoung.dev/insights/create-a-local-chat-assistant-with-openai-and-microsoft-chat-copilot/), make sure you check that out, since I covered installing the Microsoft Chat Copilot open source repository on your local machine (which is great if you want to build a chat copilot for your own purposes).

This blog post, however, will focus on the different pieces that were represented in Microsoft’s Chat Copilot example, or some other common elements when you are building a chat copilot.  I’m going to call out below, where Microsoft’s offerings fit in and then describe some of the more mainstream options that are available.  In a future blog post, I’ll look through each specific piece and do a deep dive comparison of the options.

Before we explore the different pieces of a Chat Copilot Generative AI application, I wanted to briefly describe the flow that is required.  I found the following very helpful in describing custom development with a Chat interface:

![Source: https://www.antoinebernard.com/diy-support-chatbots-creating-your-first-ai-assistant-on-your-own-documentation/](https://cdn.sanity.io/images/lanua4su/production/11bed825d852a7c4511ce2ea7d35877c4d97c3f6-921x492.png)

Let's take a few moments to break down the diagram above and delve a little deeper into each of its parts. We'll also discuss some of the options you may have when building these components of the application.

## Large Language Models (LLM)

As previously mentioned, the LLM is the most important part of any Generative AI application. While it is possible to use these LLMs directly with ChatGPT ([https://chat.openai.com/](https://chat.openai.com/)), it may not always be the most useful option. Additionally, there are many LLMs available beyond just OpenAI's models, such as LLaMa (Meta) and BERD (Google), each with different use cases. Some are more specific to chat interfaces, while others are more code or multimedia specific.

If you are new to this field, it's worth taking a look at [https://huggingface.co/](https://huggingface.co/), which is a website that lists the different models and allows for filtering based on criteria such as licensing and model types. Having a solid list of available models is a great place to start, and it's important to note that you may use multiple models in your specific application to build better and more rounded applications.

It's important to note that in many cases, using multiple LLMs simultaneously can be helpful in improving your application's ability to return relevant information.

For those who read my first article on setting up the Microsoft Chat Copilot, that open-source project allows you to use either Azure Open AI’s LLMs or Open AIs LLMs.  It is also probably possible that you could connect to other similar LLMs as well, just some modifications may be required to connect/authenticate to those different systems.

## Embeddings

An Embedding is a machine-readable representation of text that is converted into a multi-dimensional array of float numbers. These numbers represent a vector, and similar phrases or words will have very similar vectors. By collecting and storing these vectors, an application can match up similar phrases that are searched for in the future by users.

## Vector Database/Store or Search

The Embeddings or Vectors are saved in a database, typically a database specialized in storing vectors, like a Vector Store/Database such as Azure Cognitive Search, Chroma, or Pine Cone (there are many more to choose from, but we’ll start with some of the most popular ones here). Think of a Vector database and the creation of Embeddings as creating a long-term memory store for our application, which helps it to perform better over time.

To make the most of storing large amounts of vectors and get faster results, you'll need a Vector store designed specifically for that purpose. One important aspect of such a store is effective indexing of the data. When you map a user prompt to a vector and calculate the closest distance for vectors in the database, the multi-dimensional requests can take a lot of time. While you could store vectors anywhere, even in a notepad file, a Vector store designed for storing vectors will provide many benefits.

Another popular topic that frequently arises when discussing Vector Searching (such as using Azure Cognitive Search, which can act as both a Vector database and a Search index) is the ability to perform Hybrid Searches. Hybrid search in the context of Generative AI involves using both traditional word searching and vector searching to improve the relevance of search results for a given query. This approach combines the strengths of both technologies to provide more accurate and useful results. For example, a hybrid search for a chat copilot application could involve using vector searching to match similar phrases or words and then refining the results using traditional word-searching techniques.

## Large Language Model SDK/Orchestrators

Honestly, there doesn’t seem to be an official name to describe these from my research.  Often times I’ve seen these lumped into the LLM category, however, that just doesn’t make sense.  These tools do not act like an LLM and instead work in the middle of everything in the Chat Copilot application.  When a prompt in your chat interface comes in, your request is handled first by the LLM SDK/Orchestrator, which can then add additional context to that request.  An example, is that maybe you want to have the LLM respond in a different language or just a different tense, this is just one example of what the Orchestrator can handle for the application.  Another use case is combining the initial prompt with additional context about the user, such as previous prompts they entered, or using some of the embedding data to help the LLM respond with more focused content.  I will cover this topic further in upcoming blogs and walk through specific use cases.

In our previous blog, where we set up the open-source Microsoft Chat Copilot ([https://github.com/microsoft/chat-copilot](https://github.com/microsoft/chat-copilot)), its orchestrator was handled by Microsoft’s specific option called `Semantic Kernel`.  This SDK can be extended with .Net which is one of the main reasons you would choose to use Semantic Kernel over other potential options if you are a .Net shop.  Some drawbacks however might be that you need to host additional services to handle this orchestration layer, as well as it’s not as popular or has quite as many features as the next option I’m going to describe in the next paragraph.

Another more popular option is to use an SDK called LangChain (mentioned above in our diagram).  This SDK is built to handle more use cases (detailed below) and even works with Typescript/Node, making it a better option in many cases, because it can be used alongside a more modern Next.js application ([https://vercel.com/templates/next.js/langchain-starter](https://vercel.com/templates/next.js/langchain-starter)).  I will admit the Vercel starter kit doesn’t look quite as nice as the Microsoft Chat Copilot, but there are many more options and it’s super easy to spin that up in Vercel to begin testing out these different use cases.

![Source: https://datasciencedojo.com/blog/understanding-langchain/](https://cdn.sanity.io/images/lanua4su/production/dee3ac8dcd5439577459b39534d48fd3f9a6eebc-891x708.webp)

I’ll go into more detail about the different modules that are available with LangChain and use cases where they can be useful in a future blog post.  For some of the Proof of Concepts I’m working on, I’m likely going to use LangChain because it’s feature-rich and easy to use.

## Chat Interface

Finally, it’s briefly mentioned above, but you’ll need a chat interface, this can be pretty simple, so I won’t call it out specifically below.  In our Microsoft Chat Copilot example in my previous post, this was simply a frontend React application running as a SPA, but you could build this with Angular, React/Next.js, .Net, Blazor, etc.  It really doesn’t matter, just the application will connect typically via API to your Orchestrator/LLM SDK.  So with that described let’s now take a closer look at each of the pieces more closely.

Getting Started with Chat Copilot Generative AI Development

Hello Friends, welcome back to my blog.  Today I’m spending some time playing with Open AI, which is a fun and popular topic.  What I want to achieve, is the ability to create a chat interface locally (for local development or to start building a plugin for future use cases).  What I’m going to use is this repository: [https://github.com/microsoft/chat-copilot](https://github.com/microsoft/chat-copilot), which is a free open-source tool that you can get up and running with on your local machine.  Up to this point, my only experiences with Chat GPT and Open AI have come in the form of using [https://chat.openai.com](https://chat.openai.com/) which is nice, but what if I wanted to start building a plugin and extending this further?  Now the GitHub repository that I’m going to walk through, allows you to use either OpenAI or Azure OpenAI.  Until a few days ago, I was still a little unclear what that difference meant, and because of that difference, it’s important to address some things before we get too far into this blog post.


## So what is the difference?

Really it’s quite straightforward (and no they aren’t the same thing), Open AI is just a public Large Language Model (LLM) that’s being trained all the time by various inputs from the public ecosystem, while Azure Open AI, is just taking all those LLM’s that you know and love and moving them and adding security around them.  I would assume they are still learning from the public, but when you use Azure’s version of Open AI, any data that you supply to it, will stay within the confines of your Azure ecosystem.  Also, you can use other Azure Cognitive Services, which allows you to use more rich information potentially from your users or the data they provide to power this LLM.  And you don’t need to worry about that data escaping into the general population.  That is why, today I will be setting everything up to use Open AI only, not Azure Open AI, and it’ll be important whatever I plan to build, must be public information. (no sharing customer or trade secrets to the public LLM)


## What is a LLM

Well as I’ve already written out the acronym, it stands for Large Language Model, but what does that really mean

A Large Language Model is a type of artificial intelligence that specializes in language processing. These models are trained on vast amounts of text data and are able to generate human-like text outputs, such as in a chat interface. The technology behind LLMs has advanced rapidly in recent years, leading to many exciting new applications in natural language processing and conversational AI.  (Did you know that Chat GPT just wrote that paragraph 🙂)

In today’s article, we aren’t really going to dig too deep into the nuances of Chat GPT 3.5 turbo vs. 4, but we will be using mostly Chat GPT 3.5 turbo in our local development environments, because relatively cheaper to run since it’s not trained on vast amount of parameters like Chat GPT 4.  Keep in mind that the more you train a model, the more likely it will be able to provide valuable insights vs. random gibberish.  Although I will say, even 3.5 does an excellent job responding to your input, just keep in mind that version 4 is even better.


## Getting Started: Repository Setup

Before we can get too deep into this setup, there are a few things we must do first.  First, you should go on over to GitHub to the repository that we’ll be setting up today: [https://github.com/microsoft/chat-copilot](https://github.com/microsoft/chat-copilot).  Likely we will need to make modifications to this repository, so instead of just cloning this repository, I went ahead and forked it over to my own account instead: [https://github.com/dylanyoung-dev/chat-copilot-play](https://github.com/dylanyoung-dev/chat-copilot-play).  This is where we will begin.  At this point, let’s go ahead and clone this repository locally.  


## Getting Started: Open AI - API Access

Now that we have our repository locally, we need to get the API key access information for the Open AI service we’ll be using.  Since I don’t currently have access to Azure Open AI, plus honestly for these simple use cases, I don’t have a need for it, I’ll be configuring a key in Open AI directly (and will walk through the steps).  But if you want to start with Azure Open AI, the steps are pretty much the same, and you can use those Azure-based LLMs instead, just follow the steps in the Readme for further information ([https://github.com/dylanyoung-dev/chat-copilot-play#instructions](https://github.com/dylanyoung-dev/chat-copilot-play#instructions)).

To get our API key that we’ll need to connect to our local Chat Copilot, we’ll need to head over to [https://platform.openai.com/overview](https://platform.openai.com/overview) and make sure you already have an account and are logged in to this account.  Once you are logged in, on the top right, you’ll see a picture of yourself (if you configured this) and something that says “Personal”, it’s possible it could say something else if you’ve configured your account differently than the personal account that I have configured.  Either way, click on that and it’ll show a new navigation dropdown, and click on “View API Keys”.  This is where we will go to create our API key.  It’s quite easy to create our API key, just click on “Create new secret key” and then give that key a name, for this blog post, I am going to give my key the name “Microsoft Chat Copilot”, and then click “Create secret key” and then copy your created key, keeping in mind this is the only time you’ll see the key, but you can always go in and delete keys and then recreate them if you lose track of your API key.


## Setup our Chat Copilot

Alright now it’s time to setup the repository that we cloned in the previous step.  You can follow along with my instructions here, or you can read more in the Read me here: [https://github.com/microsoft/chat-copilot#instructions](https://github.com/microsoft/chat-copilot#instructions).  For this blog, I am on a Windows machine so I am going to go through those steps on the readme.  The steps will help you configure this React (for the frontend) and .Net for the Server side application, at some point in the future, I’ll cover how to do these steps with a Next.js or Node-based application, since that makes more sense to me than using .Net.

Let’s open a favorite terminal, make sure you are in “Administrator” mode, and then make sure you are also in the root of your repository.  Once you are there, you’ll need to make sure you install the prerequisites and configure your environment, so you’ll need to change the directory to the scripts folder, which you can do with a simple `cd .\scripts\` in your favorite terminal.  Once you are there now, run the following command:

```powershell
> .\Install.ps1
```

What this will do, is use Chocolatey (if you don’t have it already, it’ll also install this for you) to install dotnet-7.0-sdk which is needed for the Server side interactions, nodejs (for the frontend) and yarn.  Honestly, I’m not sure why it installs yarn, since a lot of users use NPM instead.  Be aware that this step took a couple of minutes to complete, so walk away and grab something to drink while you wait.

Now with everything installed, you can configure the application.  In this step, you’ll need your API key from Open AI that we got in a previous step.  If you are configuring the Copilot with Open AI like I am, you would be running the script like this:

```powershell
> .\Configure.ps1 -AIService 'OpenAI' -APIKey '<Your Key Here>'
```

This will configure SSL as well as configure a .env file here: `webapp\.env` 

Once everything goes smoothly (hopefully), then it’s time to start our two services (the React frontend and the API Layer using .Net).  To start those services, we now (assuming you haven’t had any issues up to this point) need to run the following command:

```powershell
> .\Start.ps1
```

Now hopefully it magically starts for you, however, it didn’t for me, and I think a lot of that had to do with Yarn being used.  The API started just fine, but since I had problems with the Frontend I did end up making some changes.  Luckily you don’t need to stop the running API layer, and can make changes to the front-end and then simply run `.\Scripts\Start-Frontend.ps1` or you can even start a new terminal window (as an Administrator), and navigate to the web app route here: `.\webapp\` and simply run `npm run start` since the React application is using CRA (Create React App).  This setup definitely leaves a lot to be desired so I might make some changes and release a new blog on how to configure that new setup.

I chose to set up a new terminal and run `npm run start` and then I found there were a lot of dependency issues that I needed to work through.  So I just wanted to detail those below, to hopefully help others with those issues.  First off, Chocolatey attempted to install the latest Node version (20 or the latest version), but I use Node Version Manager on my machine to manage multiple Node/NPM versions across different projects I work on, and the method of installation with Chocolatey didn’t exactly work well with that setup, so I needed to ensure I have NVM configured to use the latest version by just running the following commands in my terminal/powershell:

```powershell
> nvm install "20.3.1"
> nvm use 20.3.1

# And to confirm
> node -v
v20.3.1
> npm -v
v9.6.7
```

Once you are done, you can navigate to your `.\webapp\` folder and run `npm install or npm i` and see where that gets you. For me, I was having dependency issues, which I don’t think were related to the node/npm versions I had running.  So I ended up just running a `npm i --legacy-peer-deps` to get it to install correctly.  Though honestly, that doesn’t seem to be a good way to handle it, but some of their dependencies didn’t work with React 18.2.  

Once you are done, you should now be able to run `npm run start` to start your React front end. You should now see in your browser at [`http://localhost:3000/`](http://localhost:3000/) the running React application.  Should look something like the view below:

![](https://cdn.sanity.io/images/lanua4su/production/bf957bb24239d49274e8115d3c472d1d81352afd-1017x590.jpg)

We are now all set to start building a plugin or enhancing this application to add new capabilities or enhance the UI.  I think one of my next tasks is to update this to use Chakra UI with React and use the Sitecore Blok design system.

But before we end today’s topic…..


## Why Setup a Chat Copilot Locally

So we’ve done all this work and we now have a chat client locally, but what’s the next step from here?  Well the purpose behind getting this setup, is so that we can start to create a custom Plugin for Open AI.  This could be a copilot that you plan to release as a separate app on the web with a built-in plugin or this could be a local development environment to build your plugin to then release to the Open AI marketplace.  We’ll continue this series on Open AI Chat Copilots in the future, so be on the lookout for my next blog on this topic.