Knowledge

Knowledge bases provide context to your agent’s AI models. They allow your agent to answer questions based on documents, websites, or other data sources.

Creating a knowledge base

Create a knowledge source in src/knowledge/:

import { Knowledge, DataSource } from "@botpress/runtime";

export default new Knowledge({
  name: "documentation",
  description: "Product documentation",
  sources: [
    // Add knowledge sources
  ],
});

Data sources

You can add knowledge from several different data sources:

Website source

There are multiple ways to index websites:

From a sitemap

Index a website using a sitemap XML file:

import { Knowledge, DataSource } from "@botpress/runtime";

const WebsiteSource = DataSource.Website.fromSitemap(
  "https://example.com/sitemap.xml",
  {
    filter: ({ url }) => !url.includes("/admin"),
    maxPages: 1000,
    maxDepth: 10,
  }
);

export default new Knowledge({
  name: "website-docs",
  sources: [WebsiteSource],
});

From a base URL

Crawl a website starting from a base URL (requires the Browser integration):

import { Knowledge, DataSource } from "@botpress/runtime";

const WebsiteSource = DataSource.Website.fromWebsite(
  "https://example.com",
  {
    filter: ({ url }) => !url.includes("/admin"),
    maxPages: 500,
    maxDepth: 5,
  }
);

export default new Knowledge({
  name: "website-docs",
  sources: [WebsiteSource],
});

From `llms.txt`

Index pages referenced in an llms.txt file:

import { Knowledge, DataSource } from "@botpress/runtime";

const WebsiteSource = DataSource.Website.fromLlmsTxt(
  "https://example.com/llms.txt"
);

export default new Knowledge({
  name: "website-docs",
  sources: [WebsiteSource],
});

From specific URLs

Index a specific list of URLs:

import { Knowledge, DataSource } from "@botpress/runtime";

const WebsiteSource = DataSource.Website.fromUrls([
  "https://example.com/page1",
  "https://example.com/page2",
  "https://example.com/page3",
]);

export default new Knowledge({
  name: "website-docs",
  sources: [WebsiteSource],
});

Website source options

Option	Type	Description
`id`	`string`	Optional unique identifier for the source
`filter`	`(ctx) => boolean`	Filter function receiving `{ url, lastmod?, changefreq?, priority? }`
`fetch`	`string \| function`	Fetch strategy: `'node:fetch'` (default), `'integration:browser'`, or custom function
`maxPages`	`number`	Maximum pages to index (1-50000, default: 50000)
`maxDepth`	`number`	Maximum sitemap depth (1-20, default: 20)

File source

Index files from a local directory (development only):

const FileSource = DataSource.Directory.fromPath("./docs", {
  filter: (filePath) => filePath.endsWith(".md") || filePath.endsWith(".txt"),
});

export default new Knowledge({
  name: "local-docs",
  sources: [FileSource],
});

Directory sources only work during development (adk dev). For production, use website sources or upload files to Botpress Cloud.

Directory source options

Option	Type	Description
`id`	`string`	Optional unique identifier for the source
`filter`	`(filePath: string) => boolean`	Filter function to include/exclude files

Table source

Index data from a table:

import { Knowledge, DataSource } from "@botpress/runtime";
import OrderTable from "../tables/OrderTable";

const TableSource = DataSource.Table.fromTable(OrderTable, {
  id: "orders",
  transform: ({ row }) => {
    return `Order #${row.id}\nCustomer: ${row.userId}\nTotal: $${row.total}\nStatus: ${row.status}`;
  },
});

export default new Knowledge({
  name: "orders-kb",
  sources: [TableSource],
});

Using knowledge in conversations

Provide knowledge bases to your conversation handlers:

import { WebsiteKB } from "../knowledge/docs";

export default new Conversation({
  channel: "*",
  handler: async ({ execute }) => {
    await execute({
      instructions: "You are a helpful assistant.",
      knowledge: [WebsiteKB],
    });
  },
});

Refreshing knowledge

Refresh knowledge bases to update their content:

import { Workflow } from "@botpress/runtime";
import WebsiteKB from "../knowledge/docs"

export default new Workflow({
  name: "refresh-knowledge",
  schedule: "0 0 * * *", // Daily
  handler: async () => {
    await WebsiteKB.refresh();
  },
});

Force refresh

Force re-indexing of all content even if unchanged:

await WebsiteKB.refresh({ force: true });

Refresh a specific source

Refresh a single data source within a knowledge base:

await WebsiteKB.refreshSource("my-source-id", { force: true });

Reference

Knowledge props

Show Knowledge props

name

string

required

Unique name for the knowledge base.

description

string

Optional description of what the knowledge base contains.

sources

DataSource[]

required

Array of data sources to index. Can include Website, Directory, or Table sources.

Get started

ADK

Studio

Integrations

Webchat

Desk

Creating a knowledge base

Data sources

Website source

From a sitemap

From a base URL

From `llms.txt`

From specific URLs

Website source options

File source

Directory source options

Table source

Using knowledge in conversations

Refreshing knowledge

Force refresh

Refresh a specific source

Reference

Knowledge props

​Creating a knowledge base

​Data sources

​Website source

​From a sitemap

​From a base URL

​From llms.txt

​From specific URLs

​Website source options

​File source

​Directory source options

​Table source

​Using knowledge in conversations

​Refreshing knowledge

​Force refresh

​Refresh a specific source

​Reference

​Knowledge props

Creating a knowledge base

Data sources

Website source

From a sitemap

From a base URL

From `llms.txt`

From specific URLs

Website source options

File source

Directory source options

Table source

Using knowledge in conversations

Refreshing knowledge

Force refresh

Refresh a specific source

Reference

Knowledge props