Beyond Basic Web Search: Building a Specialized AI Agent with Bing Custom Search

by Posted on
5
(1)

When building AI agents for enterprise applications, standard web search often returns irrelevant results that compromise the quality of your solution. During a recent client project, our team needed an AI assistant capable of providing developers with specific documentation and code samples. While Bing Search returned results from various sources including outdated forum posts and unofficial tutorials, we required focused, authoritative content from trusted developer resources.

This challenge led us to implement Bing Custom Search—a powerful solution for grounding AI models within a curated content ecosystem. The impact on our specialized AI assistants has been transformative, enabling domain-specific expertise that consistently delivers accurate results.

If you’ve read my previous post about migrating from Bing Search APIs to Azure OpenAI Agent grounding, you’ll recall how we established basic web search capabilities. Today, I’ll demonstrate how to create truly specialized agents using Bing Custom Search.

Why Custom Search Transforms AI Agent Development

Regular web search serves general-purpose queries effectively. However, when developing AI agents for specific domains, unrestricted internet access often hinders rather than helps. Domain-specific agents require expert-level knowledge within their specialized area.

This became evident during our project. Our AI agent consistently referenced outdated Stack Overflow answers and third-party tutorials containing inaccuracies. The client identified multiple instances where the agent provided deprecated solutions, impacting developer productivity and trust in the system. We invested significant resources filtering noise and validating responses.

Custom search addresses these challenges by creating focused search experiences limited to trusted sources. Think of it as providing your AI agent with a curated reference library rather than unrestricted internet access.

Setting Up Your Custom Search Instance

Before we dive into the code, let’s get our Custom Search instance configured. We’re going to create a dedicated “Grounding with Bing Custom Search” resource in Azure. This is specifically designed for AI agent integration.

Step 1: Creating the Grounding with Bing Custom Search Resource

First, create the Azure resource that will host your custom search configurations. Navigate to the Azure Portal and search for “Grounding with Bing Custom Search”. Configure the resource as follows:

  • Click “Create” to start setting up your resource
  • Choose your subscription and resource group (I recommend using the same resource group as your AI Foundry project)
  • Give it a descriptive name like “MyApp-CustomSearch-Grounding”
  • Select your region (Global is typically fine)
  • Choose your pricing tier based on expected usage
  • Review your settings and click “Create”
  • Once deployed, you’ll have an empty resource ready for configuration

This approach ensures everything remains managed within Azure, providing integrated authentication and consolidated billing.

Step 2: Creating Your Custom Search Configuration

After resource deployment, create your search configurations:

  1. Navigate to Configurations:
    • Go to your newly created resource in the Azure Portal
    • In the left navigation, under “Resource Management”, click on “Configurations”
    • Click “Create new configuration“
  2. Define Your Search Scope: Now comes the fun part – defining what your search will include: Allowed domains: These are the domains your search will actively include. For a developer-focused agent, I typically add:
    • docs.microsoft.com
    • learn.microsoft.com
    • github.com/Microsoft
    • devblogs.microsoft.com
  3. Blocked domains: Sites you explicitly want to exclude. I always block low-quality content farms and sites known for outdated information. Slices: You can create different “slices” of your search for different types of content. For example, one slice for official documentation, another for code samples, and another for community discussions.
  4. Save Your Configuration:
    • Once you’ve defined your search scope, save the configuration
    • You can create multiple configurations within the same resource for different domains or use cases

Strategic domain selection is crucial. Each configuration functions as a specialized search engine tailored to your specific requirements.

Step 3: Connect to Your AI Foundry Project

Similar to regular Bing Search grounding, establish connection with your AI Foundry project:

  1. In your AI Foundry project, go to “Connected Resources“
  2. Click “Add Connection”
  3. Select your Grounding with Bing Custom Search resource
  4. This creates a connection that your agents can use
  5. Reference this connection name in your code (for example, “custom-search-connection”)

Building Your Custom Search Agent

Let’s extend our previous implementation to leverage custom search capabilities instead of, or alongside, regular web search.

Building a Maintainable Agent Architecture

The architecture has been completely redesigned for improved maintainability, testability, and scalability. Let me walk you through the enhanced design approach.

App settings

You’ll notice I’ve moved away from separate configuration sections to an array-based approach. This makes it much easier to manage multiple agents – just add another object to the array instead of creating new top-level sections.

{
  "AzureAI": {
    "ConnectionString": "your-azure-ai-project-connection-string"
  },
  "AzureOpenAI": {
    "Endpoint": "https://your-resource-name.openai.azure.com",
    "ApiKey": "your-api-key",
    "DeploymentName": "gpt-4o"
  },
  "Agents": [
    {
      "Name": "Bing Grounding Agent",
      "Description": "Bing Grounding Web Search Agent",
      "Instructions": "You are a helpful assistant that can search the web for current information.When users ask questions that require up-to-date information, use the Bing search tool to find relevant information and provide accurate, grounded responses.Always cite your sources when providing information from search results.",
      "Deployment": "gpt-4o",
      "Tools": [
        {
          "ToolType": "BingGroundingSearch",
          "ConnectionName": "MyBingGroundingSearch"
        }
      ]
    },
    {
      "Name": "Bing Custom Search Agent",
      "Description": "Bing Custom Search Agent",
      "Instructions": "You are a specialized AI assistant with access to curated, high-quality information sources.When users ask questions, search through your custom knowledge base to provide accurate,up-to-date information. Always prioritize official documentation and trusted sources.",
      "Deployment": "gpt-4o",
      "Tools": [
        {
          "ToolType": "CustomBingGroundingSearch",
          "ConnectionName": "MyAppCustomSearchGrounding",
          "ConfigurationName": "sites"
        }
      ]
    }
  ]
}

This approach lets you define multiple agent types in configuration without changing code. Want a different agent? Just add another configuration section!

The Agent Options Model

First, let’s define our configuration model. This maps directly to our JSON configuration:

namespace BingGroundingAgent
{
    public interface IAgentOptions
    {
        string Deployment { get; set; }
        string Name { get; set; }
        string Description { get; set; }
        string Instructions { get; set; }
        List<ToolsOptions>? Tools { get; set; }
    }

    public class ToolsOptions
    {
        public string ConnectionName { get; set; } = string.Empty;
        public string? ConfigurationName { get; set; } = string.Empty;
        public string ToolType { get; set; } = string.Empty;
    }

    public class AgentOptions : IAgentOptions
    {
        public string Deployment { get; set; } = string.Empty;
        public string Name { get; set; } = string.Empty;
        public string Description { get; set; } = string.Empty;
        public string Instructions { get; set; } = string.Empty;
        public List<ToolsOptions>? Tools { get; set; }
    }

    public class AgentsConfiguration
    {
        public List<AgentOptions> Agents { get; set; } = new();
    }
}

This gives us strongly-typed configuration binding and makes it easy to add new agent types.

The Agent Service Layer

Next, I created a service layer that handles all the Azure AI interactions. This separates the Azure-specific logic from the agent logic:

using Azure.AI.Agents.Persistent;
using Azure.AI.Projects;
using Azure.Identity;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Logging;
using System.ClientModel;
using ToolDefinition = Azure.AI.Agents.Persistent.ToolDefinition;
using ToolResources = Azure.AI.Agents.Persistent.ToolResources;

namespace BingGroundingAgent
{
    public class AgentService
    {
		private readonly IConfiguration _configuration;
		private readonly ILogger<AgentService> _logger;
		private PersistentAgentsClient? _persistentClient;
		private Connections? _connections;
		private AIProjectClient? _projectClient;
		public AgentService(IConfiguration configuration, ILogger<AgentService> logger)
        {
			_logger = logger ?? throw new ArgumentNullException(nameof(logger));
			// Validate configuration
			_configuration = configuration ?? throw new ArgumentNullException(nameof(configuration));
			var connectionString =
				_configuration["AzureAI:ConnectionString"]
				?? throw new InvalidOperationException(
					"Azure AI connection string is not configured"
				);
			var deploymentName =
				_configuration["AzureOpenAI:DeploymentName"]
				?? throw new InvalidOperationException(
					"Azure OpenAI deployment name is not configured"
				);
			var endpoint =
				_configuration["AzureOpenAI:Endpoint"]
				?? throw new InvalidOperationException("Azure OpenAI endpoint is not configured");
			var apiKey =
				_configuration["AzureOpenAI:ApiKey"]
				?? throw new InvalidOperationException("Azure OpenAI API key is not configured");
			// Create Azure AI Project client
			_projectClient = new AIProjectClient(
				new Uri(connectionString),
				new DefaultAzureCredential()
			);
			_connections = _projectClient.GetConnectionsClient();

			// Create Azure AI persistent client
			_persistentClient = new PersistentAgentsClient(
				connectionString,
				new DefaultAzureCredential()
			);
		}

		public async Task<PersistentAgent?> GetAgentAsync(string name)
		{
			try
			{
				var agents = _persistentClient.Administration.GetAgentsAsync();
				var agent = await agents.Where(a => a.Name == name).FirstOrDefaultAsync();
				return agent;
			}
			catch (Exception ex)
			{
				this._logger.LogError(ex, "Failed to get agent {AgentName}", name);
				return null;
			}
		}

		public async Task<Connection?> GetConnectionByToolResourceNameAsync(string toolName)
		{
			ArgumentException.ThrowIfNullOrWhiteSpace(toolName);

			if (_connections == null)
			{
				throw new InvalidOperationException("Connections client is not initialized");
			}

			await foreach (
				var connection in _connections.GetConnectionsAsync(
				)
			)
			{
				if (string.Equals(connection.Name, toolName, StringComparison.OrdinalIgnoreCase))
				{
					return connection;
				}
			}
			_logger.LogWarning("Connection with name '{ConnectionName}' not found", toolName);
			return null;
		}

		public async Task<PersistentAgent?> CreateAgentAsync(AgentOptions agentOptions)
		{
			var agent = await this.GetAgentAsync(agentOptions.Name);

			if (agent == null)
			{
				// Setup tools internally
				List<ToolsOptions> toolList = agentOptions.Tools;
				List<ToolDefinition> toolDefinitionList = new();

				foreach (ToolsOptions tool in toolList)
				{
					if (tool.ToolType == "BingGroundingSearch")
					{
						var bingConnection = await this.GetConnectionByToolResourceNameAsync(
							tool.ConnectionName
						);

						var bingGroundingTool = new BingGroundingToolDefinition(
							new BingGroundingSearchToolParameters(
								[new BingGroundingSearchConfiguration(bingConnection.Id)]
							)
						);

						if (bingConnection?.Id is null)
						{
							throw new InvalidOperationException("Failed to get Bing Search connection");
						}
						toolDefinitionList.Add(bingGroundingTool);
					}
					else if (tool.ToolType == "CustomBingGroundingSearch")
					{
						var connection = await this.GetConnectionByToolResourceNameAsync(
							tool.ConnectionName
						);

						var customConfiguration = new BingCustomSearchConfiguration(connection.Id, tool.ConfigurationName)
						{
							Count = 5,              // Set the number of results  
							SetLang = "en",         // Set the language  
							Market = "en-us"        // Set the market  
						};


						// Wrap the configuration into an IEnumerable  
						IEnumerable<BingCustomSearchConfiguration> configs = new List<BingCustomSearchConfiguration> { customConfiguration };

						BingCustomSearchToolDefinition bingCustomSearchTool = new(
							 new BingCustomSearchToolParameters(configs) 
						);
						toolDefinitionList.Add(bingCustomSearchTool);
					}
				}

				ToolResources toolResources = new();

				if (toolDefinitionList.Count > 0)
				{
					agent = await _persistentClient.Administration.CreateAgentAsync(
						agentOptions.Deployment,
						agentOptions.Name,
						agentOptions.Description,
						agentOptions.Instructions,
						tools: toolDefinitionList,
						toolResources: toolResources
					);
				}
				else
				{
					agent = await _persistentClient.Administration.CreateAgentAsync(
						agentOptions.Deployment,
						agentOptions.Name,
						agentOptions.Description,
						agentOptions.Instructions
					);
				}
			}

			return agent;
		}

		public async Task<PersistentAgentThread> CreateThreadAsync()
		{
			return await _persistentClient.Threads.CreateThreadAsync();
		}
		public async Task<PersistentThreadMessage> CreateMessageAsync(string threadId, string query)
		{
			return await _persistentClient.Messages.CreateMessageAsync(threadId, Azure.AI.Agents.Persistent.MessageRole.User, query);
		}
		public AsyncCollectionResult<Azure.AI.Agents.Persistent.StreamingUpdate> CreateStreaming(string threadId, string agentId)
		{
			return _persistentClient.Runs.CreateRunStreamingAsync(threadId, agentId);			
		}
		public async Task<List<MessageTextUriCitationAnnotation>> GetMessageUrlCitationsAsync(string agentId,string threadId)
		{
			List<MessageTextUriCitationAnnotation> citations = new();
			var afterRunMessagesResponse = _persistentClient.Messages.GetMessagesAsync(threadId);
			var messages = await afterRunMessagesResponse.ToListAsync();
			PersistentThreadMessage lastMessage = messages[0];

			foreach (var contentItem in lastMessage.ContentItems)
			{
				if (contentItem is MessageTextContent textItem)
				{
					foreach (var citation in textItem.Annotations)
					{
						MessageTextUriCitationAnnotation urlCitation =
							(MessageTextUriCitationAnnotation)citation;
						citations.Add(urlCitation);
					}
				}
			}

			return citations;
		}
	}
}

The Agent Class

Now a generic agent class that can handle any type of search agent:

using System.ClientModel;
using Azure.AI.Agents.Persistent;
using Microsoft.Extensions.Logging;

namespace BingGroundingAgent;

public sealed class AzureAgent
{
    private readonly ILogger _logger;
    private readonly AgentService _agentService;
    private readonly AgentOptions _agentOptions;
    private PersistentAgent? _agent;
    private PersistentAgentThread? _thread;

    public AzureAgent(ILogger logger, AgentService agentService, AgentOptions agentOptions)
    {
        _logger = logger ?? throw new ArgumentNullException(nameof(logger));
        _agentService = agentService ?? throw new ArgumentNullException(nameof(agentService));
        _agentOptions = agentOptions ?? throw new ArgumentNullException(nameof(agentOptions));
    }

    public async Task InitializeAsync()
    {
        try
        {
            // Define the agent using the provided agent options
            _agent = await _agentService.CreateAgentAsync(_agentOptions);
            _logger.LogInformation(
                "{AgentName} initialized successfully. Agent ID: {AgentId}",
                _agentOptions.Name,
                _agent?.Id
            );
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Failed to initialize {AgentName}", _agentOptions.Name);
            throw;
        }
    }

    public async Task<AsyncCollectionResult<StreamingUpdate>> ProcessQueryAsync(string userQuery)
    {
        ArgumentException.ThrowIfNullOrWhiteSpace(userQuery);
        if (_agent == null)
        {
            throw new InvalidOperationException(
                "Agent is not initialized. Call InitializeAsync first."
            );
        }

        try
        {
            _logger.LogInformation("Processing query: {Query}", userQuery);

            // Create a new thread for the conversation
            var thread = await _agentService.CreateThreadAsync();
            _thread = thread;

            // Add user message
            await _agentService.CreateMessageAsync(thread.Id, userQuery);

            // Get response from agent
            return _agentService.CreateStreaming(thread.Id, _agent.Id);
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Error processing query: {Query}", userQuery);
            throw;
        }
    }

    public async Task GetCitationSourcesAsync()
    {
        if (_agent == null)
        {
            throw new InvalidOperationException(
                "Agent is not initialized. Call InitializeAsync first."
            );
        }

        if (_thread == null)
        {
            throw new InvalidOperationException("No active thread. Process a query first.");
        }

        var citations = await _agentService.GetMessageUrlCitationsAsync(_agent.Id, _thread.Id);

        if (citations.Count > 0)
        {
            Console.WriteLine();
			Console.WriteLine("---------------------------------");
			Console.WriteLine("References:");

            foreach (var citation in citations)
            {
                Console.WriteLine($"* {citation.UriCitation.Uri}");
            }
        }
    }
}

Bringing It All Together

using Azure.AI.Agents.Persistent;
using BingGroundingAgent;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;

var builder = Host.CreateApplicationBuilder(args);

var configuration = new ConfigurationBuilder()
    .SetBasePath(Directory.GetCurrentDirectory())
    .AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
    .AddEnvironmentVariables()
    .Build();

builder.Services.AddSingleton<IConfiguration>(configuration);
builder.Services.AddSingleton<AgentService>();

// Register agents from configuration
var agentsConfig = new AgentsConfiguration();
configuration.GetSection("Agents").Bind(agentsConfig.Agents);

// Register each agent as a named service
foreach (var agentConfig in agentsConfig.Agents)
{
    var agentType = agentConfig.Name; // Fallback to name if type is not set
    builder.Services.AddKeyedSingleton<AzureAgent>(
        agentType,
        (provider, key) =>
            new AzureAgent(
                provider.GetRequiredService<ILogger<AzureAgent>>(),
                provider.GetRequiredService<AgentService>(),
                agentConfig
            )
    );
}

builder.Services.AddLogging(logging =>
{
    logging.AddConsole();
    logging.SetMinimumLevel(LogLevel.Information);
});

var host = builder.Build();
var logger = host.Services.GetRequiredService<ILogger<Program>>();

// Get available agents from configuration
var availableAgents = new AgentsConfiguration();
configuration.GetSection("Agents").Bind(availableAgents.Agents);

// Let user select which agent to use
Console.WriteLine("Available agents:");
for (int i = 0; i < availableAgents.Agents.Count; i++)
{
    var agent = availableAgents.Agents[i];
    Console.WriteLine($"{i + 1}. {agent.Name}");
}

Console.Write("Select an agent (1-{0}): ", availableAgents.Agents.Count);
var selection = Console.ReadLine();

if (
    !int.TryParse(selection, out int agentIndex)
    || agentIndex < 1
    || agentIndex > availableAgents.Agents.Count
)
{
    logger.LogError("Invalid agent selection");
    return;
}

var selectedAgentConfig = availableAgents.Agents[agentIndex - 1];
var agentKey = selectedAgentConfig.Name;
var selectedAgent = host.Services.GetRequiredKeyedService<AzureAgent>(agentKey);

try
{
    logger.LogInformation("Initializing {AgentName}...", selectedAgentConfig.Name);
    await selectedAgent.InitializeAsync();

    while (true)
    {
        Console.Write("\nYou: ");
        var input = Console.ReadLine();
        if (string.IsNullOrEmpty(input) || input.ToLower() == "exit")
            break;

        logger.LogInformation("Processing query: {Query}", input);

        Console.Write("Assistant: ");
        await foreach (var update in await selectedAgent.ProcessQueryAsync(input))
        {
            try
            {
                if (update.UpdateKind == StreamingUpdateReason.MessageUpdated)
                {
                    if (update is MessageContentUpdate messageContent)
                    {
                        // Filter out citation markers and special characters
                        var text = messageContent.Text;
                        if (!string.IsNullOrEmpty(text))
                        {
                            Console.Write(text);
                        }
                    }
                }
            }
            catch (Exception updateEx)
            {
                logger.LogWarning(updateEx, "Error processing streaming update");
            }
        }

        // Output references (citations)
        await selectedAgent.GetCitationSourcesAsync();
    }
}
catch (Exception ex)
{
    logger.LogError(ex, "An error occurred: {Message}", ex.Message);
}

logger.LogInformation("Application completed");
await host.StopAsync();

Want to switch between regular Bing Search and Custom Search? Just select your agent after running the app and ask a question!

The Power of Custom Search Configuration

The AgentService implementation showcases the flexibility of custom search configuration:

var customConfiguration = new BingCustomSearchConfiguration(connection.Id, tool.ConfigurationName)
{
    Count = 5,              // Set the number of results  
    SetLang = "en",         // Set the language  
    Market = "en-us"        // Set the market  
};

The second parameter references the configuration name created in Azure (“sites”). Multiple configurations within the same resource enable diverse search strategies:

  • “sites”: for general web content
  • “docs”: for official documentation
  • “community”: for forums and community content
  • “news”: for recent announcements

Each configuration maintains completely different search scopes, enabling highly specialized agent creation.

What’s Next?

This architecture opens up so many possibilities. I’m currently experimenting with:

  • Federated search: Searching across multiple custom configurations simultaneously
  • Dynamic configuration: Automatically adjusting search parameters based on user behavior
  • Semantic enhancement: Using vector embeddings to improve search relevance before hitting Bing

The combination of Azure OpenAI Agents with Bing Custom Search, built on this flexible architecture, provides exceptional capabilities for specialized AI assistant development. Success depends on establishing a solid foundation that can evolve with changing requirements.

Complete Source Code

Find the complete source code for all examples in this post on GitHub: Custom Search Agent Repository

How useful was this post?

Click on a star to rate it!

Average rating 5 / 5. Vote count: 1

No votes so far! Be the first to rate this post.

Published by Ramin Ahmadi