OpenAI Agents SDK

04/08/2025

• OpenAI Agents SDK
  • Agents are the core building block in your apps. An agent is a large language model (LLM), configured with instructions and tools. The most common properties of an agent you'll configure are:
    • instructions: also known as a developer message or system prompt.
    • model: which LLM to use, and optional model_settings to configure model tuning parameters like temperature, top_p, etc.
    • tools: Tools that the agent can use to achieve its tasks. @function_tool decorator is used to define a tool.
  
  
  • By default, agents produce plain text (i.e. str) outputs. If you want the agent to produce a particular type of output, you can use the output_type parameter. When you pass an output_type, that tells the model to use structured outputs instead of regular plain text responses.

    
        from pydantic import BaseModel
        from agents import Agent
    
    
        class CalendarEvent(BaseModel):
            name: str
            date: str
            participants: list[str]
    
        agent = Agent(
            name="Calendar extractor",
            instructions="Extract calendar events from text",
            output_type=CalendarEvent,
        )
    

  
  
  • Supplying a list of tools doesn't always mean the LLM will use a tool. You can force tool use by setting ModelSettings.tool_choice. Valid values are:
    • auto, which allows the LLM to decide whether or not to use a tool.
    • required, which requires the LLM to use a tool (but it can intelligently decide which tool).
    • none, which requires the LLM to not use a tool.


• Segment Selector: AI-assisted polygon segment selection
  • This is a PoC that explores the possibilities of AI-assisted geometric tasks using OpenAI Agents SDK. The things to check are:
    • Parameterization of geometric context (prompt → parameters)
    • How AI can understand and process geometric context
    • Methods for providing geometric context to AI systems
    • AI's ability to make decisions based on geometric context
  
  
  • The role of the agent is to select segments from a polygon based on the user's request like "select the top left segment". The user's request is given as a natural language prompt, and the agent will select the segments based on the prompt.
  
  
  • Direction Context: Given the user's request, the agent will select the directions and segments from the polygon based on the similarity between the direction vectors.

    
        class SegmentSelectorConfiguration:
            DIRECTION_VECTORS = {
                "right": (1, 0),
                "left": (-1, 0),
                "top": (0, 1),
                "bottom": (0, -1),
                "right_top": (1, 1),
                "left_top": (-1, 1),
                "left_bottom": (-1, -1),
                "right_bottom": (1, -1),
                "right_up": (0.866, 0.5),
                "up_right": (0.5, 0.866),
                "up_left": (-0.5, 0.866),
                "left_up": (-0.866, 0.5),
                "left_down": (-0.866, -0.5),
                "down_left": (-0.5, -0.866),
                "down_right": (0.5, -0.866),
                "right_down": (0.866, -0.5),
            }
    
            # normalize direction vectors
            for key, value in DIRECTION_VECTORS.items():
                DIRECTION_VECTORS[key] = np.array(value) / np.linalg.norm(np.array(value))
    
            assert np.allclose(np.linalg.norm(list(DIRECTION_VECTORS.values()), axis=1), 1.0)
    
    
        class SegmentSelector:
            def _get_segment_similarities(
                self,
                target_vector: Tuple[float, float],
                similarity_threshold: float,
            ) -> List[Tuple[int, float]]:
                segments_vectors = self._compute_segments_vectors(self.polygon_segments)
    
                similarities = []
                for idx, vector in enumerate(segments_vectors):
                    similarity = np.dot(vector, target_vector)
                    mask = similarity >= similarity_threshold
                    if mask.sum().item() >= len(vector) // SegmentSelectorConfiguration.MASK_MATCHING_DIVIDER:
                        similarities.append((idx, similarity[mask].sum()))
    
                return similarities
    
            def _select_segments_by_vector(
                self,
                target_vector: Tuple[float, float],
                n_per_direction: int,
                similarity_threshold: float,
            ) -> List[int]:
                similarities = self._get_segment_similarities(
                    target_vector,
                    similarity_threshold,
                )
    
                similarities.sort(key=lambda x: x[1], reverse=True)
    
                sorted_indices = [idx for idx, _ in similarities]
    
                return sorted_indices[:n_per_direction]
    
            @staticmethod
            @function_tool
            def get_target_vector(direction: str) -> List[float]:
                """Get the vector for a given direction"""
                try:
                    return SegmentSelectorConfiguration.DIRECTION_VECTORS[direction].tolist()
    
                except:
                    traceback.print_exc()
                    raise Exception
    

  
  
  • Visualization of the agent's behavior

    

    From the left,
    f"Select all segments in the right bottom of the following polygon: {BOUNDARY_COORDS}",
    f"Select the top left segments of the following polygon: {BOUNDARY_COORDS}",
    f"Select 4 segments in the right and left of the following polygon: {BOUNDARY_COORDS}",
    f"Select randomly 3 segments of the following polygon: {BOUNDARY_COORDS}",
    
    

    

    From the left,
    f"Select all segments of the following polygon: {BOUNDARY_COORDS}",
    f"Select any one segment of the following polygon: {BOUNDARY_COORDS}",
    f"Select 2 segments in the right and left of the following polygon: {BOUNDARY_COORDS}",
    f"Select randomly half of the segments of the following polygon: {BOUNDARY_COORDS}",
    
    

    

    From the left,
    f"Select index 3 segment of the following polygon: {BOUNDARY_COORDS}",
    f"Select indices 3, 5, 6, 1 segment of the following polygon: {BOUNDARY_COORDS}",
    f"Select segments with odd indices of the following polygon: {BOUNDARY_COORDS}",
    f"Select indices any 3 segments from the indices 0, 1, 2, 3, 4 of the following polygon: {BOUNDARY_COORDS}",
    
    

    
        import os
        import sys
        import pytest
        
        from agents import Agent, Runner
    
        sys.path.append(os.path.abspath(os.path.join(__file__, "../../src")))
    
        from src.testsets import TestSets
        from src.selector import SegmentSelector, SegmentSelectorConfiguration, SegmentSelectorOutput
    
        
        @pytest.fixture(scope="module")
        def agent():
            return Agent(
                name=SegmentSelectorConfiguration.NAME,
                model=SegmentSelectorConfiguration.MODEL_NAME,
                model_settings=SegmentSelectorConfiguration.MODEL_SETTINGS,
                instructions=SegmentSelectorConfiguration.INSTRUCTIONS,
                output_type=SegmentSelectorOutput,
                tools=SegmentSelector.tools(),
            )
            
        @pytest.fixture(scope="module")
        def testsets():
            return TestSets()
        
        def test_a(agent, testsets):
            testcase_a = testsets.TestcaseA
            
            # f"Select all segments in the right bottom of the following polygon: {BOUNDARY_COORDS}"
            request = testcase_a.TEXTS[0]
            response = Runner.run_sync(agent, request)
            assert 8 in response.final_output.selected_indices
            assert 7 in response.final_output.selected_indices
            assert 1 not in response.final_output.selected_indices
            assert 2 not in response.final_output.selected_indices
        
            # f"Select the top left segments of the following polygon: {BOUNDARY_COORDS}"
            request = testcase_a.TEXTS[1]
            response = Runner.run_sync(agent, request)
            assert 1 in response.final_output.selected_indices
            assert 2 in response.final_output.selected_indices
            assert 8 not in response.final_output.selected_indices
            assert 7 not in response.final_output.selected_indices
        
        
            # f"Select 4 segments in the right and left of the following polygon: {BOUNDARY_COORDS}"
            request = testcase_a.TEXTS[2]
            response = Runner.run_sync(agent, request)
            assert 2 in response.final_output.selected_indices
            assert 8 in response.final_output.selected_indices
            assert len(response.final_output.selected_indices) == 4
        
            # f"Select randomly 3 segments of the following polygon: {BOUNDARY_COORDS}"
            request = testcase_a.TEXTS[3]
            response = Runner.run_sync(agent, request)
            assert len(response.final_output.selected_indices) == 3
        
        def test_b(agent, testsets):
            testcase_b = testsets.TestcaseB
        
            # f"Select all segments of the following polygon: {BOUNDARY_COORDS}"
            request = testcase_b.TEXTS[0]
            response = Runner.run_sync(agent, request)
            assert len(response.final_output.selected_indices) == 10
        
            # f"Select any one segment of the following polygon: {BOUNDARY_COORDS}"
            request = testcase_b.TEXTS[1]
            response = Runner.run_sync(agent, request)
            assert len(response.final_output.selected_indices) == 1
            
            # f"Select 2 segments in the right and left of the following polygon: {BOUNDARY_COORDS}"
            request = testcase_b.TEXTS[2]
            response = Runner.run_sync(agent, request)
            assert len(response.final_output.selected_indices) == 2
            assert (
                0 in response.final_output.selected_indices
                or 1 in response.final_output.selected_indices
            )
            assert (
                5 in response.final_output.selected_indices
                or 6 in response.final_output.selected_indices
            )
            
            # f"Select randomly half of the segments of the following polygon: {BOUNDARY_COORDS}"
            request = testcase_b.TEXTS[3]
            response = Runner.run_sync(agent, request)
            assert len(response.final_output.selected_indices) == 5
            
        def test_c(agent, testsets):
            testcase_c = testsets.TestcaseC
            
            # f"Select index 3 segment of the following polygon: {BOUNDARY_COORDS}"
            request = testcase_c.TEXTS[0]
            response = Runner.run_sync(agent, request)
            assert len(response.final_output.selected_indices) == 1
            assert response.final_output.selected_indices[0] == 3
            
            # f"Select indices 3, 5, 6, 1 segment of the following polygon: {BOUNDARY_COORDS}"
            request = testcase_c.TEXTS[1]
            response = Runner.run_sync(agent, request)
            assert len(response.final_output.selected_indices) == 4
            assert sorted(response.final_output.selected_indices) == sorted([3, 5, 6, 1])
            
            # f"Select segments with odd indices of the following polygon: {BOUNDARY_COORDS}"
            request = testcase_c.TEXTS[2]
            response = Runner.run_sync(agent, request)
            assert all(i % 2 == 1 for i in response.final_output.selected_indices)
            
            # f"Select indices any 3 segments from the indices 0, 1, 2, 3, 4 of the following polygon: {BOUNDARY_COORDS}"
            request = testcase_c.TEXTS[3]
            response = Runner.run_sync(agent, request)
            assert all(i in [0, 1, 2, 3, 4] for i in response.final_output.selected_indices)
    

---