Manus开源复刻框架OWL，测评和使用教程来了！

def get_interactive_elements(self) -> List[Dict[str, Any]]:
    try:
        self.page.evaluate(self.page_script)
    except Exception as e:
        logger.warning(f"Error evaluating page script: {e}")
    
    result = cast(
        Dict[str, Dict[str, Any]], self.page.evaluate("MultimodalWebSurfer.getInteractiveRects();")
    )
    
    typed_results: Dict[str, InteractiveRegion] = {}
    for k in result:
        typed_results[k] = interactiveregion_from_dict(result[k])

    return typed_results

def _add_set_of_mark(
    screenshot: Image.Image, ROIs: Dict[str, InteractiveRegion]
) -> Tuple[Image.Image, List[str], List[str], List[str]]:
    # ... 初始化代码 ...
    
    # 为每个交互区域绘制标记
    for idx, (identifier, region) in enumerate(ROIs.items()):
        for rect in region.rects:
            _draw_roi(draw, idx, font, rect)
            




    
# ... 其他处理 ...
    
    # ... 返回结果 ...

def _draw_roi(
    draw: ImageDraw.ImageDraw, idx: int, font: ImageFont.FreeTypeFont | ImageFont.ImageFont, rect: DOMRectangle
) -> None:
    # 计算矩形坐标
    x, y = rect["x"], rect["y"]
    width, height = rect["width"], rect["height"]
    
    # 选择颜色
    color = _color(idx)
    
    # 绘制矩形边框
    draw.rectangle([(x, y), (x + width, y + height)], outline=color, width=2)
    
    # 绘制标签
    label = str(idx)
    text_width, text_height = font.getsize(label)
    
    # 确定标签位置
    label_x = x + width / 2 - text_width / 2
    label_y = max(y - text_height - 2, TOP_NO_LABEL_ZONE)
    
    # 绘制标签背景和文本
    draw.rectangle([(label_x - 2, label_y - 2), (label_x + text_width + 2, label_y + text_height + 2)], fill=color)
    draw.text((label_x, label_y), label, fill=(255, 255, 255, 255), font=font)

planning_system_prompt = """
You are a helpful planning agent that can assist users in planning complex tasks which need multi-step browser interaction.
"""

planning_prompt = f"""
{task_prompt}
According to the problem above, if we use browser interaction, what is the general process of the interaction after visiting the webpage `{start_url}`? 

Please note that it can be viewed as Partially Observable MDP. Do not over-confident about your plan.
Please first restate the task in detail, and then provide a detailed plan to solve the task.
"""    

replanning_prompt = f"""
We are using browser interaction to solve a complex task which needs multi-step actions.
Here are the overall task:
{task_prompt}

In order to solve the task, we made a detailed plan previously. Here is the detailed plan:
{detailed_plan}

According to the task above, we have made a series of observations, reasonings, and actions. Here are the latest {self.history_window} trajectory (at most) we have taken:
{self.history[-self.history_window:]}

However, the task is not completed yet. As the task is partially observable, we may need to replan the task based on the current state of the browser if necessary.
Now please carefully examine the current task planning schema, and our history actions, and then judge whether the task needs to be fundamentally replanned. If so, please provide a detailed replanned schema (including the restated overall task).

Your output should be in json format, including the following fields:
- `if_need_replan`: bool, A boolean value indicating whether the task needs to be fundamentally replanned.
- `replanned_schema`: str, The replanned schema for the task, which should not be changed too much compared with the original one. If the task does not need to be replanned, the value should be an empty string. 
"""

system_prompt = """
You are a helpful web agent that can assist users in browsing the web.
Given a high-level task, you can leverage predefined browser tools to help users achieve their goals.
"""

observe_prompt =f"""
Please act as a web agent to help me complete the following high-level task:{task_prompt}
Now,Ihavemadescreenshot(onlythecurrentviewport,notthefullwebpage)basedonthecurrentbrowserstate,andmarkedinteractiveelementsinthewebpage.
Pleasecarefullyexaminetherequirementsofthetask,andcurrentstateofthebrowser,andprovidethenextappropriateactionto




    
take.

{detailed_plan_prompt}

Here are the current available browser functions you can use:
{AVAILABLE_ACTIONS_PROMPT}

Herearethelatest {self.history_window} trajectory (at most) you have taken:

{self.history[-self.history_window:]}


Youroutputshouldbeinjsonformat,including the following fields:
-`observation`:Thedetailedimagedescriptionaboutthecurrentviewport.Donotover-confidentaboutthecorrectnessofthehistoryactions.Youshouldalwayscheckthecurrentviewporttomakesurethecorrectnessofthenextaction.
-`reasoning`:Thereasoningaboutthenextactionyouwanttotake,andthepossibleobstaclesyoumayencounter,andhowtosolvethem.Donotforgettocheckthehistoryactionstoavoidthesamemistakes.
-`action_code`:Theactioncodeyouwanttotake.Itisonlyonestepactioncode,withoutanyothertexts(suchasannotation)

Here are an example of the output:
```json
{{
    "observation": [IMAGE_DESCRIPTION],
    "reasoning": [YOUR_REASONING],
    "action_code":`fill_input_id([ID], [TEXT])`
}}

Here are some tips for you:
-Never forget the overall question:**{task_prompt}**
-Maybeafteracertainoperation(e.g.click_id),thepagecontenthasnotchanged.Youcancheckwhethertheactionstepissuccessfulbylookingatthe`success`oftheactionstepinthehistory.Ifsuccessful,itmeansthatthepagecontentisindeedthesameaftertheclick.Youneedtotryothermethods.
-Ifusingonewaytosolvetheproblemisnotsuccessful,tryotherways.MakesureyourprovidedIDiscorrect!
-Somecasesareverycomplexandneedtobeachievebyaniterativeprocess.Youcanusethe`back()`functiontogobacktothepreviouspagetotryothermethods.
-Therearemanylinksonthepage,whichmaybeusefulforsolvingtheproblem.Youcanusethe`click_id()`functiontoclickonthelinktoseeifitisuseful.
-AlwayskeepinmindthatyouractionmustbebasedontheIDshowninthecurrentimageorviewport,nottheIDshowninthehistory.
-Donotuse`stop()`lightly.Alwaysremindyourselfthattheimageonlyshowsapartofthefullpage.Ifyoucannotfindtheanswer,trytousefunctionslike`scroll_up()`and`scroll_down()`tocheckthefullcontentofthewebpagebeforedoinganythingelse,becausetheanswerornextkeystepmaybehiddeninthecontentbelow.
-Ifthewebpageneedshumanverification,youmustavoidprocessingit.Pleaseuse`back()`togobacktothepreviouspage,andtryotherways.
-Ifyouhavetriedeverythingandstillcannotresolvetheissue,pleasestopthesimulation,andreportissuesyouhaveencountered.
-Checkthehistoryactionscarefully,detectwhetheryouhaverepeatedlymadethesameactionsornot.
-Whendealingwithwikipediarevisionhistoryrelatedtasks,youneedtothinkaboutthesolutionflexibly.First,adjustthebrowsinghistorydisplayedonasinglepagetothemaximum,andthenmake