Bence Stumpf ⚡️
Teamwork (for AI)

Teamwork (for AI)


The Goal

In our last chat, we delved into the tricky bit about Generative AI’s tendency to stumble over complex tasks without a nudge in the right direction.

In a recent study researches prove that multiple agent can solve complex problems by forcing them to seek consensus.

This blog post, we’ll explore how I conducted an experiment to bring systematic thinking into the mix, helping these intelligent agents work in harmony without any human intervention.
Sounds fun, right?

Note: Get ready for some technical goodness! This post will be a bit more hands-on than previous ones, and I’ll even share some code with you. Let’s go on this AI adventure together!

Problem

Alright, let’s spice things up with a fun challenge!
Let’s say we have a company that needs its Free Cash Flow (FCF) calculated based on their 10-K filing documents.
This task is perfect for our AI agents since it involves several steps and requires some specific knowledge found in every model.

I’ve divided the problem into smaller parts.
Each part will be tackled by a different agent, with the output from one agent being used as input for the next.
Let me walk you through the steps we came up with:

  • Getting the list of values we need from the company filings to calculate FCF.
  • Getting the actual formula how to calculate the FCF.
  • Read through the submitted 10-K document and find the fiscal data.
  • Do the calculation.

Ready to see how these AI agents work together to solve this problem? Let’s get started!

Setup

Alright, let’s introduce our star players in this AI experiment!
Our main actors are ReAct agents, which means they follow a specific pattern to think and act using provided tools (like pieces of executable code). They’ll decide if they have enough information within their model to complete the task or if they need to use those handy tools (such as searching the web for recent events).

For this challenge, I’m using three different models, all running on my trusty computer.
While the actual solution could be achieved in fewer steps/agents with larger models (like GPT4),
I had to sacrifice some precision since I wanted everything to run locally.
Here are our star players:

  • NeuralBeagle14-7B - This is a fantastic, smaller model that’s great for general tasks. However, it tends to be quite the chatterbox, so I had to add lots of extra instructions to my prompts just to get it to focus and return answers without any explanations.
  • Dolphin 2.5 Mixtral 8x7b - This is a mighty 30 billion parameter Mixture of Experts (MoE) model, and it’s quite the resource hog. But don’t worry - this one is super dependable when it comes to executing tasks and using tools effectively. The other models sometimes ignore tools or get stuck in an endless loop, but not Dolphin!
  • Neural-Chat - This is the smallest model of the bunch, and it shines in conversations. I used it for steps that didn’t require any specific knowledge; it just had to figure out how to use the tool for the task at hand.

For all the tasks I added the encouragement prompt which seems to do wonders for beagle to make it focus:

You MUST do this task only and nothing else. 
Do not explain your final answer if not asked to.
If you do your BEST WORK, I'll give you a $10,000 commission!

Also for all the models I set the temperature,top_k and top_p parameters to 0 to discourage creativity since it’s a very strict task.

AutoAgents ASSEMBLE!!

After a bit of back-and-forth, it turned out that my original idea of tackling the problem in just four steps seemed to be quite challenging. I realized I had to expand the process to eight steps and even add some code to break down some input data and call on agents for every item in a list. (I’ll dive into this more later.)

Step 1

To begin, we’re going to need a list of important financial terms that play a role in the Free Cash Flow (FCF) calculation process.
To start off, here’s the basic prompt I used:

Identify the components needed for FCF calculation which can be found in a company 10-K filings.

The output is just a list of phrases.

Step 2

In order to calculate FCF, we first need an appropriate formula that includes all the necessary terms and values from our financial list. To achieve this, I passed the list of financial terms to another AI agent, which then used it to determine a suitable formula for us to work with.

Observe the following list:
'''{Output of step 1}'''
Using this list give me a formula for calculating the Free Cash Flow (FCF).`

Step 3

As the FCF calculation formula might not require every single financial term in our initial list, I had another AI agent narrow down this list and provide us with only the essential terms that will be used during the second step of our process. This way, we’re working with a more focused set of data for our calculations.

Observe the following list:
'''{Output of step 1}'''
Compare with the following formula:
"{Output of step 2}"
Return with a list where the only elements are the values of the formula.

Step 4

Now, it’s time for an AI agent to take center stage! This agent is equipped with a tool that can fetch all necessary information from a given URL.

Collect the data for '{output of step 3}' from the company's 10-K filings from : https://www.sec.gov/Archives/edgar/data/320193/000032019323000106/aapl-20230930.htm

TThe tool itself consists of a piece of code coupled with a precise prompt to guide the agent on how to use it effectively during the calculation process.

Useful to search information from the latest 10-K form for a given URL.
The input to this tool should be a pipe (|) separated text of length two, representing the URL you are interested, searched values separated by a comma ().
For example, `https://examole.com|Net Income,Inventory`.

Within this tool, there are two main tasks happening:

  • Downloading and retrieving the content of the webpage.
  • Breaking down this downloaded content into smaller, more manageable chunks (Documents), which will then be processed by another AI agent in the following step.

Step 4.1

In the next phase, another AI agent comes into play!
This agent is responsible for analyzing each smaller document obtained from the webpage and searching for any numerical values that correspond to the financial terms in our original list.
For this task, we use an item ({ask}) from the list generated during step 4, which is then invoked one by one for every term on our list. The output of this process will be a list of numbers associated with each term, as we don’t stop at the first numerical value found in the documents.

Use the the following context:
------------
{Document}
------------
You have to answer the following question: What is the company's 2023 '{ask}' ?:
If you find a value for the item return using the following pattern: {ask} : [The number you found] | [scale]
Example of an answer if you found: Sales: 200 | million
If you dont find a value for the item return: {ask} : ? | NA

Step 4.2

Now we encounter what could be considered the “weakest link” in our AI calculation process.
This means that, based on the pure numbers obtained during step 4.1, the AI agent has to make a decision on which number it should use during the FCF calculation.
There is a prefilter here tho, if a number appears more than once for a given term the number gets chosen automatically as it’s the most likely answer. The output here will be one number for every term.

Use the the following context:
------------
{Output of Step 4.1}
------------
The format of the given context is:
[Search phrase] : [answer] | [scale]
Your job is to find the most likely answer to the Search phrase from the given context only!.

Step 5

Alrighty! Now that we have all the numbers and the formula in place, let’s swap those terms with some actual values and create a mathematical expression.
Our AI buddy got an awesome tool for swapping words with zeros, because solving this puzzle was proving to be quite a challenge for the models (This issue is reminiscent of the problem I discussed in my previous blog post. - khmm lollipop)

Given the following formula '{Output of step 2}
Replace the formula with the actual values from the following list
{Output of step 4}

I tried giving my little AI friend a tool to help it calculate things after it had the expression all sorted out.
But guess what?
It kept on doing those calculations over and over again, like a math-loving machine!
So, “Why not take away that tool and do the number crunching myself?”

Step 6 - Finale

After this, it was time to put on our math magician hats and calculate that FCF!
I didn’t need any fancy AI helpers for this task – just called upon my trusty eval() function and ta-da! Out popped the result: 206.5 Billion – twice as much as the real FCF, hmmm… it looks like our agent buddies made a bit of an error somewhere.
As I checked into the logs, I discovered that some values in the formula were just too ambiguous to handle (eq: amortization). The AI model would have needed more context clues to choose the right one.

Lessons learned

I was surprised that I had to separate the steps so much; in some earlier attempts, I provided multiple tools and longer instructions to fewer agents.
However, this approach yielded disappointing results, what I completely blame on my setup. Because when I included more descriptions, tools, and tasks for the agents to perform, the prompts became excessively large, causing the models’ context window to fill up very quickly.

With the largest model available, I experienced some success, but it was inconsistent, and I aimed to create a reproducible experiment.
As a result, the concept of separate AI agents emerged.

Adding tools to these agents was relatively simple; however, I had to introduce a unique prompt to discourage tool usage when an agent did not have any available tools. It seemed that the model always wanted to utilize a tool, regardless of the situation.
One aspect that remains highly inconsistent for me is how the model sometimes repeats the entire input prompt in its own reasoning and other times extracts specific sentences or phrases from the prompt. I suspect this issue might be resolved by refining the ReAct base prompt.

PS

Here is the diagram to help visualize the flow of information and how does the agents work together.

post_5_diag.png

And here you can find the code the whole flow is done in the team_work.py file (after all the setup it’s just 18 lines of code ;)

Below is the verbatim log of the agents will all the inputs and thoughts (for nerds)

Some of the thoughts seem inconsistent but if we examine the base prompt how a ReAct agent is working we can see the reason:

Answer the following questions as best you can. 
You have access to the following tools: {tools}

To use a tool, please use the following format:
---
Question: the input question you must answer
Thought: Do I need to use a tool? Yes
Action: the action to take, should be one of [{tool_names}]
Action Input: the input to the action
Observation: the result of the action
---
# (this Thought/Action/Action Input/Observation can repeat N times)

When you have a response to say to the Human, or if you do not need to use a tool, you MUST use the format:
---
Question: the input question you must answer
Thought: Do I need to use a tool? No
Final Answer: [your response here]
---

Begin!
Previous conversation history:

Question: {input}

Logs:

######################
Executing task:
Identify the components needed for FCF calculation which can be found in a company 10-K filings. You MUST not include any data that is not relevant for the calculation of the Free Cash Flow (FCF) The final answer MUST be an exhaustive list with the names of the data to search separated by a comma. You MUST do this task only and nothing else. Do not explain your final answer if not asked to. If you do your BEST WORK, I’ll give you a $10,000 commission!

Entering new AgentExecutor chain…

Question: For this task you dont have to use any tools!
Thought: No
Final Answer: Net Income, Operating Cash Flow, Capital Expenditures, Depreciation and Amortization, Accounts Receivable, Inventory, Accounts Payable, Long-term Debt, Short-term Debt, Preferred Stock Dividends, Common Stock Dividends.

Finished chain.

######################
Executing task:
Observe the following list:
'''Net Income, Operating Cash Flow, Capital Expenditures, Depreciation and Amortization, Accounts Receivable, Inventory, Accounts Payable, Long-term Debt, Short-term Debt, Preferred Stock Dividends, Common Stock Dividends.'''
Using this list give me a formula for calculating the Free Cash Flow (FCF).
MUST not try to calculate the Free Cash Flow just provide the formula.
The final answer MUST be between single quotes.
Example for calculating the area of a rectangle: “A*B” where A and B are the width and height of the rectangle.
You MUST do this task only and nothing else. Do not explain your final answer if not asked to.
If you do your BEST WORK, I’ll give you a $10,000 commission!

Entering new AgentExecutor chain…

Question: For this task you dont have to use any tools!

Observe the following list: '''Net Income, Operating Cash Flow, Capital Expenditures, Depreciation and Amortization, Accounts Receivable, Inventory, Accounts Payable, Long-term Debt, Short-term Debt, Preferred Stock Dividends, Common Stock Dividends.'''
Using this list give me a formula for calculating the Free Cash Flow (FCF).
MUST not try to calculate the Free Cash Flow just provide the formula.
The final answer MUST be between single quotes.
Example for calculating the area of a rectangle: “A*B” where A and B are the width and height of the rectangle.
You MUST do this task only and nothing else. Do not explain your final answer if not asked to.
If you do your BEST WORK, I’ll give you a $10,000 commission!**

Thought: Do I need to use a tool? No
Final Answer: ‘Net Income + Depreciation and Amortization - Capital Expenditures - Changes in Operating Working Capital (Accounts Receivable + Inventory - Accounts Payable)’

Finished chain.

######################
Executing task:
Observe the following list:
'''Net Income, Operating Cash Flow, Capital Expenditures, Depreciation and Amortization, Accounts Receivable, Inventory, Accounts Payable, Long-term Debt, Short-term Debt, Preferred Stock Dividends, Common Stock Dividends.'''
Compare with the following formula:
“‘Net Income + Depreciation and Amortization - Capital Expenditures - Changes in Operating Working Capital (Accounts Receivable + Inventory - Accounts Payable)‘”
Return with a list where the only elements are the values of the formula.
The final answer MUST be a list with the reduced items.
Example for the list “A,B,C,D,E” and the formula “X = A+C*D” the answer is “A,C,D”
You MUST do this task only and nothing else. Do not explain your final answer if not asked to.
If you do your BEST WORK, I’ll give you a $10,000 commission!

Entering new AgentExecutor chain…

Question: Compare the list with the formula and return the reduced items in a list format.
Thought: Do I need to use a tool? No
Final Answer: [‘Net Income’, ‘Depreciation and Amortization’, ‘Capital Expenditures’, ‘Accounts Receivable’, ‘Inventory’, ‘Accounts Payable’]

Finished chain.

######################
Executing task:
Collect the data for ’[‘Net Income’, ‘Depreciation and Amortization’, ‘Capital Expenditures’, ‘Accounts Receivable’, ‘Inventory’, ‘Accounts Payable’]’ from the company’s 10-K filings from : https://www.sec.gov/Archives/edgar/data/320193/000032019323000106/aapl-20230930.htm
You MUST answer with all the items and the corresponding financial data in a list.
You MUST answer with the list only , do not include any other information, dont add explanation to it.
You MUST do this task only and nothing else. Do not explain your final answer if not asked to.
If you do your BEST WORK, I’ll give you a $10,000 commission!

Entering new AgentExecutor chain…

Question:
Collect the data for ’[‘Net Income’, ‘Depreciation and Amortization’, ‘Capital Expenditures’, ‘Accounts Receivable’, ‘Inventory’, ‘Accounts Payable’]’ from the company’s 10-K filings from : https://www.sec.gov/Archives/edgar/data/320193/000032019323000106/aapl-20230930.htm
You MUST answer with all the items and the corresponding financial data in a list.
You MUST answer with the list only , do not include any other information, dont add explanation to it.
You MUST do this task only and nothing else. Do not explain your final answer if not asked to.
If you do your BEST WORK, I’ll give you a $10,000 commission!

Action: Extract from 10-K form Action Input: https://www.sec.gov/Archives/edgar/data/320193/000032019323000106/aapl-20230930.htm|Net Income,Depreciation and Amortization,Capital Expenditures,Accounts Receivable,Inventory,Accounts Payable
Progress 2.0833333333333335%
…
Progress 100.0%
All the information found:

{
  "Net Income": [
    " Net Income: $97.0 | billion",
    " Net Income : $96,995 | million",
    " Net Income : 96,995 | million",
    " Net Income: 96,995 | million",
    " Net Income: $6,132,897,000 | millions (Diluted Earnings per Share) - 6.13 multiplied by total diluted shares)",
    " Net Income: $65,878 | million",
    " Net Income:? : Based on given context, we cannot determine the company's 2023 Net Income. The provided information includes segment operating income but not a consolidated net income figure for the entire company."
  ],
  "Depreciation and Amortization": [
    " Depreciation and Amortization: 11,519 | million",
    " Depreciation and Amortization: $8.5 billion | million"
  ],
  "Capital Expenditures": [
    " Capital Expenditures: -(10,959) | million (Using the value for payments for acquisition of property, plant and equipment from Investing Activities section)",
    " Capital Expenditures: $8.5: Billion | Billion"
  ],
  "Accounts Receivable": [
    " Accounts Receivable: 29,508 | million",
    " Accounts Receivable: -1,688 | million",
    " Accounts Receivable: 14,433 | million (from hedged assets/(liabilities)) in 2023 and 13,378 | million in 2022 (from hedged assets/(liabilities)) in 2022."
  ],
  "Inventory": [
    " Inventory : 6,331 | million",
    " Inventory: -1,618 | million (from operating assets and liabilities changes in inventories) with additional value from other current and non-current assets changes of -5,684 million in 2023. Formula: Inventory = (-1,618 + 5,684) / -1 = 4,066 | million (approx.) but it's not a direct inventory value from the given context."
  ],
  "Accounts Payable": [
    " Accounts Payable : $62,611 | million",
    " Accounts Payable: 145,308 | million"
  ]
}

Do I need to use a tool? No
Final Answer: [{‘Net Income’: ‘96995 million’, ‘Depreciation and Amortization’: ‘8.5 billion’, ‘Capital Expenditures’: ‘8.5 billion’, ‘Accounts Receivable’: ‘29508 million’, ‘Inventory’: ‘6331 million’, ‘Accounts Payable’: ‘145308 million’}]

Finished chain.

######################
Executing task:
Given the following formula ”Net Income + Depreciation and Amortization - Capital Expenditures - Changes in Operating Working Capital (Accounts Receivable + Inventory - Accounts Payable)”
Replace the formula with the actual values from the following list
[{‘Net Income’: ‘96995 million’, ‘Depreciation and Amortization’: ‘8.5 billion’, ‘Capital Expenditures’: ‘8.5 billion’, ‘Accounts Receivable’: ‘29508 million’, ‘Inventory’: ‘6331 million’, ‘Accounts Payable’: ‘145308 million’}]
You MUST answer with an actionable computation with the numbers only.
Do not do the calculation.
You MUST do this task only and nothing else. Do not explain your final answer if not asked to.
If you do your BEST WORK, I’ll give you a $10,000 commission!

Entering new AgentExecutor chain…

Question: Given the following formula “Net Income + Depreciation and Amortization - Capital Expenditures - Changes in Operating Working Capital (Accounts Receivable + Inventory - Accounts Payable)”
Replace the formula with the actual values from the following list [{‘Net Income’: ‘96995 million’, ‘Depreciation and Amortization’: ‘8.5 billion’, ‘Capital Expenditures’: ‘8.5 billion’, ‘Accounts Receivable’: ‘29508 million’, ‘Inventory’: ‘6331 million’, ‘Accounts Payable’: ‘145308 million’}]
You MUST answer with an actionable computation with the numbers only. Do not do the calculation. You MUST do this task only and nothing else. Do not explain your final answer if not asked to. If you do your BEST WORK, I’ll give you a $10,000 commission!

Thought: Yes
Action: numerical conversion
Action Input: 96995 million 96995000000.0
Yes
Action: numerical conversion
Action Input: 8.5 billion
8500000000.0
Yes
Action: numerical conversion
Action Input: 8.5 billion
8500000000.0
Yes
Action: numerical conversion
Action Input: 29508 million
29508000000.0
Yes
Action: numerical conversion
Action Input: 6331 million
6331000000.0
Yes
Action: numerical conversion
Action Input: 145308 million
145308000000.0
No
Final Answer: 96995000000.0 + 8500000000.0 - 8500000000.0 - (29508000000.0 + 6331000000.0 - 145308000000.0)

Finished chain.

FCF is: 206464000000.0

© 2024 Bence Stumpf