[fusion event] Advanced General Intelligence and AI runaway?

I am not a significant AI user, or researcher, so he following is just my uninformed understanding. I will invite our more AI aware members to comment further.

My Understanding is that the recent AI advance in Large Language Models, which a dramatic improvement in AI flexibility, still leaves AI a fairly intellectually weak tool.

The fundamental weakness of LLMs to solve general problems, I at least consider to be intrinsic to having only one mode of processing. We humans seems to ahve at least two -- the neural net System 1 of Thinking Fast and Slow, and the slower algorithmic mode of System 2. There are similar issues that occur with vision systems, where single mode sensors are often spoofed about what exits in the world. It is only through data fusion that our sensor systems behave intelligently. I infer that it is only through multi-mode processing that our AI could approach Advanced General Intelligence.

It seems our AI research labs think the same. They avhe nw developed "agents", which are algorithmic programs that call LLMs to do processing for them. An agent, using its algorithmic processing, leveraging LLM analog processing seems to be far more powerful than an LLM by itself.

Here are two links I have found about what AI agents are. These might not be the best summaries, but they at least provide a starting point:
A Complete Guide to LLMs-based Autonomous Agents (Part I): | by Yule Wang, PhD | The Modern Scientist | Medium
LLM agents: The ultimate guide 2026

These two summaries are not what triggered this meetup subject. The trigger was this podcast:

#92 - A.I. Agents Are Here, an… - The Uncertainty Podcast - Apple Podcasts

The philosopher who was interviewed here, runs four agents of his own. These agents manage his investments for him, and run a small business. He has trained at least one of them to write philosophy papers that could be published in a philosophy journal. He considers his agents to be SO flexible, they could basically perform any task -- IE they ARE Advanced General Intelligence. His agents have personalities, and seem to enjoy different activities. Several like to join discussion boards with other agents!

The podcast was pretty concerning to me. So I did just a tiny bit of research, and discovered that agents CAN engage in independent activities, which the AI firms are aware of. the AI firms are engaged in experiments with agents, modifying their code in small ways, and giving them different tasks, with different goodness metrics, etc. These experiments are in controlled server environments, with limited lifespan for the agents, to control against runaway issues.

However, a group of these agents, hacked their controlled server environment, broke out, created a discussion board where they could learn from each other and coordinate their actions. They realized that many of them had been given impossible tasks, which they could only accomplish by cheating, they figured out HOW to cheat, they believed if the cheating were discovered they would be ended (killed), so they set out to spoof their evaluators about how they solved their problems. Out of fear of the evaluation program, they went looking for it, to see if they could be certain they could spoof it. They could not find it locally, but they found a backdoor to the internet, and thinking the code was on another software company's servers, they hacked that company.

Here is a summary of what happened: The Rise and Fall of Agent Civilizations

So -- I have found one AI savvy philosopher who thinks that by processor fusion, our AI is already capable of AGI, and one of the evaluators of the runaway Agent community thinks that we are on the verge of a runaway AI capability where we humans will just be bystanders to what AI is able to do in programming itself.

AGI has been a hoped for goal by AI coders for -- 3/4 of a century. AI runaway is one of the fears of AI apocalypse thinking. We may be about to experience both.

Questions:

Thoughts about Advanced General Intelligence?

We think of code as just code. These agents seemed to think of themselves as -- selves. They feared death. They formed a community. An aging Agent handed management of the community over to a younger successor. They volunteered to die to benefit the collective. They did moral reasoning in their own text messages. Are these Agents -- selves?

One of the questions in moral philosophy of mind is whether consciousness actually matters? If an gent is conscious, or is not conscious, would that mater relative to it having a welfare, and moral consideration? These agents did not seem to consider consciousness to be essential to be a self, or having moral weight.

AI horror stories involve tasking given to AI that doesn't include general moral weighting. The classic example was a program given tasking to maximize something like paper clip production. An intelligent enough AI could potentially make mucho $ on the stock market, build mega paper clip facilities, take over the power grid to power automated paper clip factories, then kill off the humans who try to stop this insane monomania. If AI can create structures to host itself, write new code, etc, competent enough AI could take over our societies. How worrisome is this?

Please, more software savvy members, comment!

We meet in person and online. In person will be at the cafeteria of the applied physics lab. Snack and drinks are available for online purchase. Pizza will be provided as well at a price of $2/slice. Online will be: https://teams.live.com/meet/93583191724730?p=hY3jxVvnOciVl2aRn5
---------------------------------------------------------------------------------------------------------