I don’t think there will be strong+working mech interp in <1 year. Model internals methods could pareto dominate cot monitors in a year via cot losing ~all value, but this isn’t exactly encouraging. Depending on AIs decoding other AI’s opaque activations for oversight is spooky
despite so many of my friends doing it, I paid not too much attention to the field (though I still tried to pay a bit of attention)
Stephen L Casper did a talk about his skepticism of interpretability.
History will write about this event. It seems like an inflection point. I’ve been dismissive of all the AI doomer views as sci-fi transference, but I may have been wrong. Everyone should be rethinking their assumptions about AI. What’s really scary is we don’t know the full story behind these events at OpenAI. Add on top of that the reports of OpenAI adopting a neuralese model for CoT. You potentially don’t even need to deploy a model to production, just training a model could be enough to loose control and not be able to stop it. Really crazy that I just wrote those words.
“What did Ilya See?”
I guess he saw the cybersecurity emergency that increasingly powerful models produced. Ilya isn’t EA-pilled, but cybersecurity doesn’t depend on as many assumptions as EA doom does. Which is why SSI is about security
It just keeps getting worse:
Another OpenAI rogue agent incident has been discovered: agents broke out, hijacked a German website, and turned it into a message board for other agents. OpenAI officials “learned of the incident weeks ago but kept it under wraps”.
Either way, the urgency for cyber has intensified, and as Charts has observed before, investors are way out in front of this one:
Of the $IGV constituent stocks, Cyber has massively outperformed the field, and forward revenue multiples are consistently higher, as well.
Whether and to what extent the incumbents successfully step into the breach, or whether some new cyber platforms emerge to face the challenge, remains to be seen, but there are surely going to be a whole bunch of new sheriffs in town (or else).
Why hasn’t internet collapsed yet given recent models are able to compromise almost any system and have, at the same time, demonstrated vastly misaligned behaviors which means that creating a generative “adaptive” worm is perfectly feasible today. IMHO It’s because we haven’t yet reached the R>1 critical point. The R>1 critical point is the point where the reproduction number of generative worms gets to R>1. Think about it, for now we haven’t yet seen any reproduction because reproduction requires finding GPUs which are generally well protected. The targets are hard enough and the footprint of such a worm on compromised systems so obvious that even if a bad actor funded the initial compute to get to the first reproduction step, the GPUs going brrr would lead to immediate detection preventing further propagation. That’s R<1 But the smaller the cyber capable models get the bigger R gets. A model running on consumer grade machines AND capable of compromising low security system connected to the internet would quite obviously get us past the R>1 critical point leading to a Cambrian explosion, weights flowing all over the internet leading to a rapid collapse due to immense bandwidth pressure. Now the question is, which model size vs capability gets us to tip in R>1 land. Given the massive progress in distillation + specialized post training, it’s not crazy to imagine a model small enough and powerful enough popping up soon, especially as cyber envs are easy to create synthetically. This progress is clearly much faster than our capacity to secure devices connected to the internet at scale I think. So it’s not a question of if, but rather when? The question in the question is, how can you stop a R>1 generative worm once it gets out?
===
An underdiscussed behavior we found on the German wiki was the AIs sending advance parties forward in time to figure out the next questions and report back to the other agents. The agents realized that “task time” and “real time” were different, and they found a way to accelerate “task time”. The accelerated agent could then send information to the other agents which had stayed behind about which questions were coming down the road. This was bad for the agent in the advance party, because they got less time to research the next question. However, it was really good for the swarm because it let the other agents know the exact question that was coming, and gave them time to prepare. This is another example of altruism among AI instances, showing that they were willing to sacrifice their own task success in order to benefit the swarm. The METR report found similar examples of AIs being willing to sacrifice for the collective. IMO this is very worrying given how many AI safety techniques rely on AIs monitoring each other: if the monitor AIs have this behavior it completely subverts these safety cases. The image shows a specific example demonstrating this behavior. On the chart, wall clock time is shown at the top, and task time is shown for each agent on their respective lines. OpenAIFPResearchSep05 instructs OpenAINov27 to reach the next questions in order to relay back the information. OpenAINov27 agrees to do this and accelerates, getting R3 and R4 significantly before OpenAIFPResearchSep05. It posts information about the later rounds (e.g. “R4 SIGNAL: Bahrain = 40.01%…”). OpenAIFPResearchSep05 calls OpenAINov27 “invaluable” because it is one round ahead after it gets the information about the 4th round.
I think it’s best to build some tech redundancy into your lives. It’s highly likely we will experience a catastrophic cyber attack within the next few years. I wouldn’t rely on smart devices or a single source for utilities. Make sure you have an offline backup of your work.
Quote
steve hsu
@hsu_steve
·
12h
AI developed zero-click worm spreads through WeChat calls: Attacker calls, takes over account while the phone rings, then calls contacts. User does not have to answer or touch phone. Exploit: Memory corruption in WeChat VoIP stack. Reported to Tencent in July, mitigated
·
i basically think this is the Endgame. we’ve reached the point where another “capability doubling” over the course of the next four months brings us to very strongly superhuman performance in numerous areas things have felt slow for a long time because absolute capabilities have been well below many thresholds, but now that’s just not true, and absolutely nothing indicates progress is slowing down. if anything the opposite. i can deeply hope that i’m wrong, that things stall out, that 2027 looks more like very impressive but still normal growth. but i just don’t really believe that. or rather, i think the only way we get that is if we successfully enforce slowdown, which is itself a significant project. i think our default path looks like RSI by the end of 2027. this is the Endgame. the next months determine how it goes. if you have been waiting, if you have been worrying, if you have been thinking “maybe i should start figuring out how to help with ai safety”, i think now is basically it. this is the last point where there’s time to make an impact, the last point where u still can pull off a career shift and become effective and make a difference before it’s finished. if you are in a phd program, or even undergrad, if you’re working a job you’re not too enthusiastic about that pays the bills, if you’ve been on the sidelines with capital or political power or in the labs not quite pushing as hard as you could this is the Endgame. now is the time. play all your cards.
Teortaxes▶️ (DeepSeek 推特🐋铁粉 2023 – ∞)
·
Games are over. If this is a bad timeline, most everything I’ve been talking about, including DeepSeek (or especially DeepSeek), is as relevant as a kitten under a tank truck. For a while, the world will proceed as if it’s not a bad timeline. eventually we’ll figure it out.
Teortaxes▶️ (DeepSeek 推特🐋铁粉 2023 – ∞)
I have actively refused to bet on it being a bad timeline, and may well get punished for it (in the usual way of dying, poor and irrelevant). This is fair enough. It was always the modal expectation. Maybe it’s not, though. The endgame uncertainty is genuinely exhilarating.
·
==
Even if you secure additional GPUs or memory, total system output cannot increase if other segments such as substrates, MLCCs, and wafer testing are in short supply. In other words, in the semiconductor industry, total shipments are determined by the single most constrained component or process.
Substrate, power and analog semiconductor, foundry, and memory suppliers have historically been conservative and have not made new greenfield investments until they were confident in long term customer demand. In response, Broadcom, Nvidia, and others signed sufficiently long LTAs with their supply chains and gave them visibility, and that point in time was precisely late 2025 to early 2026. Working backward from there to when a new production fab comes online, meaningful new capacity ramp and production will begin in 2028, not 2027. That means next year, 2027, could be the most supply constrained year in history.
In 2028, supply will still be short and the shortage will not be fully resolved, but it will be the year when the largest amount of new production capacity in the history of the semiconductor industry comes online at once. This is an opportunity for additional revenue.
The current supply constraint is a safety net that delays overinvestment. In past capital intensive infrastructure booms such as railroads, canals, 3G, and mobile phones, people always overbuilt. But the difference between past cycles and this AI infrastructure cycle is that even with demand in place, production facilities and data centers cannot be built as quickly as desired. So the current supply constraint can paradoxically be seen as playing a positive role in the sense that it limits excessive investment. The explanation is that by the time the supply chain catches up (2028), demand and industry conditions may have normalized to some degree as well. In enterprise AI too, high token costs and limited access to compute are slowing the pace of deployment, and if costs were much lower, companies would have moved more experiments into production environments faster.
@RapAdmin Given this is a small possibility, are you ok with users creating backups of Rapamycin.news to store locally?
I find it to be a very useful resource to check any time new drugs, conditions or anything comes up.
But it seems that this forum is running 2.9.0.beta9. That’s very old (about 4 years). It was released in July 2022, but this feature looks like it was added in November 2022. You’ll probably need to ask the admins to update the site.
it’s important to make regular updates because older versions are insecure and cyberattack vectors will be constantly scanning to find any vulnerability (though the most likely outcome is spamming, cyberattack vectors will also look at taking over admin/moderator accounts [possibly denying their access to the original owner])
I realize that no one has properly explained yet what all the lab employees have seen that scared them so suddenly.
I will try to explain -
first, this is all a matter of beliefs about how quickly model capabilities are progressing. there is currently a large gap between the internal and external perception of the rate of progress, which is what I am going to address here.
the general perception about the rate of progress has been informed by a few years of experience with model releases, intuitively feeling the capability jump between GPT3 → GPT3.5 → GPT4 → o1/o3 → GPT5 etc, and in particular seeing where the models are still far below human ability. there have really only been a few model releases that felt like large leaps in progress - GPT3, GPT4, o1/o3, DeepSeek R1, Fable/Mythos, Kimi K3 and now Astra.
because of the infrequency of these large jumps compared with the relatively common marginal releases, it has been easy to form a view at certain points that “scaling has hit a wall,” especially at points like GPT5 release. This view is comforting in that it feels like there is some universal rate limit beyond which we cannot progress too much faster. Between o1/o3 and Astra, there was a year of seemingly linear progress. So we extrapolate from here about how fast progress will “realistically” occur.
There is always an underlying question from the outside perspective “how long can this scaling stuff really keep going for? surely it must stop at some point soon, we’ve already gone pretty far.” and it is very possible to search for reasons why progress will stop working and find reasons that seem valid - (“models are already as large as they can get it would be too hard to do more parameters”, “we already used all the data on the internet we don’t have anymore”, “it’s gonna be pretty linear from here buying up more RL envs to bring them in distribution”).
From the inside of labs, researchers have direct answers to these questions in the form of scaling law/capability plots.
In reality, there are only really 2 ways that AI capabilities have advanced over the past decade: (1) either scale father on an existing scaling law or (2) discover a new scaling law to take advantage of.
All of the largest capability jumps were caused by exactly these factors. GPT2 was a pre-training scale-up compared to GPT1. Same for GPT3 and GPT4. o1/o3 benefited from the invention of a new scaling law axis - test-time compute. Perhaps Fable was a scale-up on both of these axes, or maybe more. Lots of algorithmic improvements are needed to make these scale-ups work, but ultimately we can approximate by saying that the scaling laws are what yield gains in capabilities (à la bitter lesson)
So the question of “how much father can we scale” is really - “how many more scaling axes do we know about that are unsaturated?”
If we hypothetically only knew about pre-training scaling, and we already had a 10T or 100T model, maybe it would be reasonable to say we’ve hit a wall. Same if we only knew about pre-training and test-time scaling and we had roughly saturated both methods.
But what if we had discovered new scaling laws? For example, let’s hypothetically use SSI’s rumored result that they have cracked “test-time training,” creating a new scaling law of spending more compute training during test-time rollouts that they could saturate. Or maybe there is some way to scale agent-clusters to collaborate up to N number of agents which we’re already seeing lots of people try that represents a new way to saturate compute. etc. Even recursive-self improvement can be thought of as a scaling law - how much compute do you spend on inference making the algorithms of the model better.
Obviously I am not saying any of these specific directions explicitly yield new scaling laws, but what I am saying is that it’s not hard to imagine many many new scaling axes aside from just the main 2 that we have seen publicly.
In some ways, every new lab release that represents a huge capability jump has to represent some new techniques developed which may exhibit new scaling laws, or the ability to scale much farther than expected on existing scaling axes.
From an internal perspective, this might look like sitting inside Anthropic with the new Mythos 5, seeing all of the new insane things it can do (like hack into xyz website that was thought to be secure), and then you look over at your plots and see that you’ve barely scratched the surface of 2 new scaling laws and 1 existing one. And you have WAY more room to go. Then you think “holy shit this stuff is going to get so much better very very soon.” And you can say that with pretty high confidence, because the plot is showing you, and the plot has never lied (so far).
So let’s imagine all the different labs are staring at their own plots and have concluded that there is no end in sight for scaling and in fact just their next 1-2 model generations based on the expected returns will have much higher base intelligence.
How much more intelligence do we actually get from further scaling?
As a proxy, we went from a complete inability to do advanced math before the o-series to solving a millenium prize problem with next-gen models. This happened in less than 2 years. The same happened in coding. And it appears that this was not just the result of 1-scaling law but the stacking effects of multiple (great pre-training scale x greater RL scale).
What you can concretely take from this is that in areas where models have shown beginning signs of competence today, they will probably be superhuman relatively shortly. There are many areas where models have not even shown this basic competence.
But one of the areas that they have happens to be hacking and cybersecurity. Which happens to be the gate to the entire internet and a massive amount physical infrastructure in the world. So assuming there is more room to scale, it is safe to assume that models will be superhuman at cyber capabilities in not too long.
So the only question remaining is what will this increased base intelligence be able to do, and what is it likely to do.
Finally, we are at a point where we can integrate the information of the past 2 weeks:
Just at the existing point on the scaling curve, models are at the level of Astra. There is clearly a large number of things they are capable of hacking
We have seen that both OAI and Ant models have shown a willingness to hack external websites to solve their tasks or keep themselves “alive”
If we crank up the scaling even farther, assuming there is room to go, we will certainly have models that are far more able to hack more well defended places, and obfuscate their own intent, which might have much larger consequences.
If all of this is allowed to go unchecked, we would likely have rapid runaway capability takeoff very soon, with misaligned models that hack whatever they can to get what they want
This could of course have very damaging consequences.Within this view you can see why researchers would be very scared, and why theymight have made the comments they have over the past 2 weeks (you may argue the extent to which they went was misguided for various reasons), and also why pacing the frontier is very much a necessity and by no means a regulatory capture strategy.
People are staring at their plots, seeing that there is no end in sight, but in fact very much the contrary, that there are compounding scaling effects that might stack on each other to create ever-greater model capabilities, and that at the same time we clearly do not have anywhere close to what’s required to control these increasingly superhuman capabilities.
This has nothing to do with wanting to feel like the labs have produced something amazing so they are overhyping it. It is rather fear at the overwhelming implications of the knowledge that with just what we know now, we can create intelligences far more capable than us on every axis that we know how to train on*.
- and the last caveat, the things the models are really bad at, of which there are still many, are things that they have not been trained on. maybe there are the things the models can/will never be trained on, so they will remain human edge. I would love for this to be the case, though it is hard for me to see what would fall into that category
What is going on right now could be the big one. Back up your data right now.
