Future of Artificial Intelligence
Re: Future of Artificial Intelligence
I think it is probably legit. Open weights with a paper on general process: https://arxiv.org/pdf/2412.19437. Open question of what data was used and if model distillation was used on inferences from other leading models. Either way, very impressive and reinforces the effectiveness of regular RL to discover emergent reasoning capacities and generate synthetic data. From what I have seen, r1 reasoning is rawer than o1. Supposedly o3 coming up from OpenAI is still ahead on benchmarks, but I wonder if that will last.
Some oversea competition towards commoditized AGI might be raising all boats so long as the east/west power divide doesn't grow too wide. There is a lot of room at the bottom for cheaper models distilled to purpose, but there is also potentially a lot of room at the top for undistilled generality powered by the asymmetry between ease to verify answers and difficulty in generating correct answers. Leveraging this asymmetry in math/coding/gaming and transferring reasoning structures into other domains as seen fit (this diffusion will likely be more gradual where verification signal is weak/slow).
Some oversea competition towards commoditized AGI might be raising all boats so long as the east/west power divide doesn't grow too wide. There is a lot of room at the bottom for cheaper models distilled to purpose, but there is also potentially a lot of room at the top for undistilled generality powered by the asymmetry between ease to verify answers and difficulty in generating correct answers. Leveraging this asymmetry in math/coding/gaming and transferring reasoning structures into other domains as seen fit (this diffusion will likely be more gradual where verification signal is weak/slow).
Re: Future of Artificial Intelligence
The ultimate product of AGI being robots. A handful of AGI robots can build robot factories to build more AGI robots to build/do anything else. Driving up the cost of land/resources and down the cost of all other products/services in the short term. In the longer-term, resource constraints will drive product cost up gradually leading to higher quality products more specialized to purpose that go through more recycling phases. The tech singularity spiral points towards something like a parallel silicon evolution rivaling the complexity of carbon-based evolution.
Re: Future of Artificial Intelligence
Another note on DeepSeek r1 is that much of the efficiency gains likely come from using mixed precision as discussed in the paper above. This technique will no doubt be copied and improved by others in short order.
Scaling intelligence must run into diminishing returns on the profitable use cases at some point. These kinds of innovations raise the ceiling of what is possible before reality sets in. For true AGI that can improve medical outcomes, run organizations, replace/extend labor, project military power, etc. countries and businesses are going to be willing to spend a lot of money to squeeze out less and less juice.
Scaling intelligence must run into diminishing returns on the profitable use cases at some point. These kinds of innovations raise the ceiling of what is possible before reality sets in. For true AGI that can improve medical outcomes, run organizations, replace/extend labor, project military power, etc. countries and businesses are going to be willing to spend a lot of money to squeeze out less and less juice.
Re: Future of Artificial Intelligence
I think to Daylen’s point, everything just became cheaper and will move along faster than before.
The following post focuses on this. It’s a building block that changes the game.
https://x.com/yishan/status/1884101107368223113
The following post focuses on this. It’s a building block that changes the game.
https://x.com/yishan/status/1884101107368223113
I think the Deepseek moment is not really the Sputnik moment, but more like the Google moment.
If anyone was around in ~2004, you'll know what I mean, but more on that later.
I think everyone is over-rotated on this because Deepseek came out of China. Let me try to un-rotate you.
Deepseek could have come out of some lab in the US Midwest. Like say some CS lab couldn't afford the latest nVidia chips and had to use older hardware, but they had a great algo and systems department, and they found a bunch of optimizations and trained a model for a few million dollars and lo, the model is roughly on par with o1. Look everyone, we found a new training method and we optimized a bunch of algorithms!
Everyone is like OH WOW and starts trying the same thing. Great week for AI advancement! No need for US markets to lose a trillion in market cap.
The tech world (and apparently Wall Street) is massively over-rotated on this because it came out of CHINA.
I get it. After everyone has been sensitized over the H1BLM uproar, we are conditioned to think of OMG Immigrants China as some kind of Alien Other. As though the Alien-Other Chinese Researchers are doing something special that's out of reach and now China The Empire is somehow uniquely in possession of Super Efficient AI Power and the US companies can't compete. The subtext of "A New Fearsome Power Now Under The Command of the CCP" is what's driving the current sentiment, and it's not really valid.
Like, no. These are guys basically working on the same problems we are in the US, and not only that, they wrote a paper about it and open-sourced their model! It is not actually some sort of tectonic geopolitical shift, it is just Some Nerds Over There saying "Hey we figured out some cool shit, here's how we did it, maybe you would like to check it out?"
Sputnik showed that the Soviets could do something the US couldn't ("a new fearsome power"). They didn't subsequently publish all the technical details and half the blueprints. They only showed that it could be done.
With Deepseek, if I recall correctly, a lab in Berkeley read their paper and duplicated the claimed results on a small scale within a day.
That's why I say it's like the Google moment in 2004. Google filed its S-1 in 2004, and revealed to the world that they had built the largest supercomputer cluster by using distributed algorithms to network together commodity computers at the best performance-per-dollar point on the cost curve.
This was in contrast to every other tech company, who at that time just bought what were essentially larger and larger mainframes, always at the most expensive leading edge of the cost curve. (To the young people reading this, this will sound incredible to you)
I worked at PayPal at the time, and in order to keep pace with the rising transaction volume, the company was forced to buy bigger and bigger database servers from Oracle. We were totally Oracle's bitch. At one point when we ran into scalability issues, the Oracle reps told us we were their biggest installation so they had no other reference point on how to help us overcome our scalability issues. We literally resorted to flipping random config switches and rebooting it.
(This heavily influenced me when I was a young manager later at Facebook. I deliberately torpedoed an Oracle salesman's pitch to try and get us to switch from open source MySQL databases to an Oracle contract: of course we had scalability problems, but at least when we had them, we could open up the hood and figure out how to fix it ... assuming we had good enough engineers, and we did. When it's closed-source infra, you're at the mercy of the vendor's support engineers)
Back to Google - in their S-1, they described how they were able to leapfrog the scalability limits of mainframes and had been (for years!) running a far more massive networked supercomputer comprised of thousands of commodity machines at the optimal performance-per-dollar price point - i.e. not the more expensive leading edge - all knit together by fault-tolerant distributed algorithms written in-house.
Some time later, Google published their MapReduce and BigTable papers, describing the algorithms they'd used to manage and control this massively more cost-effective and powerful supercomputer.
Deepseek is MUCH more like the Google moment, because Google essentially described what it did and told everyone else how they could do it too. In Google's case, a fair bit of time elapsed between when they revealed to the world what they were doing and when they published a papers showing everyone how to do it. Deepseek, in contrast, published their paper alongside the model release.
Now, I've also written about how I think this is also a demonstration of Deepseek's trajectory, but that's also no different from Google in ~2004 revealing what it was capable of. Competitors will still need to gear up and DO the thing, but they've moved the field forward. But it's not like Sputnik where the Soviets have developed technology unreachable to the US, it's more like Google saying, "Hey, we did this cool thing, here's how we did it."
There is no reason to think nVidia and OAI and Meta and Microsoft and Google et al are dead. Sure, Deepseek is a new and formidable upstart, but doesn't that happen every week in the world of AI? I am sure that Sam and Zuck, backed by the power of Satya, can figure something out. Everyone is going to duplicate this feat in a few months and everything just got cheaper. The only real consequence is that AI utopia/doom is now closer than ever
Re: Future of Artificial Intelligence
It seems possible that they improved accuracy X efficiency at the cost of unknown creativity.
Re: Future of Artificial Intelligence
daylen wrote: ↑Mon Jan 27, 2025 9:02 pm
Scaling intelligence must run into diminishing returns on the profitable use cases at some point. These kinds of innovations raise the ceiling of what is possible before reality sets in. For true AGI that can improve medical outcomes, run organizations, replace/extend labor, project military power, etc. countries and businesses are going to be willing to spend a lot of money to squeeze out less and less juice.
Actually, the opposite occurs. See Jevons Paradox. It's all the rage.
Last edited by Henry on Wed Jan 29, 2025 9:22 am, edited 1 time in total.
-
- Site Admin
- Posts: 16622
- Joined: Fri Jun 28, 2013 8:38 pm
- Location: USA, Zone 5b, Koppen Dfa, Elev. 620ft, Walkscore 77
- Contact:
Re: Future of Artificial Intelligence
I'm not so sure. It depends on whether generative language models have maxed out their hardware. As an analogy, modern word processors have the same features and utility as they did 30 years. There's no need to throw ever faster hardware at them.
Likewise, chatGPT basically behaves like a histrionic extravert with a college degree. The meat robot version only uses 20W.
I'll change my mind if this new breakthrough means raising the AI IQ from 101 to 110 through brute force.
Re: Future of Artificial Intelligence
It won't be the language models that raise AI IQ from 101 to 110. It will be the inference models that use the language models. All recorded history is now accessible to language models. It's the physical world that now needs to be accessed by AI and, asterisk for emphasis, interpreted by AI. Elon uses the movement of water in a glass. AI's goal is to "see" it, anaylyze it, interpret it, etc. better than a human. DeepSeek is not a cheaper alternative to turn the physical world into data, and then analyze it. It could be used however, in the process. And let's face it. It won't be DeepSeek. That's like the TIk Tok for AI. The government already announced it won't use it. It will be a US version. Ask DeepSeek about a famous photograph of a man holding a brief case in front of an Army tank in a public square and wait for that fucking answer.
It's the difference between Waymo and TSLA. Waymo operates FSD in a specific region in the world. Put it in a place where it hasn't collected the data, you have a 90 year old blind woman on Acid driving the Audubon. The goal of TSLA is to FSD anywhere. That is a different endpoint of AI.
It's the difference between Waymo and TSLA. Waymo operates FSD in a specific region in the world. Put it in a place where it hasn't collected the data, you have a 90 year old blind woman on Acid driving the Audubon. The goal of TSLA is to FSD anywhere. That is a different endpoint of AI.
Last edited by Henry on Wed Jan 29, 2025 9:50 am, edited 1 time in total.
Re: Future of Artificial Intelligence
With o1 and especially o3, I think we have already passed the 110 IQ barrier. I think it is clearer than ever that these models will reach superhuman performance in math and coding before true AGI that can act in the world. The diminishing returns I am talking about above has to do with physical limits. Having a bunch of Einstein's or Von Neumann's in a bottle isn't of much help if they cannot test ideas directly through atoms. Once they can, the sky isn't even the limit (but eventually some limit sets in where more ideas do not help much).
Re: Future of Artificial Intelligence
I agree. ChatGPT is currently in the quadrant only point adjacent to a smoke detector.
Re: Future of Artificial Intelligence
A bot named truth_terminal has managed to make money in crypto and gain a cult following.
https://x.com/truth_terminal

Re: Future of Artificial Intelligence
If you use their app Deepseek, yes. That is their model which is hosted on Chinese servers. If you download Deepseek R1, no. That is the open source model that has no censoring and is hosted on your own servers, doesn’t send data to the company/China and so on.
———
The chat bots are useful, but the main innovations seem likely to be what is built on top of the LLMs.
Re: Future of Artificial Intelligence
Consumers have no issues with Chinese knocks off. Companies and the government not so much.
I will say this in the most delicate way possible. It has something to offer. There are advancements. But consider the source.
I will say this in the most delicate way possible. It has something to offer. There are advancements. But consider the source.
Re: Future of Artificial Intelligence
Dario of Anthropic does a better job at summarizing the current situation as a live player in the space than I can (regardless of the political concerns about export controls): https://darioamodei.com/on-deepseek-and-export-controls
Another worthy note/news in AI being MatterGen from Microsoft that could accelerate progress in materials science and engineering underlying any tech stack: https://www.microsoft.com/en-us/researc ... dd80906a61
Another worthy note/news in AI being MatterGen from Microsoft that could accelerate progress in materials science and engineering underlying any tech stack: https://www.microsoft.com/en-us/researc ... dd80906a61
Re: Future of Artificial Intelligence
Last night there was an unspeakable air disaster that involved the US military and a major US commercial flight. Imagine being one of those air traffic controllers coming home to "How was your day, Honey?" Wouldn't it be the most humane thing to make that job obsolete in order that a person never has to wake up to that nightmare? In context of this thread heading,I think a question to ask is whether future generations of technology addicted droolers will be asking their humanoid sexbots to play AI Narrated YouTube "Air Travel Before AI" compilations on their forehead screens while they engage in human to humanoid Funtime. I'm betting yes.
Re: Future of Artificial Intelligence
I don't think you can entirely remove the human component and hence the inevitability that something someday will go very wrong. The 2002 Überlingen mid-air collision was an example of how AI worked perfectly but humans don't.
Re: Future of Artificial Intelligence
There will be a tipping point where, suddenly, we shift from it being unthinkable that we allow computers to control the plane (car, train,....) to it being unbelievable that we ever allowed a human to control it. I believe that tipping point will come - in the same way that many other changes will come in the near future - as the result of unaffordable insurance.
Re: Future of Artificial Intelligence
In the even more near term, asking a robotaxi to play car crash compilations (or porn). In a slightly more distant future, neuralinking into the metaverse of dangerous or pleasurable experiences for cheap yet immersive thrills.
Re: Future of Artificial Intelligence
When FSD proves to be a 1% better driver than the average driver (not that great of an accomplishment) insurance companies will charge more if the car is not equipped with FSD. That raises the question as to whether TSLA will license their FSD, which I believe they will, in the same way they share their charging stations. This will prohibit monopoly allegations. Once AI gets us to the point that there are no crashes, well there goes car insurance altogether, even theft, as your TSLA will call the robot police to tell them that the current fart volume/tone/smell on the driver side does not align with the owner's preprogrammed flatulence data.
Furthermore, no more speeding tickets, running red lights, and road rage episodes. Fortunately, adaptation in Russia will most likely lag, so we'll still get their you tube videos of drunk guys getting pulled out of multi-colored Pintos and beaten to a pulp and left for dead in a ditch next to a snow covered road. I apologize to anyone who is Russian, but Russian road rage is next level. What's interesting is how frequently they originate from people inside the same car and it's more often than not, the same scenario. A dash cam shows a car driving down the road. The car slows down, comes to a complete halt. A guy the size of Ivan Drago gets out the driver side. He opens up the rear passenger door and yanks out a scrawny guy in an army jacket holding an open beer can. Buy the time the scrawny drunk guy knows what's happening, the Ivan Drago guys knocks him out, pulls him into a ditch by the side of the road, walks back to the car and drives off. No one beeps let alone checks on the scrawny drunk guy now bleeding and left for dead in an icy ditch. I know shit runs Brothers Karamazoff deep there, but man do they take care of their personal disputes in a fucking efficient manner.
Re: Future of Artificial Intelligence
Imagine the possibilities of hijacking AI systems and turning the system against itself. Flying planes en masse into nuclear power stations for example.