From Node 20 EOL to Node 27 LTS and Everything in Between

Hey, Marco, Mateo, absolutely delighted to have you guys here.

Hello. It's great.

So before I let you introduce yourself, let me just make a quick note about myself. I have a Node.js connection with you guys. I think the three of us work for Irish companies using Node.js. So that's a quick interesting fact.

Well, first of all, my name is Javier Perez. I'm a technical product management leader within HeroDevs. I cover over twenty-five technologies in the JavaScript ecosystem and a few others outside the JavaScript ecosystem, and of course Node.js.

I was saying that my connection here with Mateo and Marco is that — well, now it looks like many years ago, about twelve years ago — I worked for a company called FeedHenry, an Irish company in Waterford, Ireland, and we were using Node.js way back when it was version zero point seven, I believe, point ten, back in 2012. I remember dot ten and then dot twelve. We went through all of that. We don't have to talk about the history with the fork and then joining again and all that. That's well documented.

But that's when I worked very closely with Node.js. We were doing RSS technology and mobile apps connecting to cloud services — high-intensity I/O, which at the time was very new, but we knew that was the technology we had to use. And I don't think we were wrong. Our CTO and engineering team were not wrong.

The only thing I have to say is I'm just excited and delighted to be here with you guys, that you are working with Node on a daily basis. A lot of things have happened. Things change every six months, right? And sometimes significantly. So that's exciting to talk about. With that, why don't we start with you, Mateo? Would you briefly introduce yourself? Tell us about your story with Node, OpenJS, and TSC.

Of course. Yeah, of course, of course.

So okay, I started using Node, I think, in 2010, something like that. In 2011, I started my PhD. I was working in Ruby and Java before that, and I wanted something that was as fast as Java but as easy to develop as Ruby. Ruby was very fast for developing stuff in, but very slow at runtime — at least at the time. I think it still is.

And then I saw this video... and I was like, this is the technology. But to be honest, that was not the moment. The moment was when I understood NPM and when NPM was launched, which wasn't immediate — it was sometime later on. And when I noticed — going back to talking about this with you folks at HeroDevs — the moment when I saw that NPM would essentially solve how to reuse software at scale, which was a huge problem for the Java world and even the Ruby world before that. Nobody figured out how to do that before NPM. That's the main invention of NPM, and the result has been the biggest registry in the world — full of good things and also full of malware, which is the biggest target. So it's the juiciest target, and so on and so forth.

Having been one of the people attacked by state actors to gain access to my computer — I'm unlucky for that. But it's been great. So I started using Node very early on, then predicted left-pad months before it happened, and I've been in the trenches for a long time.

We actually share a little bit of history. In 2013, 2014, I was working for a small company called NearForm, which had the office next door — literally next door — to FeedHenry. So we share this little bit of history. A few people moved from FeedHenry to NearForm and vice versa over the years. It's been quite an interesting journey. We were probably in Ireland at the same time, working next door to each other. Those were fun years.

I started contributing to Node.js after the whole drama. The IOJS split was big, and then the foundation came. I had a lot of things I wanted to fix in Node and I couldn't, and then after the foundation came to be I started to do that. I started contributing a little bit to Node, and eventually after a few years I became a member of the CTC, which then merged into the TSC. Then I ended up — I don't know — picking the short straw and getting elected as the chair of the Node.js TSC.

We'll talk more about TSC. We could spend hours just talking about the stuff you've done, Matteo. How about Pino and Fastify?

I gathered that over the last year, all the modules that I maintain in the ecosystem have been downloaded forty-two billion times.

Crazy. Unbelievable.

And Fastify, as the name says, is the fastest framework, right?

Yeah. Maybe.

Marco, tell us about you. How did you get involved with Node and TSC?

Yeah, I'm definitely not one of the early adopters. I started getting interested in Node about eight years ago. Before that I was doing Java, and I was always like, I don't want to write all this boilerplate code — it was boring. And at the time Node.js was the freshest technology, still cutting edge.

It's been a while, and I started contributing a few years ago when I joined NearForm. I joined, I think, the week after Matteo left. So that was a coincidence. Then I started getting involved more with the security of the JavaScript ecosystem and getting interested in all these attacks that were going on.

That's great. And by the way, Marco is just only like, twenty-two, right? You're very young, Marco.

No, I'm twenty-eight.

Twenty-eight. Okay. That's very young.

Okay. I want to talk about all the things that are happening with Node.js in recent times. Before that — one of the reasons we organized this event with OpenJS — we've been collaborating with OpenJS for a few years now. In fact, we started exactly two years ago with the ecosystem sustainability program there. As you know, HeroDevs looks after the software that reaches end of life. And OpenJS is pretty open about: we want more and more innovation, more contributions, but we also want to take care of security and sustainability.

We know — and we could talk for hours about how hard it is sometimes for organizations with large deployments to just upgrade, migrate from one version to another. The risk of breaking changes and all of that. But we just recently had the end of life for Node 20. On April thirtieth, at the end of April, less than a month ago, Node 20 — which was a very significant version with significant usage — reached end of life. End of maintenance, end of updates, minor releases, patches.

Sorry to interrupt you — do you know how many times Node 20 was downloaded in April?

I'm not going to go to npm right now, but tell me.

Ninety-five million times.

Just saying.

I just wanted to say here that most people deploy Node.js in an app, especially in certain deployments, and they don't upgrade at all after that. What we are seeing as we get more adoption is that a lot of companies, a lot of teams are not updating at all. And for them, updating Node is such a big thing, because typically they have fallen so far behind in the dependency tree that they need to do major refactoring and restructuring to update their full dependency tree. So instead of doing it in small chunks, they need to do a big bang release.

But yes, Node 20 has not been budging from the ninety-million threshold. And in April, to be honest, the biggest download month for Node.js was around one hundred and twenty-five million, back in June or July of last year. Matteo has a lot of good data there. I've seen some of your charts — really good stuff.

Yeah, I have my charts on this other screen, so I have my facts right.

Marco, you showed me a screenshot of all those Node 20 releases — you directly worked on releasing most of them. Tell us about that experience. How was it?

I think about seventy percent of the Node 20 releases when it went LTS were done by me and other volunteers.

Releasing Node is a lot of work, and it requires a lot of patience because things tend to break, and it involves talking with many people and asking for help on parts that you might not know. And it's a lot of work. I'm happy that HeroDevs is sponsoring my open source work on the Node releases, because one of the problems of sustainability is funding volunteers to do this kind of tedious and sometimes boring work. And I guess that's also one of the main reasons why we're shifting the release schedule.

Let's talk about that. One of the big news items from the Node community is changing the release schedule. Do you want to start with that, Marco? What went into that decision?

Well, first of all, for everyone not familiar — we have releases every six months. The odd numbers are not LTS. The even numbers are the LTS: the eighteens, twenties, twenty-twos, and so on. But now that's changing to one per year, and all the even numbers are going to be LTS.

And the other thing I love about this community is that everything is really in the open, right? I went and found the discussion forum where this was discussed months ago.

Yeah, so the main driver for this is that maintaining four releases at the same time — because sometimes there are four releases that need to go out — is too much work, especially for security releases. It causes a lot of stress for mostly volunteer contributors. And the other problem is that maintenance diverges too much. The LTS release is so different from the latest version that it's very hard to backport fixes. So it becomes a lot of work that may be too much to be sustained by volunteers.

So the idea was: instead of having so many major releases with major changes, why don't we try to reduce the scope of the releases we keep? So in 2027, we will release Node 27, which will be the first odd version to become LTS. This means we will only have one major release per year, which will simplify life for releasers. And I think it also helps companies — you only have to bump one major instead of two.

Do you want to comment on that, Matteo? Any other items that were discussed?

By the way, one nice thing is that version twenty-seven is going to align with the year, right? In 2028.

The biggest thing is aligning with the years.

So Node 26 will be LTS, and then Node 27 in 2027 will also be LTS. There are a few things. The fundamental part comes from data. This decision, above all others, comes from real data. Nobody uses the non-LTS releases. Almost no one. So there's no point. All the work that releasers do to get these out is for very few people. And module authors then have these releases that need to be tested and maintained, but they only last six months. Again, it's just not worth it.

It's well known in the open source space across all projects: if there's a formal LTS long-term support commitment by the community, people take advantage of that. Why would anyone go to a non-LTS version?

Yeah, exactly. Some people do because they want to test new things. And the way the new program is structured, there is a long alpha version where we can ship major changes and breaking changes. So if you want to live at the edge, you can use that. We still do alpha releases, so the work will be done. However, the major difference is that with those alphas there will be no security guarantee. So there are no backports to do, nothing. It reduces the amount of work needed for a security release, which is one of our biggest concerns right now, given the huge stream of valid reports coming in.

That's the other big point of this conversation — security, vulnerabilities. Before I jump into that, I think there's another important piece: every six months we get a lot of new functionality. It's not just fixing CVEs, right? I just want to ask you guys — how do you do it? You have so many volunteers, you're constantly adding new APIs, improving performance...

To be honest, let's unpack that. Are we improving performance? Yes, a little bit, maybe. But the biggest concern is not breaking backward compatibility and minimizing the reasons why people can't upgrade. That's the main driver. As I always say, we could make Node.js significantly faster, but it's going to break stuff. Do you want to go faster? How much are you willing to break? So there is that. And we have proven that even achieving forty or fifty percent Node.js compatibility is enough for some people — there are projects that do that, and it's fine. But in reality, companies with large deployments are not going to migrate. It is hard. It is really hard.

Marco, what else?

I have one more thing to say. People are not upgrading. Every single new block of users we get — the usage of Node increases, but they don't move from that version. It's really stratifying itself. I checked, and there is still a significant amount of downloads of Node 16. It's still downloaded twenty million times per month or something. And it's like — why are people even testing with this? It's history now. It's really problematic.

On the feature side, we keep shipping new things. But we are actually very conservative to some extent. We were allowed to ship way more. People wanted to ship way more, and we tried to keep it a little more constrained than what people actually wanted to ship.

You want to add there, Marco?

I find it unbelievable, and kudos to everyone in the Node.js community because they keep adding stuff. The reality is that to maintain backwards compatibility and not break legacy systems and frameworks based on internals — we try not to break users. It's crazy. Sometimes we decide not to remove things because the impact, even of a simple major change, could affect too many people, too many packages. The real problem is the ecosystem. For example, packages that have been on npm for the past fifteen years that rely on some internals we want to deprecate, but we can't because the package is not maintained and has fifty million downloads per week. This is why Node.js tends to be very conservative. We could deprecate a lot of APIs, remove a lot of legacy internals that have been exposed, but we can't. So when I see the changelog of any given major, I always think we should have deprecated so much more, and we could have done so much more, but we can't.

What I'm hearing confirms what I knew about this community and this project — it's a well-run project. You guys do a fantastic job, all the volunteers, all the collaborators. They have that commitment to keep it up without breaking things as much as possible. Every new version will bring some breaking changes and some deprecated APIs, but there's a commitment. And that's, in my opinion, the key to the success of Node.js and why so many organizations and apps adopted it: there's a formal process, and there's a commitment to long-term support.

Now let's jump into — well, I guess it's not fun, but it's interesting and sometimes even a sad situation with security. One of the major changes recently announced by the TSC, Mateo, was the suspension or pause of the bounty program. There was a blog post explaining it, but maybe you can comment here for our audience — while clarifying that it doesn't change the process to fix vulnerabilities.

Two problems. Okay. So first of all, since the beginning of last year, we saw an incredible amount of AI slop being used to report vulnerabilities. And that was massive. So a few things are true at the same time. When you report a vulnerability in Node.js, you can get a vulnerability that impacts all users of Node and can essentially be a silver bullet for a denial-of-service attack. Super bad. That is one of the things we can get. Or we can get a vulnerability that technically is a vulnerability, but it depends on the environment — the Moon and Venus all being in alignment, and the only way to trigger it is to jump three times on the right foot, then three times on the left foot, and pronounce the magical words. This is the type of thing that gets reported.

To be honest, they are real vulnerabilities. I'm not saying they're not. I'm just saying that they are at best tiny gadgets that in a longer chain could be exploited to increase the penetration of an attacker, but on their own are of very little importance. They are CVEs though, so we need to treat them as such. But the cost of finding them in a codebase is close to zero.

So at that point the question becomes: should we have a bounty for them? Should we pay for it? I am spending tokens to generate tokens to get money. Let's put it this way: spending a hundred euros in tokens to get something that pays a thousand euros as a bounty — it's a great investment. A hundred dollar subscription turned into a thousand euros a month from bounties. People can live on this.

But is this something we want to encourage or sustain? Now if we have companies willing to sponsor it, absolutely. We are very happy to triage and do the work. But the reality is there was no sponsor anymore. And to be honest, we wanted to stop the slop. The slop was the biggest problem because a lot of people would see it as a quick money scheme with absolutely no competence whatsoever in the reporting. I'm fine talking with an AI, but the AI is more competent than a lot of what we have seen. We have literally seen copy and paste of the prompt pasted inside a report. The competence was not even in copying and pasting things correctly.

We want to talk about AI, of course. The audience wants to talk about AI. But first, Marco, the other important thing — the bounty program being suspended doesn't change the security reporting process. That remains unchanged.

Can you walk us through that process? The intake, the triage?

Yeah, so the intake is through HackerOne. We receive reports on HackerOne. We do triaging of the reports — it's volunteer-based, so we have some people doing triage work when they have time. Not many people are interested in doing this work because we receive so many reports, and there's currently an asymmetry between the time to generate a report and the time to triage one. It takes some time to check that something is true and to reproduce it. That's one of the biggest problems.

But so after we have received the report on HackerOne, and we have enough reports to create a release, we do an announcement saying that we'll be doing a security release in two weeks, for example. In private, we create the patch for those vulnerabilities. And we have to coordinate a few releases to release at the same time as the updated version. That's the hardest part, because you have to coordinate two or three people — sometimes even more than that — because you want to put the people who fix the vulnerabilities and the releasers together and do the work at the same time. So when there is a security release, there is a lot of work. That couple of weeks are very intense for everyone involved.

And that's one of the reasons why we changed the schedule — to have less work during those periods. And of course, it goes without saying: the more information in the report the better. If you already have a suggested fix, put all of that in your report.

It's super rare that people include a fix. Most of the time it's impossible. Most people, even if there is something, don't include one. But we welcome them. Sometimes there are very good patches. Sometimes there are not.

The process works, but it's very intense also because it's chained. We have a few dependencies of Node that we maintain inside the Node.js organization. Especially one called LLHTTP, which is our HTTP parser. We need to do releases of that HTTP parser — and by the way, this is an interesting piece of technology, you should get Paolo to talk about it sometime. Anyway, there is the HTTP parser. Then we have Undici, which is the library we use for Fetch, along with a lot of new WhatWG-based APIs. I am the main maintainer of Undici, and at every security release of Node, a bunch of Undici fixes go in. Currently Node has three LTS releases — Node 22, Node 24, and the active one, Node 26 — and we need to issue patches for all three. Node 22 is using Undici 6, and the other versions have their own.

It's so much work that this community, these contributors, these collaborators put in. And then we have thousands and thousands of organizations running live production systems using JavaScript, Node.js, TypeScript, and so on. That's why companies like HeroDevs go back and contribute to these communities. You're not only helping — you're also gaining a lot of expertise and experience being part of those communities.

All right, let's talk about AI, because of course people want to talk about AI. There are a ton of new tools out there and people reporting vulnerabilities with that AI slop. But I also heard, and Marco, you wrote a nice blog about it, that it's not just about reporting — context and knowledge are the most important piece. But I also want to talk about the fact that it's actually getting better. Maybe now you're getting some real vulnerabilities reported by these LLMs.

We do. We do. We got real vulnerabilities. This is real. The capabilities of the LLMs — most of them are still just bugs, but now it's even true vulnerabilities. I think the rate has increased significantly. Last time I checked, I think it was sixty-six percent versus fifty percent previously. At some point it was super bad, but then it recovered. So far I would say that most things that get reported are plausible enough and non-trivial to debug. The trivial stuff has been documented. We've been on a campaign to document in the security MD all the things that make a good vulnerability versus a bad one.

Is Node.js involved with Mitre?

We are not, unfortunately. We asked for access. We have not been granted access so far. I'm sure it will happen — it has to be one of the key open source projects. But anyway, even without that, we see improvement in LLM-reported issues. And what I was reading that is super interesting is that beyond human capacity, they can chain multiple vulnerabilities and create a real-life exploit.

Yeah, that's the hard part. That's the novelty — that single specific thing. In some cases, CVEs that might be just low or medium severity, while chained with other CVEs, can become something really, really critical.

Marco, and I have to ask you: so we get new CVEs, new fixes for vulnerabilities. But then you have to go back and patch those end-of-life versions for HeroDevs. What goes into that effort?

Yeah, so... when I say that I work on Node 12, Node 14, Node 16, other collaborators are like, what? Working on old versions of Node is quite hard because of very old dependencies. The other big problem is how insecure these old versions are. It happens a lot. And as I'm fixing vulnerabilities, I ask myself: how did we survive with this six, seven years ago? Because as we've moved on, we've found a lot of vulnerabilities, a lot of bugs. Whenever there is a new security release, I always check whether it applies to end-of-life versions, and most of the time it does. So as we move on, we still find bugs in the old versions. It's crazy.

Just to be clear for the audience: end of long-term support means no more upstream fixes. For Node 20, for example, you will have to go to a commercial offering like HeroDevs. That's obviously a big challenge, and it makes all the sense in the world for a project to keep moving — and frankly, moving as fast as possible. Which of course brings the challenge of migrating and breaking changes. And as Matteo was saying earlier, we see that companies and organizations are not necessarily migrating. They keep using versions like 18, 20, and some older ones.

It's a lot of work, Marco, and you and some of the other engineers here at HeroDevs put a lot of effort into that and the testing. Actually, that was another thing I was going to ask you guys. It has to be a crazy amount of testing every release. How do you do it?

Yes. The hard part of the release is getting the CI green. We have a lot of tests on so many exotic platforms. We have something called Canary in the Goldmine where we run the candidate release against npm packages from the ecosystem to make sure we don't break packages. We have so many CI jobs that take hours. Every time one fails, you have to rerun it and wait a few hours. So it's pretty annoying, but it's important.

It never passes.

It never passes. At some point I was literally trying to improve reliability. We get a reliability report every day showing how many flaky tests we have, and for a while I ended up patching them and feeding that report to AI. A few people complained that the PRs were not so good, but at least I was trying to help. And then I stopped. But the problem is very tricky because even landing those fixes is hard. It's not straightforward.

Yeah, it is a tremendous amount of work. Any developer building releases can relate to that.

I want to end with one more thing. There was a recent session in London where the Node.js community got together. There were a number of initiatives discussed. That's when this release schedule change was announced. Is there anything you want to add, Marco or Matteo, in terms of what's coming for Node.js? For security, for example — any other news or changes around how to handle this increase in CVE reports?

I'm also sitting on the board of the OpenJS Foundation, and we are currently working with a group of vendors who have offered to cover the development of the bounty program. We are working on finding a structure that makes sense for them to support the program and give them something, but also makes sense for the maintainers to spend their time. If we put a bounty on something that is relatively easy to get, it's literally a get-rich-quick scheme. It probably should only be on high and critical vulnerabilities.

To be honest, it could just be: let's take this pile of money and burn tokens to find all the vulnerabilities ourselves. It has become a numbers game. I would really love to give big bounties to researchers that do real, solid work and find real things that can impact millions. And we have had a few — we've had some very, very big ones. Usually though it's tiny edge cases, or maybe it can only work if you don't put your node server behind a CDN and a load balancer, which is frankly not the case for anybody at this point.

Or somebody keeps reporting the inspector protocol issue. We have written it everywhere: if you use the inspector protocol, you are on your own. These are debugging tools for development. Don't leave that open in a production system. And people keep reporting it as an attack scenario. But literally, you can't secure the thing. There's no way it can be secured — it's literally designed to allow somebody to take over that node instance. It's impossible.

I used to have a presentation when I was doing more app security work saying: look, there are millions of vulnerabilities out there. Pay more attention to the documented exploits. But even those are often tiny.

Let's talk about a vulnerability that I handled in the last security block — in Undici. In Node.js we ship a WebSocket implementation which comes from the spec. Not a great standard, but that's a different topic. Anyway, the WebSocket standard offers an option to do packet compression, because WebSocket packets have their own framing on top of TCP — it's HTTP, WebSocket, and then their own framing on top. And you can compress those packets. I didn't know that, but we support it.

And it turns out that if somebody uses Node.js to connect to a remote WebSocket instance that you don't trust for whatever reason, that server can send you a silver bullet packet that causes a denial of service and crashes your app — because it's, say, a hundred kilobytes that expands to a hundred gigabytes in memory, and you crash. Easy peasy.

And so there was that. And the other side of it — still talking about Undici — is that it's a feature you have to explicitly enable. You only get this vulnerability if you enable an experimental feature and write code that enables it manually. So you have that range: on one end, it's hard to exploit. You need an application that connects to a WebSocket that accepts a remote WebSocket server as an input, which makes it very limited in scope.

So many, so many details. That goes into, for me, the value that the project and all the volunteers bring to this ecosystem.

I was just looking quickly through the questions. I think we covered them. There was a question about the type of vulnerabilities coming from AI. I just want to end by saying: time flies when you're having fun. I'm sure we could talk for another two or three hours on this content.

I hope the audience found this valuable. We're going to continue to do webinars like this. Let me just repeat that HeroDevs is a founding member of the OpenJS Foundation Ecosystem Sustainability Program. We support the open source community while also running a business for organizations that still run end-of-life versions of Node. We support from version 12 — Marco, is that right?

Twelve, fourteen, sixteen, eighteen, and now twenty.

We also have custom versions of twenty-two and twenty-four before they go end-of-life, for customers who want to be prepared before the end-of-life date.

Excellent. Well, thank you very much, Matteo and Marco, for joining. I appreciate you taking the time. I'd love to continue the conversation, and thanks everyone for watching. Bye-bye, folks. Bye-bye.

‍