The following is not the typical read you see on this blog. It’s a mark on the history of web development. Denis Pushkarev, the author of core-js, the library who provides a way polyfillers to modern web transpilers like Babel, does a breakdown on how he has been supporting the project for over 8 years, while not getting any financial compensation over it. Even worse, it’s common for Denis to receive hate messages from developers who user core-js, indirectly refering to him as a slave.
So, what’s next?
Hi. I am (@zloirock) a full-time open-source developer. I don’t like to write long posts, but it seems this is high time to do it. Initially, this post was supposed to be a post about the start of active development of the new major version of
core-js and the roadmap (it was moved to the second half), however, due to recent events, became a really long post about many different things… I’m fucking tired. Free open-source software is fundamentally broken. I could stop working on this silently, but I want to give open-source one last chance.
If you or your company use
core-js in one way or another and are interested in the quality of your supply chain, support the project:
Write me if you want to offer a good job on Web-standards and open-source.
- It is the most complex and comprehensive polyfill project. At the time of publishing this post,
core-jscontains about a half thousand polyfill modules with different levels of complexity - from
Symbol— that designed to work together. With another architecture, each of them could be a separate package - however, it is not so convenient.
- It is maximally modular — you can easily (or even automatically) choose to load only the features you will be using. It can be used without polluting the global namespace (someone calls such a use case “ponyfill”).
- It is designed for integration with tools and provides all that’s required for this — for example,
@babel/transform-runtime, and similar SWC features are based on
- It is one of the main reasons why developers can use modern ECMAScript features in their development process each day for many years, but most developers just don’t know that they have this possibility because of
core-jssince they use
core-jsindirectly as it’s provided by their transpilers / frameworks / intermediate packages like
- It is not a framework or a library, which usage require the developer to know their API, periodically look at the documentation, or at least remember that he or she is using it. Even if developers use
core-jsdirectly — it’s just some lines of import or some lines in the configuration (in most cases — with mistakes, since almost no one read the documentation), after that, they forget about
core-jsand just use provided by
core-jsfeatures from web-standards — but sometimes this is the most of JS standard library that they use.
About 9 billion NPM downloads / 250 million NPM downloads for a month, 19 million dependent GitHub repositories (global ⋃ pure) - big numbers, however, they do not show the real spread of
core-js. Let’s check it.
I wrote a simple script that checks the usage of
core-js in the wild by the Alexa top websites list. We can detect obvious cases of
core-js usage and used versions (only modern).
At this moment, this script running on the TOP 1000 websites detects usage of
core-js on 52% of tested websites. Depending on the phase of the moon (the list, websites, etc. are not constants), results may vary by a few percent. However, it’s just a naive detection on initial pages using a modern browser that loses many cases, manual check shows that it’s additional dozens of percent. For example, let’s leave the initial pages of some websites from the screenshot above where
core-js was not found by this script, without repetition of each company (at first — MS that’s already on the screenshot) websites (be patient, after the series of screenshots the number of pictures will decrease):
With such a manual check, you can find
core-js on about 75-80 of the top 100 websites while the script found it on about 55-60. On a larger sample the percentage, of course, decreases.
Wappalyzer allows detection of used technologies, including
core-js, with a browser plugin and has previously shown interesting results, but now on their website, all the most popular technologies’ public results are limited to only about 5 million positives. Statistics based on Wappalyzer results are available here and show
core-js on 41% and 44% of 8 million mobile and 5 million desktop tested pages. Built With at this moment shows
core-js on 54% of TOP 10000 sites (however, I’m not sure about the completeness of their detection and see the graph from another reality).
Anyway, we can say with confidence that
core-js is used by most of the popular websites. Even if
core-js is not used on the main site of any large corporation, it’s definitely used in some of their projects.
What JS libraries are more widespread on websites? It’s not React, Lodash, or any other most talked-about library or framework, I am pretty sure only about “good old” jQuery.
However, for the above reasons, almost no one remembers that he or she uses
Why am I posting this? No, not to show how cool I am, but to show how bad everything is. Read on.
Let’s start the next part with one popular
Harmony 6 standard with great hope.
It was a time when ECMAScript transpilers started to become popular and develop actively. However, at the same time, polyfills almost have not evolved according to users’ and real-life projects’ needs. They were not modular. They were not been able to use without global namespace pollution - so they were not suitable for libraries. They were not one complex - it was required to use some different polyfill libraries from different authors and somehow make them work together - but in some cases, it was almost impossible. Too many necessary fundamental language features were just missed.
To fix those problems, at the end of 2012, initially for my own projects, I started to work on the project that later was called
core-js. I wanted to make the life of all JS developers easier and in November 2014, I published
core-js as an open-source project. Maybe it was the biggest mistake in my life.
Since I was not the only one who faced these issues, after a few months,
core-js had been integrated into Babel (
6to5 at that moment) that appeared a couple of months before
core-js publishing - some of mentioned above issues were critical for this project too.
core-js began to be distributed as
6to5/polyfill, after rebranding -
babel-polyfill. After some months of collaboration, has appeared a tool that became
babel-runtime after rebranding and evolution. After some months
core-js was integrated into the key frameworks.
Ensuring compatibility for the whole Web
I didn’t promote myself or the project. This is the second mistake.
However, it was just the start of the required work. Many years of hard work followed. Almost every day I spent some hours on
core-js and related projects (mainly Babel and
6 2015 followed new proposals, new versions of ECMAScript, new non-ECMAScript web standards, new engines and tools, etc. The evolution, the improvement of the project, and the adaptation to the current state of the Web have never stopped - and almost all of this work remains not visible to the average user.
The scale of required work was constantly growing.
I tried to find other maintainers or at least constant contributors for
core-js in different ways for a long time, but all attempts have failed. Almost every JS developer used
core-js indirectly and knew, for example,
babel-runtime, or that his framework polyfill all required features, but almost no one knew
core-js. In some posts about polyfilling where
core-js was mentioned, it was called “a small library”. It was not a trendy and widely discussed project, so why help maintain it if it works great anyway? Over time I lost hope for it, but I felt a responsibility to the community, so I was forced to continue working alone.
After a few years combining full-time work and FOSS became almost impossible - no one wanted to pay money for the working time devoted to FOSS, non-working hours were not enough, and sometimes
core-js required complete immersion for weeks. I thought that proper polyfilling is required for the community and money was not my priority.
I left a high-paying job and did not accept some very good options because in those positions I did not have the opportunity to devote enough time to work on open-source. I started to work on open-source full-time. No one paid me for it. I hoped sooner or later to find a job where I could fully dedicate myself to open-source and web standards. Periodically, I earned the money, required for life and work on FOSS, on short-term contracts. I returned to Russia, where it was possible to have a decent standard of living for relatively small money. One more mistake - as you will see below, money matters.
Until April 2019, for about one and a half years as a whole and about a half-year full-time without distraction to any other work, I worked on the
core-js@3 with a fundamental improvement of polyfilling-related Babel tools, the basement of the toolkit generation that now is used almost everywhere.
Shit happened 3 weeks after the
core-js@3 release. One April night, at 3 AM, I was driving home. Two deadly drunk 18-years-old girls in dark clothes decided somehow to crawl across a poorly lit highway - one of them lay down on the road, another sat down and dragged the first, but not from the road - directly under my wheels. That’s what the witnesses said. I had absolutely no chance to see them. One more witness said that before the accident they were just jokingly fighting on the road. Nothing unusual, it’s Russia. One of them died and another girl went to a hospital. However, even in this case, according to Russian arbitrage practice, if the driver is not a son of a deputy or someone like that, he almost always will be found guilty - he should see and anticipate everything, and a pedestrian owes nothing to anyone. I could end up in prison for a long time, IIRC later the prosecutor requested 7 years.
The only way not to end up in prison was reconciliation with “victims” - a standard practice after such accidents - and a good lawyer. Within a few weeks after the accident, I received financial claims totaling about 80 thousand dollars at the exchange rate at that time from “victims’” relatives. A significant amount of money was also needed for a lawyer.
Maybe it’s not inconceivable much money for a good software engineer, but, as I wrote a little above, for a long time I worked on the
core-js@3 release full time. Of course, no one paid me for this work, and I completely exhausted all my financial reserves, so, sure, I hadn’t such money and I didn’t have a chance to find the required money from available sources. The time I had was running out.
By that time
core-js already was used almost as widely as it’s now. As I wrote above, for a long time I looked for contributors for
core-js without any success. However,
core-js is a project that should be actively maintained and it can’t stay just frozen. My long-term imprisonment would have caused problems not only for me - but it’s also the death of
core-js and problems for all who use it - for half of the Web. The notorious bus factor.
Some months before that, I started raising funds to support the
core-js development (mainly it was posted READMEs on GitHub and NPM). The result was… $57 / month. Fair pay for full-time work on ensuring compatibility for the whole web 😂
I decided to make a little experiment - to ask for help from the
core-js users - those who will suffer if
core-js will be left without maintenance. I added a message on
I understood that hardly I’ll get all the required money on donations, however, every dollar mattered. I added a job search message to get a chance to earn another part. I was thinking that some lines in the NPM installation log asking to help, which can be hidden if it’s required, is an acceptable price for using
core-js. The original plan was to delete this post in a few weeks, but everything went against the plan. How wrong I was about people…
Of course, I expected that someone would not like to see a request for help in their console, but the continuous stream of hate that I began to receive went through the roof. It was hundreds of messages, posts, and comments for a day. All this can be reduced to something like:
This is far from the funniest thing I’ve seen - if I wanted to, I could collect a huge selection of statements in the style collected here - but why? I already have enough negativity in my life.
Developers love to use free open-source software - it’s free and works great, they are not interested in that many and many thousands of hours of development, and real people with their own problems and needs are behind it. They consider any mention of this as an invasion of their personal space or even a personal affront. For them, these are just gears that should automatically change without any noise and their participation.
So, thousands of developers attacked me with insults and claimed that I have no right to ask them for any kind of help. My request for help offended them so much that they began to demand restricting my access to the repository and packages and move them to someone else like it was done with
left-pad. Almost no one of them understood what
core-js does, the scale of the project, and, of course, no one of them wanted to maintain it - it should do “the community”, someone else. Seeing all this hatred, in order not to be led by the haters, I did not delete the help-asking message, that initially planned to add only for a couple of weeks, just out of principle.
What about companies for whom
core-js helped and helps to make big money? It’s almost all big companies. Let’s rephrase this old tweet:
Company: “We’d like to use SQL Server Enterprise”
MS: “That’ll be a quarter million dollars + $20K/month”
Company: “We’d like to use core-js”
core-js: “Ok! npm i core-js”
core-js: “Would you like to help contribute financially?”
Company: “lol no”
A few months later, tired of user complaints, NPM presented
npm fund - it was not a problem solution, it was just a way to get rid of those complaints. How often did you call
npm fund? How often did you pay donations to someone who you saw in
npm fund? Who did you see and support at first -
core-js or one who maintains a dozen of one-line libraries dependent on each other? It also provided NPM a perfect justification for the future step (read below).
Within 9 months many thousands of developers, including developers of projects fundamentally dependent on
core-js, knew about the situation - but no one offered to maintain
core-js. Within many months I talked with maintainers of some significant projects dependent on
core-js, but without any success - they hadn’t required time resources. So I was forced to ask some of my friends who were not related to FOSS community (at first @slowcheetah, thanks him for his help) to cover me and at least try to fix significant issues until I get free.
Few users and small companies supported the
core-js - and I am very grateful to them. However, within 9 months it was collected only about 1/4 of the money that should have been collected within a couple of weeks to change something.
During the same time, despite everything, the number of
core-js downloads for a day almost doubled.
In January 2020 I ended up in prison.
I don’t wanna say many words about prison and I have no great desire remembering this. It was slave labor at a chemical factory where my health was significantly ruined and where I 24/7 had a great time in a company of drug dealers, thieves, and killers (from other regimes), without access to the Internet and computers.
After about 10 months, I was released early.
I saw dozens of articles, hundreds of posts, and thousands of comments the essence of many of which can be expressed by this:
What do you think I did? Of course, I made the same mistake. I saw some people who supported the development of
core-js, many issues, questions, and messages - sure, not so much like angry comments.
core-js became even more popular and was already used by almost the same percentage of websites as it is now.
Ensuring compatibility for the whole Web again
I returned to
core-js maintenance like it was before. Moreover, I completely stopped being distracted by contracts and any other work in favor of working on
core-js had some money on funding platforms - not so much, many times less than I received before starting work on
core-js full-time - but for me alone it was enough for life. A kind of down-shifting, full-time Open-Source for making the world better… I didn’t think about the tens of thousands of dollars in lawsuits left over from the accident. I didn’t think about my future. I thought about a better future for the Web. And, of course, I was hoping that some company would offer me a position with the opportunity to work on web standards and would sponsor my work on polyfills and FOSS.
A lot has been accomplished over the next two years - in terms of work, almost the same as in the previous 8 years. This is still
core-js@3 - but much better. However, the changelog and even the previous diff reflect only a few percent of the work done. Almost all of this work remains in the shadows, not visible to the average user.
This is a fundamental work with standards and proposals. As a side effect of this work, taking into account the hard work that was done and changes after my feedback and suggestions, I consider many of the ECMAScript proposals that have become part of the language my achievements as much as they are achievements of their champions. This is work with engines and their bug trackers in searching for bugs. This is constant automatic and (too often) manual testing in many hundreds of environments, many thousands of environments / builds / test suites combinations for ensuring proper work of standard library everywhere and collecting compat data. From a raw prototype, made in a couple of days,
core-js compat data became exhaustive data set with proper external and internal tooling. This is the design and prototyping of many features that are yet to appear in the project. And also much, much more.
As you can see above,
core-js is greater than the number of web page openings in Safari and Firefox. Thus, from a certain point of view,
During work on
core-js is the best playground for experimentation with ECMAScript proposals. In too many cases, proposals have feedback from other users after they play with experimental
core-js implementations of proposals.
core-js case. Instead of this, too often, I see ignore of my or
core-js issues or even making roadblocks by TC39 members and they even don’t hide it:
After a while, “support” came from NPM. In
npm@7 which was released at the end of 2020 as a logical continuation of
npm fund was disabled console output in post-install scripts. The result was expected - because people stopped seeing the funding request and almost no one uses
npm fund, the number of
core-js backers began to decline. Excellent support for the project from those who not only earn by distributing my work but also use it themselves -)
In addition, another factor came into play again. Higher quality - less support. Is the library well-maintained? There are practically almost no open bug reports, and if they happen, is it fixed almost instantly? Does the library already give us almost everything we want? Yes? So why should we support the maintenance of the library? The price at which this is done for the maintainers is not on the surface - for the most, it’s still just “a small library”. Many of those who backed
core-js before stopped doing it.
core-js code contains my copyright. As you can see at the top of this post, it’s present in about half of all websites. Regularly someone finds it in the source code of harmful sites / applications - but they don’t know what is
core-js and their tech level is not good enough even for finding out it. After that, the police call and threaten me, and someone even tries to blackmail me. Sometimes it’s even not funny.
I have been contacted several times by American and Canadian journalists who discover
core-js on American news and government websites. They were very disappointed that I was not an evil Russian hacker who meddles in American elections.
The endless stream of hatred decreased slightly over time but continued. However, most of it moved from something like GitHub issues or Twitter threads to my mail or IM. Today, one developer wrote to me a message. He called me a parasite on the body of the developer community that makes a lot of money spamming and doing nothing useful. He called me the same murderer as Hans Reiser, but who bought the judge and went unpunished. He wished death on me and all my relatives. And there is nothing unusual here, I get several such messages a month. In the last year, this has been added that I am a “Russian fascist”.
Some words about war
Open-source should be out of politics.
I don’t want to choose between two kinds of evil. I will not comment on this in more detail, since there are people close to me on both sides of the border who may suffer because of this.
Let me remind you what I wrote about above - I returned to Russia because it was a place where it was possible to have a decent standard of living for relatively small money and concentrate on FOSS instead of making money. Now I cannot leave Russia, because after the accident I have outstanding lawsuits in the amount of tens of thousands of dollars and I am forbidden to leave the country until they are paid off.
What do you think, how much money
core-js earn for a month?
When I started to maintain
core-js full-time, without being distracted by contracts and any other work, it was about $2500 for a month - it was about 4-5 times less than I usually had on full-time contracts. Remember, a kind of down-shifting, to make the Web better. Temporarily. Reduce issues and bugs to zero, make the highest quality product, which is used by almost everyone - and the project will be sufficiently supported, right? Right?
After a few months, monthly repeat was decreased to about $1700 (at least that’s what I thought) - $1000 via Tidelift, $600 via Open Collective, and $100 via Patreon. In addition to the monthly repeat, one-time donations periodically come - on average it was maybe $100 for a month.
Crypto? Adding a crypto wallet for donations was a very popular request. However, for all the time, only 2 transfers for a total amount of about $200 have been received on crypto wallets, the previous one was more than a year ago. GitHub sponsors? It’s not available for Russia and never was. PayPal? It’s banned for Russians. When it was available,
core-js received about 60$ for all time. Grants? I applied for a lot of grants - all were ignored.
The main part, $400 for a month, from those donations,
core-js had from… Bower, another FOSS community. I am also very grateful to all other sponsors - because of your donations, I’m still working on this project.
However, in this list there is no one big corporation or at least a company from the top 1000 website list. Let’s be honest - there are mainly individuals, and only a few small companies on the current list of backers and they pay a few dollars a month.
If someone will say that they don’t know that
core-js requires funding… Come on, I regularly see memes like this:
A year ago, Tidelift stopped sending me money. They said that because of the political situation, the Hyperwallet that they used is no longer available for Russians (but it was available for me till the previous month when I tried to update some personal data), and for safety, they will store my money on their side. Within a previous few months, I tried to get this money to a bank or a Hyperwallet account but received only replies that they will try to do something (sounds great, doesn’t it?). From the end of the previous year, they just stopped responding to emails. And now, I’ve got this:
In such an amusing way, I found out that I will not receive the money for the previous year, and this year I worked not for $1800, but for $800 a month. There were, of course, no replies to subsequent emails. However, their site indicated that I received and still receive money through them.
I wonder how the companies that support their dependencies chain through them will react to such a scam.
At the same day, on OpenCollective I saw that the monthly repeat was reduced from about $600 to about $300. Apparently, the financial reserves of
Bower have come to an end. That means that for this month I’ll get about $400 at all.
In the previous months, I measured how much time takes the work on
core-js. It turned out about… 250 hours a month - significantly more than a full day without any days off, which makes it impossible to have “real” (as many say) full-time work or work any significant contracts. $400 for 250 hours… It will be less than $2 per hour of work, for the previous year a little more - $4 per hour. Yes, in some months, I spend less time working on the project - however it does not change much.
So much willing to pay for ensuring compatibility for the whole Web. And no insurance or social security.
Awesome earning growth and career, right?
I think you understand well how much senior software engineers in key IT companies get paid. I received a lot of comparable offers, however, they are not compatible with the proper work on
Among the regular threats, accusations, demands, and insults, I often get something like “Stop begging and go to work, idler. Remove your beggarly messages immediately - I don’t wanna see them.” The funny thing is that at least some of these people get over $300,000 a year (which I know for sure because I talk to their colleagues), and (because of the nature of their work)
core-js save them many hours of work each month.
When I started working on
core-js, I was alone. Now I have a family. A few over a year ago, I became a father of son. Now I have to provide him with a decent standard of living.
I have a wife and sometimes she wants some new shoes or a bag, a new iPhone or Apple Watch. My parents are already at the age that I need to significantly support them.
I think it is obvious that it is impossible to properly support a family with the money that I have or had from
core-js maintenance. Financial reserves I used, have finally come to the end.
More and more often I hear reproaches like: “Give up your Open-Source, this is pampering. Go back to a normal job.
%USERNAME% works as a programmer for just a year. He understands almost nothing about it, works a couple of hours a day, and already earns times more than you.”
I’m damn tired. I love working on open-source and
core-js. But who or what am I doing this for? Let’s summarize the above.
- I ensure no compatibility issues and provide bleeding edge features of the web platform for most of the Web from 2014 and I’m working on it almost all my time for money that now will not be enough even for food.
- Instead of any gratitude, all I see is huge hatred from developers whose life I simplify.
- Companies that save and earn many millions of dollars on
core-jsusage just ignore
- Even in a critical situation, on an ask for help, instead of help, most of them preferred to ignore or hate.
I don’t care about the haters. Otherwise, I would leave open-source a long time ago.
I can tolerate the lack of normal interaction with the standards developers. First of all, it’s future problems for users and, when the Web will be broken, for standards developers themself.
However, money matters. I’ve had enough of sponsoring corporations at the expense of my and my family’s well-being. I should be able to ensure a bright future for my family, for my son.
The work on
core-js takes almost all my time, more than a full working day. This work ensures the proper work the most of the popular websites and this work should be paid properly. I’m not going to keep working for free or for $2 per hour. I’m willing to continue working on a project at least for $80 an hour. This is the money that have, for example,
eslint team members for an hour. And, if work on open-source requires it, I’m ready to pay off my lawsuits and leave Russia - however, it’s not cheap.
Regularly I see comments like this:
Ok guys, if you want it - let’s use such an approach.
Depending on your feedback,
core-js will soon follow one of the following ways:
Appropriate financial backing
I hope that at least after reading this post corporations, small companies, and developers will finally think about the sustainability of their development stack and will properly back
core-jsdevelopment. In this case,
core-jswill be appropriately maintained and I’ll be able to concentrate on adding a new level of functionality.
The scale of the necessary work goes through the roof, one of me is no longer enough - I can’t work more physically. Some work, for example, improving test coverage or documentation, is simple enough and takes a lot of time, but it’s not the kind of work that volunteers want to do - I don’t remember any PRs with improving test coverage of existent features. So it makes sense to attract at least one or two developers (at least students, better - higher level) on a paid basis.
Taking into account the involvement of additional maintainers and other expenses, I think that at this moment about 30 thousand dollars for a month could be enough. More money - better product and faster development. A couple times less - it makes sense to resume the work on
core-jsfull-time alone - sure, not as productive as it could be with a team.
I may be hired by a company where I will be able to work on Open-Source and Web standards
and that will give me the resources required for continuation of work.
core-jswill become a commercial project if it will not have appropriate support from users
It’s problematic to create a commercial infrastructure around the current
core-jspackages, so most likely the new
core-jsmajor release will change the license. The free version will be significantly limited. All extra functionality will be paid for.
core-jswill continue to evolve appropriately and, in the scope of this project, will be created many new tools for ensuring web compatibility. Sure, it will significantly reduce the spread of
core-jsand will cause problems for many developers, however, even some paying customers could be enough and my family will have money for paying bills.
Slow death in case I’ll see that
core-jsis not required
I have many ideas for commercial projects, I have a lot of good job offers - all this takes time, which now goes to
core-jsmaintenance. It does not mean that I’ll immediately completely stop to maintain
core-js- I’ll just maintain pro-rata donations. If they are at the current level, it will be only a few hours of maintenance a month instead of hundreds now. The project will stop the upgrowth - maybe minor bugs will be fixed and compatibility data will be updated - this time is not enough for more. After a while,
core-jswill become just useless and will die.
I still hope for the first outcome since
core-js is one of the key components of the modern digital infrastructure, but, looking at the current and the past, I am mentally getting ready for other options.
I will answer in advance some angry comments that I see regularly and that will definitely be after this post:
“Not a problem, we will just pin the
Unlike most projects,
core-jsshould be on the bleeding edge since
“It’s open-source, we will fork it, fuck off.”
I see such comments regularly, someone even tries to scare me with a fork. I’ve said already too many times that if someone will fork and properly maintain
core-js, I’d be happy - it makes no sense just to fork it without maintenance. Now I don’t see even anyone who tries to add to
“We don’t need
core-js, many alternative projects are available.”
Nobody is holding you. But where are those alternatives in real life? Sure,
core-js, and it’s not unreasonable - all of them provide only a little part of
core-jsfunctionality, they are not enough proper and complex, the number of cases where they can be used is significantly limited, they can’t be properly integrated into your project in such simple way and have other significant problems. In case of the existence of proper alternatives, I would stop working on
core-jsa long time ago.
“We can drop IE support, so we no longer need polyfills.”
“You are an asshole, we will expel you from the FOSS community.”
Now let’s move away from the negative to the second half of this post where we will talk about things that would be nice to implement in
core-js and problems of polyfilling at all.
core-js modules were required for all browsers has gone. The latest browsers have good standards support and, with common use cases, they need only some percentage of the
core-js modules for the most recent language features and bug fixes. Some companies are already dropping support for IE11 which recently was “buried” once more. However, even without IE, old browsers will always be, bugs will happen even in modern browsers, and new language features will appear regularly and anyway they will appear in browsers with a delay - so, if we want to use modern JS in development and minimize possible problems, polyfills stay with us for a long time, but they should evolve.
Here I will not (almost) write about adding new or improving existing specific polyfills (but, sure, it’s one of the main parts of
core-js development), let’s talk about some other crucial moments without focusing on minor things. If it is decided to make a commercial project from
core-js, the roadmap will be adapted to this condition.
I am trying to keep
core-js as compact as possible, but one of the main conceptions that it should follow is to be maximally useful in the modern web - the client should not load any unnecessary polyfills and polyfills should be maximally compact and optimized. Currently, a maximal
core-js bundle size with early-stage proposals is about 220KB minified, 70KB gzipped - it’s not a tiny package, it’s big enough - it’s like jQuery, LoDash, and Axios together - the reason is that the package covers almost the entire standard library of the language. The individual weight of each component is several times less than the weight of quite correct alternatives. It’s possible to load only the
core-js features that you use and in minimal cases, the bundle size can be reduced to some kilobytes. When
core-js is used correctly, this is usually a couple of tens of kilobytes - however, there is something to strive for. Most pages contain pictures larger even the full
core-js bundle, most users have Internet speed in dozens of Mbps, so why is this concept so significant?
In too many places, mobile Internet is not perfect and still 3G or even 2G. In the case of 3G, downloading of one full copy of
core-js can take a couple of seconds. However, pages contain more than one copy of
core-js and many other duplicated polyfills too often. Some (mainly mobile) Internet providers have very limited “unlimited” tariff plans and after some gigabytes reduce the speed to some Kbps. Internet speed is also often limited for many other reasons.
The speed of the page loading equals the revenue.
Illustration is from a random post by googling
The size of
core-js is constantly growing because of adding new or improving existing polyfills. This issue also is a block for adding some big polyfills - adding
Temporal, and some other features to
core-js could increase the maximal bundle size a dozen times to some megabytes.
One of the main
core-js killer features is that it can be optimized with the usage of Babel, SWC, or manually, however, current approaches solve only a part of the problem. To properly solve them, the modern web requires a new generation of the toolkit that could be simply integrated into the current development stack. And in some cases, as you will see below, this toolkit could help to make the size of your website pages even less than just without
I already wrote about some of this in
core-js@3, Babel and a look into the future post, but that were just raw ideas. Now it’s in the stage of experimentation or even implementation.
Since the future of the project is in doubt, it makes no sense to write any specific dates here, I do not promise that all of this will be done shortly, but this is what should be strived for.
New major version
core-js@3 was released about 4 years ago - it’s too much. It’s not a big problem for me to add some breaking changes (rather ensuring backward compatibility is often a challenge) and to mark a new version as a major release - it’s a big problem for users.
At this moment, about 25% of
core-js downloads are critically obsolete
core-js@2. Many users wanna update it to
core-js@3, but because their dependencies use
core-js@2 they still use the obsolete version for avoiding multiple copies (I saw such issues on GitHub in too many projects). Too frequent major updates would worsen such cases even more.
However, it’s better not to get too obsessed with compatibility with older versions. The library contains too much that’s not removed only for compatibility reasons. The absence of some long-needed breaking changes for someone will negatively affect the future. Judging by how the standards, the ecosystem, and the Web change, and how legacy accumulates, it’s better to release a new major version each 2-3 years.
Adding all the new things that we would like to see in the new major version would take many years, which is unacceptable. However,
core-js follows SemVer and makes sense to release a new major release at first with breaking changes (some of them below), most of the new features can be added in minor releases. In this case, such a release can take just about 2-3 months of full-time work and it can be the first
core-js version that reduced the size compared to the previous -)
core-js package directly
Drop critically obsolete engines support
IE is dead. However, not for all - for many different reasons, someone is still forced to make or maintain websites that should work in IE.
core-js is one of the main tools that makes life easier for them.
At this moment,
core-js tries to support all possible engines and platforms, even ES3 - IE8-. But only a small part of developers using
core-js needs support of ES3 engines - at this moment, the IE8- segment of browsers is about 0.1%. For many other users, it causes problems - bigger bundle size and slower runtime execution.
The main problem comes from supporting ES3 engines: most modern ES features are based on ES5 features, which aren’t available in those old engines. Some features (like getters / setters) can’t be polyfilled, so some polyfills (like typed arrays) can’t work in IE8- complete. Some others require heavy workarounds. In case you need to polyfill only some simple features, the main part of
core-js size in the bundle is the implementation of ES5 methods (in the case of polyfilling a lot of features, it’s only some percent, so this problem is related mainly to minimalistic bundles).
Even simple replacing internal fallbacks of ES5 features to implementations to direct usage of those native features reduces minimalistic
core-js bundle size 2+ times. After reworking the architecture, it will be reduced even more.
The IE9-10 segment of browsers already is also small - at this moment, the same 0.1%. But it makes no sense to consider dropping their support without dropping support of some other obsolete engines with similar or even greater restrictions, for example, Android 4.4.4 - in total, it’s about 1%. Raising the lower bar higher than ES5 is a more difficult decision at least because of some non-browser engines. However, even dropping IE11 support in the future will not give so many benefits as dropping IE8- support now.
ECMAScript modules and modern syntax
At this moment,
core-js should get an ECMAScript modules version in the near future. But, for example, on NodeJS, ECMAScript modules are supported only in the modern versions - but on NodeJS
core-js should work without transpiling / bundling even in ancient versions, Electron still does not support it, etc., so it’s problematically to get rid of the CommonJS version immediately.
The situation with the rest of modern syntax is not so obvious. At this moment,
core-js uses ES3 syntax. Initially, it was for maximal optimization since anyway it should be pre-transpiled to old syntax. But it was only initially. Now,
core-js just can’t be properly transpiled in userland and should be ignored in transpiler configs. Why? Let’s take a look, for example, at Babel transforms:
- A big part of transforms rely on modern built-ins, for example, transforms which use
@@iteratorprotocol - but
Symbol.iterator, iterators, and all other related built-ins are implemented in
core-jsand absent before
- Another problem is transpiling
core-jswith transforms that inject
core-jspolyfills. Obviously, we can’t inject polyfills into the place where they are implemented since it is circular dependencies.
- Some other transforms applied on
core-jsjust break its internals - for example, the
typeoftransform (that should help to work with polyfilled symbols) breaks the
However, the usage of modern syntax in polyfills code could significantly improve the readability of the source code, reduce the size and in some cases improve performance if polyfill is bundled for a modern engine, so it’s time to think about rewriting
core-js to modern syntax, making it transpilable by getting around those problems and publishing versions with different syntax for different use cases.
Web standards polyfills
I’m thinking about adding the maximum possible web standards (not only ECMAScript and closely related features) support to
core-js for a long time. First - about the remaining features from the Minimum Common Web Platform API (what is it?), but not only about them. It could be good to have one bulletproof polyfills project for all possible web development cases, not only for ECMAScript. At the moment, the situation with the support of web standards in browsers is much worse than with the support of modern ECMAScript features.
One of the barriers preventing the addition of web standards polyfills to
core-js was significantly increasing bundles size, but I think that with current technics of loading only required polyfills and technics which you could see below, we could add polyfills of web standards to
But the main problem is that it should not be naive polyfills. As I wrote above, now the correctness of ECMAScript features almost everywhere is not very bad, but we can’t say it about web platform features. For example, a
core-js will be started only in case I have such resources.
New approaches to tooling are more interesting
Someone will ask why it’s here. What do tools, like transpilers, have to do with the
core-js is just a polyfill, and those tools are written and maintained by other people. Once I also thought that it is enough to write a great project with a good API, explain its possibilities, and when it becomes popular, it will acquire an ecosystem with proper third-party tools. However, over the years, I realized that this will not happen if you do not do, or at least not control, it yourself.
For example, for many years, instance methods were not able to be polyfilled through Babel
runtime, but I explained how to do it too many times. Polyfilling via
preset-env was not able to be used in real-life projects because of incomplete detection of required polyfills and a bad source of compatibility data, which I explained from the beginning. Because of such problems, I was forced to almost completely rewrite those tools in 2018-2019, for the
core-js@3 release, after that we got the current state of statically analysis-based tools for polyfills injecting.
I am sure that if the approaches below are not implemented in the scope of
core-js, they will not be properly implemented at all.
For avoiding some questions related to the following text:
core-js tools will be moved to scoped packages - tools like
core-js-compat will become
Not only Babel: plugins for transpilers and module bundlers
At this moment, some users are forced to use Babel only due to the need to automatically inject / optimize required polyfills. At this moment, Babel’s
runtime are the only enough good and well-known ways to optimize usage of
core-js with statical analysis. It happened historically because I helped Babel with polyfills. It does not mean that it’s the only or the best place where it could be done.
Babel is only one of many transpilers. TypeScript is another popular option. Other transpilers are gaining popularity now, for example, SWC (that already contains a tool for automatic polyfilling /
core-js optimization, but it’s still not perfect). However, why do we talk about the transpilers layer? The bundlers layer and tools like
esbuild (that also contains an integrated transpiler) are more interesting for the optimization of polyfills. Rome in development for several years and still is not ready, but its conception looks very promising.
One of the main problems with statical analysis-based automatic polyfilling on the transpiler layer is that usually not all files from the bundle are transpiled - for example, dependencies. If some of your dependencies need a polyfill of a modern built-in feature, but you don’t use this built-in in your userland code, this polyfill will not be added to the bundle. Unnecessary polyfills import also will not be removed from your dependencies (see below). Moving automatic polyfilling to the bundlers layer fixes this problem.
Sure, writing or using such plugins in many places is difficult compared to Babel. For example, without some extra tools now you can’t use plugins for custom transforms in TypeScript. However, there are always options and there would be a desire.
Automatic polyfilling / optimization of
core-js should be available not only in Babel. It’s almost impossible to write and maintain plugins for all transpilers and bundlers in the scope of the
core-js project, but it’s possible to do those things:
- Improve provided by
@core-js/compat) and tools for integration with third-party projects, they should be comprehensive. For example, “built-in definitions” are still on Babel’s side that causing problems with their reuse in other projects.
- Since some tools already provide
core-jsintegration, makes sense to help them, not only Babel.
- Makes sense to write and maintain plugins for some significant tools in the scope of the
core-jsproject. What? Will see.
One of the problems of the statical analysis-based automatic polyfilling on the files layer (
usage polyfilling mode of Babel
preset-env) was explained above, but it’s not the only problem. Let’s talk about some others.
Your dependencies could have their own
core-js dependencies and they can be incompatible with the
core-js version that you use at the root of your project, so injecting
core-js imports to your dependencies directly could cause breakage.
Projects often contain multiple entry points, multiple bundles, and, in some cases, the proper moving of all
core-js modules to one chunk can be problematic and it could cause duplication of
core-js in each bundle.
I already posted the
core-js usage statistics above. In many cases, you could see the duplication of
core-js - and it’s only on the first loaded page of the application. Sometimes it’s even like what we see on the Bloomberg website:
[Some time ago this number was even more.]https://raw.githubusercontent.com/freitzzz/cinderela/master/blog/general/core_js_whats_next/115339234-87e1f700-a1ce-11eb-853c-8b93b7fc5657.webp Of course, a such number of copies and various versions of
core-js is not something typical, but a situation with some copies of
core-js is too common and you could see it on about half of the websites with
core-js, so for preventing this required a way to collect all polyfills from all entry points, bundles and dependencies of the project in one place.
Let’s call a tool for this
@core-js/collector. This tool should take an entry point or a list of entry points and should use the same statical analysis that’s used in
preset-env, however, this tool should not transform code or inject anything, should check full dependencies trees and should return a full list of required
core-js modules. Require simple ways to integrate this into the current development stack. One of those ways can be a new polyfilling mode in plugins, let’s call it
collected - that will allow loading all collected polyfills of the application in one place and remove unnecessary (see below).
Removing unnecessary third-party polyfills
Now it’s typical to see, for example, a dozen copies of
Promise polyfills with the same functionality on a website - you load only one
Promise polyfill from
core-js, but some of your dependencies load
Promise polyfills by themself -
Promise polyfill from one more
native-promise-only, etc. But it’s just ES6
Promise which is already completely covered by
core-js - and available in most browsers without polyfills. Sometimes, due to this, the size of all polyfills in the bundle swells to several megabytes.
It’s not an ideal illustration for this issue, many other examples would have been better, but since above we started to talk about the Bloomberg website, let’s take a look at this site one more time. We have no access to the source code, however, we have, for example, such an awesome tool as
bundlescanner.com (I hope that the Bloomberg team will fix it ASAP, so the result could be outdated).
As shown in the practice, since such analysis it’s not a simple work, this tool detects only about half of libraries’ code. However, in addition to 4.5 hundred kilobytes of
core-js, we see hundreds of kilobytes of other polyfills - many copies of
whatwg-fetch (for the above reason,
core-js still does not polyfill it),
object-assign (it’s a ponyfill and about them the next section),
But how many polyfills were not detected? What’s the size of all polyfills that this website loads? It seems a couple of megabytes. However, even for very old browsers, maximally a hundred kilobytes more than be enough… And this situation is not something unique - it’s a too common problem.
Since many of those polyfills contain just a subset of
core-js functionality, in the scope of
@core-js/compat, we could collect data that will show if a module is an unnecessary third-party polyfill or not and, if this functionality is contained in
core-js, a transpiler or bundler plugin will remove the import of this module or will replace it to the import of suitable
The same approach could be applied to rid dependencies from old
Globalization of pure version polyfills / ponyfills
One more popular and similar issue is a duplication of polyfills from global and pure
core-js versions. The pure version of
babel-runtime is intended for usage in libraries code, so it’s a normal situation if you use a global version of
core-js and your dependencies also load some copies of
core-js without global namespace pollution. They use different internals and it’s problematic to share similar code between them.
I’m thinking about resolving this issue on the transpiler or bundler plugins side similarly to the previous one (but, sure, a little more complex) - we could replace imports from the pure version with imports from the global version and remove polyfills unnecessary for the target engines.
That also could be applied to third-party ponyfills or obsolete libraries that implement something already available in the JS standard library. For example, usage of
has package can be replaced by
lodash methods by related modern built-in JS methods, etc.
Loading the same polyfills, for example, in IE11, iOS Safari 14.8, and the latest Firefox is wrong - too much dead code will be loaded in modern browsers. At this moment, a popular pattern is a usage 2 bundles - for “modern” browsers that will be loaded if native modules are supported,
<script type="module">, and for obsolete browsers which do not support native modules,
<script nomodule> (a little harder in a practice). For example, Lighthouse can detect some cases of polyfills that are not required with the
esmodules target, let’s check the long-suffering Bloomberg website:
From the very beginning of work on
core-js, I’m thinking about creating a web service that gives only the polyfills that are needed for the requesting browser.
The availability of a such service is the only moment in which
core-js have lagged behind another project.
polyfill-service project from Financial Times is based on this conception and it’s a great service. The main problem with this project - it’s a great service that uses bad polyfills. This project polyfill only a little part of ECMAScript features that
core-js provides, the main part of polyfills are third-party and are not designed to work together, too many don’t properly follow specs, too raw or just dangerous for usage (for example,
WeakMap looks like a step-by-step implementation of the spec text, but the absence of some non-spec magic cause memory leaking and linear access time that makes it harmful, more other - instead of patching, fixing and reusage of native implementation in engines like IE11 where it’s available, but does not accept an iterable argument,
WeakMap will be completely replaced). Some good developers try to fix this from time to time, but polyfills themselves are given unforgivably little time, so it’s still too far from something that could be recommended for usage.
Creating such a service in the proper form requires the creation and maintenance of many new components. I work on
core-js alone, the project does not have the necessary support from any company, and the development is carried out with pure enthusiasm, I need to look for funds to feed myself and my family, so I have no time and other resources required for that. However, in the scope of other tasks, I already made some required components, and discussions with some users convinced me that creating a maximally simplified service that you could start on your own server could be enough.
We already have the best set of polyfills, the proper compatibility data, and the builder which already could create a bundle for a target browser. Already mentioned above
@core-js/collector could be used for optimization - getting only the required subset of modules, plugins for transpilers / bundlers - for removing unnecessary polyfills. Missed a tool for the normalization of the user agent and a service that will bind those components together. Let’s call it
How should it look in the perfect world?
- You bundle your project. A plugin on the bundlers side removes all polyfills import (including third-party, without global pollution, from your dependencies, etc.). Your bundles will not contain any polyfills.
- You run
@core-js/service. When you run it,
@core-js/collectorchecks all your frontend codebase, all your entry points, including dependencies, and collects a list of all required polyfills.
- A user loads a page and requests a polyfill bundle from the service. The service gives the client a bundle compiled for the target browser that contains the required subset of polyfills and uses allowed syntax.
So, with this complex of tools, modern browsers will not load polyfills at all if they are not required, old browsers will load only required and maximally optimized polyfills.
Most of the above are about minimizing the size of polyfills sent to the client - but these are just a little subset of the concepts that it would be good to implement in the scope of
core-js, however, I think that it’s enough for understanding that still required a huge work and this work could significantly improve web development. Whether it will be implemented in practice and whether it will be available as FOSS or as a commercial project is up to you.
This was the last attempt to keep
core-js as a free open-source project with a proper quality and functionality level. It was the last attempt to convey that there are real people on the other side of open-source with families to feed and problems to solve.
If you or your company use
core-js in one way or another and are interested in the quality of your supply chain, support the project:
Write me if you want to offer a good job on Web-standards and open-source.
Feel free to add comments to this post here.
Denis Pushkarev, February 14th 2023
#reads #denis pushkarev #oss #web #core-js