Neural networks are getting closer to freelancers: AI has learned to perform every sixth commercial project

The development of generative artificial intelligence continues to change the remote work market. According to a new study, modern AI agents are already capable of independently executing part of real commercial projects at a level that satisfies customers. However, technology is still far from fully replacing professionals.

Such a conclusion was reached by specialists of the Center for AI Safety (CAIS) together with Scale Labs, the results of the study were published by The Decoder publication.

Remote Labor Index (RLI)

To evaluate the capabilities of modern models, researchers use the Remote Labor Index (RLI), a special indicator that measures how well artificial intelligence copes with real commercial tasks that are usually performed by freelancers.

In contrast to the usual tests, it is not the abstract capabilities of the models that are evaluated, but real projects for the execution of which customers are ready to pay money.

The study included 240 commercial orders with a total value of about $144,000 submitted by 358 verified freelancers.

Among the directions:

• 3D modeling and CAD;
• architectural design;
• graphic design;
• editing videos and creating animations;
• audio processing;
• data analysis;
• web development.

Each project was initially performed by a professional specialist, after which AI received a similar task. Independent experts then compared the results and determined whether the AI ​​work was of a quality that the customer would actually pay for.

The main indicator was the level of automation — the share of projects where the result of AI turned out to be comparable to human work.

Incredible AI capabilities

Even at the beginning of work on the index, the best AI could successfully complete only 2.5% of commercial tasks.

Now the Claude Fable 5 model became the leader of the rating, which reached 16.1%.

This means that approximately one in six real freelance projects can be completed by the model at a level that experts consider commercially acceptable.

Other systems are far behind:

• Claude Opus 4.8 – 8.3%;
• GPT-5.5 – 6.3%.

For comparison, the previous research leader – Claude Opus 4.6 – previously showed a result of only 4.17%.

The authors note that in just eight months, the maximum level of automation increased more than four times.

At the same time, the study turned out to be incomplete: due to the restrictions of the American authorities, the developers could not test 22 projects using Fable 5.

Even if we consider all these tasks as failures, the final score of the model would be 14.6%, which is still the best result among all the AI ​​tested.

The new model is not always the best

The study revealed another unexpected feature of the market.

Despite its later release, the Gemini 3 Pro showed one of the worst results among all test participants.

Its level of automation was only 1.25%, inferior even to much older language models.

According to the researchers, the release date of the model does not always reflect its ability to solve real professional tasks.

Where artificial intelligence is still seriously inferior to humans

While the overall progress was impressive, the study also showed that modern AI still routinely makes mistakes in complex professional projects.

For example, when designing jewelry, the Fable 5 model significantly outperformed previous generations of AI, but the final work still did not meet the level of an experienced designer.

Otherwise, GPT-5.5 successfully produced a good architectural rendering of the building, however, when checking the original 3D model, it became clear that the geometry of the object itself contained many serious defects.

In other words, artificial intelligence has learned to create a convincing visual picture, but so far it is not always able to correctly perform the technical part of the work.

Why AI still cannot objectively evaluate the work of other AI

The authors of the study decided to test another hypothesis — whether it is possible to abandon expensive human verification and entrust the evaluation of the results to the language models themselves.

The experiment was unsuccessful.

Practically all AI courts significantly overestimated the quality of the work of their “colleagues”.

For example:

• GPT-5.5 received scores almost three times higher than real experts;
• Opus 4.8 is about 2.5 times higher.

Although the models correctly determined the relative order of the ranking participants, their absolute estimates turned out to be very far from reality.

According to the researchers, the reason is simple: full verification of such projects requires work in specialized software.

It is not enough to look at a beautiful picture – it is necessary to open the source files in professional applications, check the correctness of the models, the structure of the project and the quality of execution in the same way as a real customer would do.

It is interaction with complex professional software that remains one of the weakest points of modern AI agents today.

A full-fledged developer workplace was used for AI tests

In order to maximize the capabilities of the models, the researchers did not run them in a browser, but in a full-fledged professional environment.

Each agent ran on a Linux virtual machine where more than 30 specialized programs were installed, including:

• Blender;
• GIMP;
• Audacity;
• other professional tools for creating digital content.

For the implementation of one project, models received up to 24 hours of computing time.

In addition, a double-check system was used: after the work was completed, the second AI agent acted as a demanding customer, found flaws and passed on comments to the first agent to finalize the result.

Full automation of freelancing is still a long way away

Despite the rapid growth of capabilities, researchers emphasize that modern models are not yet able to stably perform most professional projects without human participation.

Even the best results of Claude Fable 5 do not always correspond to the level of the finished commercial work.

However, the dynamics of development are impressive. If less than a year ago, artificial intelligence successfully coped only with single tasks, now it is already capable of qualitatively performing approximately every sixth commercial freelance project. According to the authors of the study, this rapid growth directly reflects the pace of automation of remote work and shows how rapidly AI is beginning to occupy certain niches in the market of digital professions.