Toro Rosso’s Head of IT: data is the most important resource in Formula 1

There aren’t many companies that are expected to push the limits of technological development and put their products on display in front of thousands of spectators and half a billion TV viewers multiple times every year.

Scuderia Toro Rosso is one of them. It’s one of the eleven Formula 1 teams that wheel out their state of the art racing equipment from the factories to the global arena of Grand Prix competitions more than 20 times a year. Every racing weekend thousands of hours of research and development, engineering, and manufacturing are compressed into two hours of high-speed performance on the track. It’s a show of technological excellence, innovation and teamwork.

So what makes it all tick? Well, apart from the competitive spirit, love of sport and drive to be the best, there is a hidden side in Formula 1 — the IT department. As boring as it sounds, the truth is that IT is the glue that binds it all together. Without IT it’s impossible to design and manufacture anything in Formula 1. Not to mention adjusting and perfecting the car during the season.

Every Formula 1 team has multiple data centers with hundreds or even thousands of servers, generating massive amounts of data every day. Each data center is like a digital heart that keeps the system alive. And there is a small army of IT gurus ensuring this life-giving data never skips a beat.

So, what does it take to keep this complex system functioning? There is no better person to answer this question than Raffaele Boschetti himself.

Raffaele is the Head of IT at Scuderia Toro Rosso — an Italian Formula 1 team that recently started deploying Acronis data protection technology in their datacenters. Raffaele is one of the most experienced and visionary IT specialists in the industry. A Formula 1 veteran with three years with Toro Rosso, and many years prior to that with Ferrari F1 Team, he has helped Formula 1 teams achieve amazing results. He’d do anything for Formula 1. For example, at one stage he even contacted AMD to ask them to modify computer processor logic to increase computing power. Which they did! And everyone was a winner.

Raffaele, thank for finding time in your busy schedule to talk to Acronis. Could you give us an insight into what it takes to maintain the IT infrastructure for a Formula 1 team?

Formula 1 is a data-intensive and technologically dependent sport. To give you an idea of how important our role is, think about this: if the telemetry services go down at the race, the car needs to get off the track and return to the box immediately.

In our business, it’s important to avoid any single point of failure. We have to provide reliable services with exceptional performance and without any limitation.

At the same time, we are working within physical constraints and FIA (Formula 1 governing body) regulations. At the trackside, we cannot ask for the hardware we want. We have size and weight constraints related to our service equipment because the garage space is limited. So, it’s a challenge to implement mission-critical services in that environment.

There are a lot of aspects to consider. Our goal is to prevent and deal with issues before they affect our productivity. Discipline and procedures help us to achieve that.

So what would be your worst nightmare?

Our worst nightmare would be to lose our data. Data is the most important resource we have. To reproduce lost data would take an enormous amount of time, and it’s something that we can’t afford.

Even losing physical resources or the entire garage is not so bad compared to losing data. Racing cars need operational telemetry services and our job is to guarantee their availability. Telemetry services are based on a server that receives a live stream of data from sensors when the car is running. And you can have a car on the track even with limited physical infrastructure. As long as you have the pit wall, you can see what is happening at the car.

Carbon dust is another issue we have to deal with. Carbon is the primary material in Formula 1 cars, so there are a lot of carbon particles in the garage. Since carbon is superconductive, it’s not good for our computers. Together with high humidity in some countries, it’s not a very good mix.

And then there are limited resources associated with traveling. Suppose our race engineer is using a laptop and it breaks down during the race. To replace the hardware is one thing, we need to be able to quickly restore data that the engineer was working on. And restoring a full image backup from the day before is not enough because that engineer would need to access data he was working on an hour or a just a few minutes ago.

Our vision is to provide real-time data availability in all situations, no matter what the fault is. This is what makes a difference at the track. We have to deliver the right information, the right data, in the right time.

So, the data recovery speed is very important in these circumstances.

Exactly. We need to be ready. As I mentioned before, we have clear procedures and checklists.  So, when something goes wrong, we don’t have to reinvent the wheel. At the track, you don’t have time to think.

I’ve learned a lot from my experience at Ferrari. I worked there for ten years. Discipline and procedures make all the difference. The devil is in the detail. When you’re fighting for a top-five position, it’s the details that make the difference.

And what about your infrastructure in Faenza?

Our vision is to provide reliable services with exceptional performance with continuous data availability both in normal and disastrous conditions. With that in mind, we’ve built three data centers, splitting services among these data centers to guarantee complete redundancy of all mission-critical services.

One of our datacenters contains a CFD (computational fluid dynamics) computer cluster. The other two provide capacity for backup, backend servers, user management and other services. We are able to provide business continuity or disaster recovery solution at any time, depending on the requirements.

We also have redundant network connectivity, because without a network you can’t do anything. For example, our CAD workstations have to be connected to the PLM over the network. We are also connected to our wind tunnel infrastructure in the U.K. because we need to exchange files with our aerodynamics engineers. So, if you’re not connected, your productivity is limited.

This would mean that you have a very clear disaster recovery plan?

Yes. It comes down to eliminating any single point of failure. This applies to everything, even to our carbon cutting plotters. We don’t have to launch support calls and wait till the issue is resolved.

If the machine that is used to cut carbon fiber goes down, your production capacity is immediately reduced by up to 40 percent. Because we don’t have time to submit support calls and wait for the machine to be fixed, we keep spares of everything instead, allowing us to recover within minutes.

What do you do for backups?

Since we produce a lot of data, we also need a secure backup solution. Currently, we have three levels of protection: data snapshots, allowing users to retrieve data without asking support, disk backup in the second datacenter, and tape backup for long-term archiving.

We use tape storage because the FIA requires us to keep copies of our CFD calculations for two years. Everything is encrypted to protect stored data.

Backup is not our core business, so we’re working with other vendors and are looking forward to deploying Acronis technology.

What is the computational power of your CFD cluster and is there a limit on that capacity?

Yes. We have a limit. Formula 1 CFD clusters must be capped at 25 teraflops. The limitation was imposed by the FIA since 2009. The goal was to limit the cost and investment in CFD infrastructure and get the teams aligned in terms of computation power. This is to prevent bigger teams from taking advantage of their better financial position, to make the playing field even for all.

CFD, or computational fluid dynamics, is used to perform a range of mathematical computations to model and predict how external conditions will affect a car’s performance on the track. Engineers use the CDF cluster extensively during the design stage, to test new car parts and aerodynamics in a virtual environment. CFD enables engineers to assess the car performance before the parts are even made.

Does it provide sufficient power to perform the simulations required to design a new car?

We are trying to get the most out of it. We’ve tuned it up quite a bit and we also revised our methodology. Our CFD simulations are automated — engineers submit a job for processing and at the end just get a result. The system is also heavily monitored to avoid potential downtime.

In today’s world, 25 teraflops is not a huge amount of power. It’s not enough. There are talks about allowing more power in 2018, but still, it will never be enough.

And now, the question that everybody asks: If data and technology are so important in Formula 1, then where is the human side of Formula 1? You may as well remote-control your cars on the track!

You can’t replace the human brain. Computers are very good in computation and managing large volumes of data. But let’s not compare the computing capacity of a computer with our brain.

When we talk about imagination and intuition, there is nothing better than a human brain (so far!). While we already have Artificial Intelligence and machine learning, computers still need to be trained for a specific environment.

Computers are great for increasing productivity. For example, you may have software to calculate your production perfectly, so that you know what you need to do to achieve a certain goal. But you still need to use your brain to understand that what is applicable in one situation may not be useful in another. Even though computers can give you good information, with your intuition and expertise you can revise and even reject it. It is helpful, but it’s not the final solution.

Thank you, Raffaele, and all the best for 2017!

Thank you!