The Fake Chip Scourge
For today’s newsletter I want to first feature Tiny Tapeout, a nifty little project that I think more people should know about.
Matt Venn first reached out to me a few years back and we have stayed in touch since then. He is a science communicator who picked up an interest in open source tools and software for semiconductors.
While he was doing some open source FPGA stuff, he came across a talk by Tim Edwards at EFabless about open source chips. The notion of an ASIC chip made by fully open source tools was really intriguing to him and so he managed to craft a tool chain to produce chip designs end to end.
Tiny Tapeout is the culmination of all those learnings. It teaches you everything you need to produce your own chip design and make it into reality. You don’t need to download or install anything, so the barrier to learning is quite low. You will be surprised by the endless creativity of these designs - some of which are done by people as young as 15.
Venn is a friend and I want more people to check this stuff out and support these little projects. So go check it out! And do it fast, the deadline is April 24th. Use code “Asianometry” to get $25 off your order (not sponsored).
In 2020, an ejection seat for an F-16 malfunctioned, causing the death of its pilot.
A subsequent investigation of one of the seat's essential components - the sequencer - discovered that several microchips inside were counterfeit, or fake.
It remains unclear what role those chips played in this specific incident. The sequencer in question for instance was in need of maintenance.
However, it is indicative of the significant risk fake chips present in critical situations.
In this video we look at the semiconductor industry’s counterfeit chip problem.
What is Counterfeit?
A counterfeit semiconductor has a broad range of definitions. A memo from the Department of Defense probably has the simplest two definitions:
One, a previously used electronic part that is sold as new;
And two, unauthorized copies of an authentic electronic product;
I think that is pretty good and covers a majority of incidents. But there are always edge cases so the industry has a much more comprehensive taxonomy. A counterfeit semiconductor is counterfeit because ...
One, it is an unauthorized copy;
Two, it does not conform to the original manufacturer's design, model, or performance standard;
Three, it is not produced by the original manufacturer or its authorized contractors;
Four, it is off-specification, already used, defective, or misleadingly labeled as "new";
And five, it is incorrectly marked or with wrong documentation.
A Big Business
We know that counterfeit semiconductors are a big business. But how big? Illegal doings are always hard to track. So most data you come across are rarely up to date.
Most organizations talk about it as part of a larger piracy and counterfeit trend. In 2019, the OECD estimated that the global volume of counterfeit and pirated products amounted for up to 3.3% of total global trade, or over $500 billion.
While there is a government counterfeit parts database called Government-Industry Data Exchange Program or GIDEP, it is not anonymous so counterfeit victims tend to shy away from reporting there.
An alternate data source is the Electronic Resellers Association International, a corporation that monitors incidents and issues reports. Their database tracks incidents reported to them by buyers.
For instance in 2022 they noted 504 reported semiconductor counterfeit cases in 2021, a sharp drop from 963 in 2019.
Another data point, though an older one. In November 2011, a US congressional committee presented its investigation findings on counterfeit electronics parts in the military supply chain.
They quoted an estimate from the Semiconductor Industry Association saying that counterfeiting costs the American semiconductor manufacturing industry over $7.5 billion a year in lost revenue and 11,000 jobs.
In 2010, the Bureau of Industry Security under the Department of Commerce published a report.
In it they identified 7,114 detected counterfeited microcircuit incidents across 387 companies within the defense industry in 2008. Just three years earlier in 2005, there were 3,040 reported incidents.
The types of counterfeited semiconductors span the board from microprocessors to analog chips to memory to radio frequency chips. The chips’ resale value ranged from as low as 11 cents to as high as $500 but most were in the 11 cents to $10 range.
The Poseidon FPGA Incident
During that 2011 Hearing, the Senate panel ran through several incidents of counterfeit semiconductors that got into the military's supply chains.
One such incident involved the P-8A Poseidon, which is a plane. It flies and whatever. Inside the plane is something called an ice-detection module, which warns pilots about ice forming on the aircraft.
The component controlling the module identified itself as a Xilinx FPGA unit. But this was a lie. What was inside was a counterfeit.
Boeing only found out after the plane took off, the part literally fell out of its socket and started rattling around inside the module.
BAE, the module's manufacturer, had bought 300 such FPGAs from a company in California. That company tested 50 but neglected to test the rest.
From there, investigators traced the counterfeit chip back to an affiliate of an electronics manufacturer in Shenzhen. Such incidents are not uncommon and the panel shared a few others too.
The Contractor's Dilemma
The Defense Department's contractors - as well as commercial system integrators in similar situations - find themselves in a particularly difficult spot.
They often have to use old parts for repair and maintenance. The F-15 plane was first introduced into service over 45 years ago. No way the supplier is going to be making those same parts for that long.
As I read someone else put it, it is like asking someone to build a NeXT Cube motherboard using modern circuit boards.
It is often that the original manufacturer hasn't been making those chips for years, leaving people with no choice but to go an oft-shady second-hand supplier. So their situation is understandable. Nevertheless, the US Government puts the onus of detecting and filtering out these counterfeit chips on them.
In January 2013, Obama signed into law the National Defense Authorization Act, which threatens contractors with civil and criminal penalties for this.
One notable case concerned a company called VisionTech in Florida, which knowingly sold fake chips as trademarked "military-grade" goods.
Per the indictment, VisionTech employees were instructed to tell customers that their chips came from Europe despite knowing that they really came from China and Hong Kong. And they also knowingly sent legitimate chips to customers when asked for samples, but sent fake chips for the purchased shipment.
The company made nearly $16 million in sales from these goods. The company's owner died before sentencing, but the company's Administrative Manager went to prison for 38 months for her role in the scheme.
Counterfeit Types: Recycled
Recycling is one of the most common ways how counterfeit chips enter the supply chain.
Here is the typical process. It starts with workers receiving scrap electronics. They crack open the casing and remove the printed circuit boards.
The circuit boards are then heated over an open flame to melt their solder and loosen the components' connections. The boards are then bashed against a hard surface to make them fall out.
The components then have their markings removed using various methods. After that they are sorted and re-processed. New coats and markings are applied to the chip to make them appear like as if new.
For the buyers, there is a lot that can go wrong. These e-waste components are old and worn-out as is. And then the process to harvest them subjects them to immense damage and stress. Not surprisingly, many of these straight up won't work when you try to use them.
Counterfeit Types: Relabeled
Relabeled - or re-marked - chips are chips that are not what they say they are.
Markings contain information about the chip's model, origin, certifications, and so on. Changing the markings misleads the users about their chip's capability and quality.
It is the semiconductor equivalent of making and selling a fake Louis Vuitton bag. A bad actor acquires a set of chips from some source, wipes away their markings, and then adds new markings to make them seem more premium or expensive.
This has real consequences where components cost more due to enhanced specifications. For example, military components where chips need to be hardened against radiation or temperature damage.
Suppliers missing the hardened, enhanced part might be tempted to swap in commercial parts that are cheaper and have similar functions, but do not meet the hardening requirements.
Counterfeit Types: Over-Production
Together, recycling and re-marking counterfeit make up about 80% of total counterfeit incidents. But there are other categories to consider as well.
Today, most semiconductor companies do not own their own fabs. They instead contract foundries to produce their chips. Most of these contract foundries are overseas and operate without third party oversight which can be taken advantage of.
Sometimes a bad foundry produces more of that chip that it was contracted to. This might be for yield reasons - having more good chips on hand so the foundry can claim to have a higher yield than what they actually have.
But they can also do this to resell their production as counterfeits. The foundry benefits from selling a valuable design - one that the design customer invested a great deal of R&D into - without needing to spend the R&D themselves.
The modern semiconductor manufacturing industry is one built on mutual trust. It is hard to believe that a reputable foundry would risk a multi-billion dollar reputation doing this sort of thing.
Even so, there are things that fabless manufacturers can do like split manufacturing for better assurance that their products are not being sold out the backdoor.
Counterfeit Types: Cloned
Another type of counterfeit chip are unauthorized clones. This is kind of like the foundry over-production situation, but start with a third party.
Such third parties clone chips so to save on the expensive R&D to develop their own designs. I am reminded of East Germany - and the larger Soviet Union in general - cloning chips and computers from the West.
The design IP for these clones can be obtained either through reverse engineering - like using an electron microscope to learn all the chip's various layers. This is harder if the chip is exceptionally complicated.
Or by illegally acquiring the chip's design IP. This can happen at all stages of the chip design and manufacturing process.
For instance, someone might copy RTL code at the design house so that they can then take home to turn into a foundry-ready GDSII file using their own tools.
Cloned counterfeits are harder to detect because they tend to be functionally similar to the real thing. But they aren't always as good. In practice, many cloned or reverse engineered chips deviate from the original in technology and process - which can lead to erratic behavior.
Detection
Oftentimes Fake chips get into products because the contractor did not have documented evidence of the chip's supply chain history and did not give the product a thorough inspection.
Like with the case of the Poseidon FPGA counterfeit incident, where the California-based semiconductor supplier should have more closely inspected the FPGAs they purchased from their Chinese supplier.
There are two types of methods of inspecting potentially counterfeit chips - physical and electric methods.
Physical methods are methods that look for defects directly related to the chip's physical properties. These include defects with the chip's exterior shipping or packaging, connections, IC packaging, bonding wires, or die.
For this we might use External Visual Inspection - called EVI - or more invasive methods like X-ray or even de-lidding the whole chip and exposing its innards. Since some of these methods are destructive, we can't always do all of them.
Electrical methods test for changes in the chip's component parameters - a parametric test. They might also test for whether or not the chip is functioning properly - a functional test. These tests tend to be quite effective, but are also time-consuming and expensive.
In the end, detecting counterfeits is a challenge of nuances. For instance, how do we distinguish normal physical defects on a chip's packaging from sanding procedures to blast away previous markings?
At the same time, we have to balance detection against financial and time concerns. So best practices dictate designing a set of tests that give us confidence that we have achieved a high enough test coverage rather than total assurance.
Avoidance
With die recycling such a significant problem and package re-marking getting harder to detect, fabless designers can add new features into their chip designs to help.
These are called design-for-counterfeit measures. A simple one is to give the chip a unique ID that can be easily read throughout its lifetime. This would be programmed into a non-programmable memory.
Another interesting anti-counterfeiting design is to create a unique fingerprint for the chip - called the physical uncloneable feature or PUF.
One example of this uses random variables within the semiconductor fabrication process to produce a circuit called an arbiter PUF. You send a signal through the arbiter circuit - your "challenge" - and it produces an output.
That output can be a string of numbers or whatever. The output just needs to be random but repeatable, like your Google Authenticator app for a 2-step authentication.
As with everything, counterfeiters can still tamper with these methods. Or these methods might be prohibitively difficult to implement for cost or technical reasons - for example, due to lack of space on a chip.
Side note. I want to note that I have seen some blockchain proposals using such PUFs. They take advantage of the blockchain's traceability and immutability to track a chip's supply chain history.
I am not going to comment on those other than these schemes need to work in concert with existing measures and require broad acceptance of the scheme by a concert of players.
E-Waste Incentive and Opportunity
Anyway. While these fancy methods can be effective, in the end most of these counterfeit chips - especially the recycled ones - enter the supply chain because the opportunity and incentive are there.
Worldwide in 2019, just 17.4% of the world's electronic waste was documented to be collected and recycled. These are often harvested to produce fake semiconductor components.
Harvesting and desoldering old chips from e-waste can cost as little as 10 cents per chip. And as I mentioned, certain types of chips can sell anywhere from $1 to $10 per chip.
So a fake, recycled chip can go for as much as 100 times the cost of harvesting them! This is a significant amount for any laborer.
Combine that with massive imports of improperly managed e-waste from developed areas like the United States and Europe, and that is how you get a counterfeit chip problem.
Conclusion
Our modern world relies on our unheralded semiconductors to work properly. Whether it be a soldier and his equipment, or a component in a plane, or a brake in a car, a chip that is not what it says it is can cause serious harm.
Throughout the 2010s, most fake recycled and relabeled chips came out of China because that was where most of the e-waste went.
Then in 2017, China banned the import of 24 types of solid waste including plastic and e-waste. Some e-waste is still being smuggled into China, but significant imports are redirecting to new countries.
For instance, Thailand saw a 5.7 times increase in the volume of e-waste imports in 2017 after the China bans were implemented - from 9,312 to 53,291 tons.
Other countries in Southeast Asia have seen similar effects and this will have big impacts on the counterfeit chip trade in the coming years.
The proper definition, management and disposal of such e-waste is not only vital to human health and the environment. But it also serves to help stem the fake chip scourge. It is worth pursuing.