When Should a Black Box Be Transparent?

Dear KV,

We have been working with a third-party vendor that supplies a critical component of one of our systems. Because of supply-chain issues, they are trying to "upgrade" us to a newer version of this component, and they say it is a drop-in replacement for the old one. They keep saying this component should be seen as a black box, but in our testing, we found many differences between the original and the updated part. These are not just simple bugs but significant technology changes that underlie the system. It would be nice to treat this component as a drop-in replacement and not worry about this, but what I have seen thus far does not inspire confidence. I do see their point that the API is the same, but I somehow do not think this is sufficient. When is a component truly drop-in and when should I be more paranoid?