Zero-Knowledge machine learning (ZKML) is a field of research and development that has been making waves in cryptography circles recently. But what is it and why is it useful? First, let's break down the term into its two constituents and explain what they are.
A zero-knowledge proof is a cryptographic protocol in which one party, the prover, can prove to another party, the verifier, that a given statement is true, without revealing any additional information beyond the fact that the statement is true. It is an area of study that has been making great progress on all fronts, from research to protocol implementations and applications.
The two main “primitives” (or, building blocks) that ZK brings to the table are the ability to create proofs of computational integrity for a set of given computations, where the proof is significantly easier to verify than it is to perform the computation itself. (We call this property “succinctness.”) ZK proofs also provide the option to hide parts of said computation whilst preserving computational correctness. (We call this property “zero-knowledge.”)
Generating zero-knowledge proofs is very computationally intensive, about 100 times as expensive as the original computation. This means that there are some computations for which it is infeasible to compute zero-knowledge proofs because the time it'd take to create them on the best hardware available makes them impractical. However, advancements in the field of cryptography, hardware, and distributed systems in recent years have allowed zero-knowledge proofs to become feasible for ever more intensive computations. These advancements have allowed for the creation of protocols that can use proofs of intensive computations, thus expanding the design space for new applications.
Zero-knowledge cryptography is one of the most popular technologies in the Web3 space since it allows developers to build scalable and/or private applications. Here are a few examples of how it is being used in practice (though note that many of these projects are works-in-progress):
As ZK tech matures we believe that there will be a Cambrian explosion of new applications since the tooling used to build them will require less domain expertise and will be a lot easier to use for developers.
Machine learning is a field of artificial intelligence (“AI”) that enables computers to automatically learn and improve from experience without being explicitly programmed. It involves the use of algorithms and statistical models to analyze and identify patterns in data, and then make predictions or decisions based on those patterns. The ultimate goal of machine learning is to develop intelligent systems that can adapt and learn on their own, without human intervention, and solve complex problems in various domains such as healthcare, finance, and transportation. Recently you may have seen advances in large language models like chatGPT and Bard or text-to-image models like DALL-E 2, Midjourney, or Stable Diffusion. As these models get better and better and are able to perform a wider variety of tasks, it will be important to know who performed these actions. Whether the action was performed by a specific model versus another, or whether it was performed by a human instead. We will explore this trail of thoughts in the upcoming sections.
We live in a world where AI/ML-generated content is becoming indistinguishable from content generated by humans. Zero-knowledge cryptography will allow us to make statements like: “a given piece of content C came out of model M applied to some input X.” We'd be able to verify that a given output was created by a large language model like chatGPT, a text-to-image model like DALL-E 2 or any other model that we create a zero-knowledge circuit representation for. The zero-knowledge property of these proofs would allow us to also hide parts of the input or the model as well if need be. A good example of this would be applying a machine learning model on some sensitive data where a user would be able to know the result of model inference on their data without revealing their input to any third party (e.g., in the medical industry).
Note: When we talk about ZKML, we are talking about creating zero-knowledge proofs of the inference step of the ML model, not about the ML model training (which, in and of itself, is already very computationally intensive). The current state of the art of zero-knowledge systems coupled with performant hardware still falls a few orders of magnitude short of being able to prove something as big as currently available large language models (“LLMs”), but there has been some progress in creating proofs of smaller models.
We did some research on the state of the art of zero-knowledge cryptography in the context of creating proofs for ML models and created an aggregation of the relevant research, articles, applications, and codebases that belong to this domain. Resources on ZKML can be found on the ZKML community's awesome-zkml repository on GitHub.
The Modulus Labs team recently released a paper titled “The cost of intelligence”, where they benchmark existing ZK proof systems against a wide range of models of different sizes. It is currently possible to create proofs for models of around 18M parameters in about 50 seconds running on a powerful AWS machine using a proving system like plonky2. A graph from this paper can be found below:
Another initiative that is working on improving the state of the art of ZKML systems is Zkonduit's ezkl library which allows you to create ZK proofs of ML models exported using ONNX. This enables any ML engineer to create ZK proofs of the inference step of their models and to prove the output to any correctly implemented verifier.
There are several teams working on improving ZK technology, creating optimized hardware for the operations that occur inside ZK proofs, and building optimized implementations of these protocols for specific use cases. As the technology matures, bigger models will be ZK-provable on less powerful machines and in a shorter period of time. We hope these advancements will allow new ZKML applications and use cases to emerge.
In order to decide whether ZKML could be used for a given application, we can examine how the properties of ZK cryptography would address an issue with machine learning. This can be illustrated as a Venn Diagram:
Definitions:
Here are a few examples of potential ZKML use cases:
During the second half of 2022, a few different teams and individuals working in the ZKML domain got together and created the ZKML community. It is an open community where its members discuss the latest research and experiments in the ZKML domain and share their findings. If you want to learn more about ZKML and start talking to people working in the field, it is a great place to ask questions and get familiarized with the topic. Also, check out the awesome-zkml resource aggregator!
dcbuilder.eth, and the Worldcoin Team