Meta Knowledge Condensation for Federated Learning
less than 1 minute read
Published:
Learning note for the paper: Meta Knowledge Condensation for Federated Learning.
阅读论文：Meta Knowledge Condensation for Federated Learning
一句话：local clients提取meta knowledge；server-side像普通的训练classifier一样，训练global model
1. 要解决什么问题
a. 目前FL需要多轮次的通讯成本，即使有了one-shot FL 但效果大幅下降。因此本文提出meta knowledge representation method 减少通讯次数。
b. FL中data heterogeneity issue要解决，如何保证clients不biased。 ## 2. 用了什么方法
a. Client：Dynamic weight assignment & knowledge sharing strategy。
i. 前者根据real data在model上的predict loss施加一个权重：the weight of each sample is inversely proportional to its prediction loss in the current model。也就是说，在更新distilled data时候，会更相信当下表现较好（loss较小）的sample。
ii. 后者很naive。假设所有clients不是同时开始的，存在时间差。那么，抛弃unconditional initialization（随机初始化），随机选用他人client在前一刻得到的syn sample作为初始化。减少了异质性。
b. Server-side：聚合clients的meta knowledge，然后像正常的一样训练。但为了缓解多个clients中数据的biases，引入了额外的synthetic training samples（pseudo meta knowledge）。
具体而言，对uploaded meta knowledge建模统计，然后从中sample即可（真的ok吗？）。就是用uploaded的数据拟合了一个generator。
c. Iterative：在central结束训练后，广播网络模型参数和meta knowledge。总体计算复杂度为O(N*n)。
d. 伪代码：
 ## 3. 效果如何
a. from 74.07% to 92.95% on MNIST with a restricted communication budget (10 rounds)
b. Results table:
 ## 4. 还存在什么问题&可借鉴之处
a. increasing the meta knowledge size can not necessarily improve final performance？ 原论文p12有一定的解读。
b. 写的还是比较清晰的。intro总体框架就是：快速给出FL在de-centralized manner的通病，给出自己的方法，总分说自己方法。最后说自己在几个dataset上的表现，然后在overall一下，分点说自己的贡献。但整个intro感觉重复了三遍，读来有点话痨了。
我觉得应该加一点FL和DD的点。
Homomorphic encryption aims at encrypting the tokens and when dealing with the ciphertext, the results stays the same as dealing with the tokens directly. So we say that it keeps homorphic.
If we define a operation $\bigtriangledown$, then to the Encoder and Decoder algorithm, satisfy:
Homomorphism comes from the field of algebra and generally includes four types: additive homomorphism, multiplication homomorphism, subtraction homomorphism and division homomorphism. Satisfying additive homomorphism and multiplicative homomorphism at the same time means that it is algebraic homomorphism, that is, Full Homomorphic. If all four homomorphisms are satisfied at the same time, it is called arithmetic homomorphism.
For computer operations, achieving full homomorphism means that homomorphism can be achieved for all processing. Homomorphism that can only implement some specific operations is called Somewhat Homomorphic.
Challenge
The problem of homomorphic encryption was first proposed by Ron Rivest, Leonard Adleman and Michael L. Dertouzos in 1978 (the same year Ron Rivest, Adi Shamir and Leonard Adleman also co-invented the RSA algorithm). But the first “fully homomorphic” algorithm was not presented and mathematically proven until 2009 by Craig Gentry in his paper “Fully Homomorphic Encryption Using Ideal Lattices”.
Algorithms that satisfy only additive homomorphism include Paillier and Benaloh algorithms; algorithms that satisfy only multiplicative homomorphism include RSA and ElGamal algorithms.
Homomorphic encryption is of great significance in the era of cloud computing and big data. At present, although cloud computing brings advantages including low cost, high performance, and convenience, from a security perspective, users are not afraid to put sensitive information directly on a third-party cloud for processing. If there is a more practical homomorphic encryption technology, everyone can use various cloud services with confidence, and various data analysis processes will not reveal user privacy. After the encrypted data is processed by the third-party service, the encrypted result is obtained. This result can only be decrypted by the user himself, and the third-party platform cannot obtain any valid data information in the whole process.
On the other hand, for blockchain technology, homomorphic encryption is also a good complement. Using homomorphic encryption technology, smart contracts running on the blockchain can process ciphertext, but cannot know the real data, which greatly improves privacy security.
Currently, fully homomorphic encryption schemes mainly include the following three types:
Scheme based on ideal lattice: The ideal lattice-based scheme proposed by Gentry and Halevi in 2011 can achieve a security strength of 72 bits, and the corresponding public key size is about 2.3 GB, while the processing time of refreshing the ciphertext takes several hours. ten minutes.
Schemes based on approximating the GCD problem on integers: The scheme proposed by Dijk et al. in 2010 (and subsequent schemes) adopts a more simplified conceptual model and can reduce the public key size to the order of tens of MB.
Schemes based on the Learning With Errors (LWE) problem: Brakerski and Vaikuntanathan et al. proposed related schemes around 2011; Lopez-Alt A et al. designed a multi-key fully homomorphic encryption scheme in 2012, which is close to real-time multi-party encryption. The need for secure computing.
At present, the known homomorphic encryption technology often requires higher computing time or storage cost, and there is still a gap between the performance and strength of traditional encryption algorithms, but the attention in this field has always been high. The author believes that in the near future, There will be solutions that are close to practical.
