这个文章来源于我2018年做的一次技术分享活动。 讲的是2018年很火的两个CPU底层安全漏洞分析，里面有一些有趣的知识点很值得复习。 原来的分享是英文的，这里简单把脚本翻译一下，炒个冷饭。

Meltdown与Spectre漏洞学习笔记

最近（2018年）有两个安全漏洞Meltdown和Spectre，引起了人们的注意。据说黑客可以利用这两个漏洞来攻击几乎所有的计算机。

这两个漏洞是当前很火的话题，人们讨论他们并且也为之感到恐慌。英特尔公司的股价在2018年1月3日下跌高达3.5%。

我做了一些研究和调查，尝试理解他们是否真的那样糟糕，是不是世界末日真的来临了。理论上，这两个漏洞足以影响过去20年内生产的所有计算机设备。

今天这个分享是关于Meltdown和Spectre。我个人并不是微处理器的专家，但是我会从工程师的角度尝试去分享我所理解的技术细节和学习到的知识。

Recently there are two announced security vulnerabilities, which are said to have an ability to expose personal data to hackers and affect nearly every computers. 

These are really hot issues. People talk about them and panic about them. The stock price of Intel fell 3.5% on Jan. 3rd.

So I did some searching and investigations about them, and tried to figure if it is really that bad, if it is really the end of the world. It sounds not good that these two vulnerabilities affect nearly all the computer devices produced in the last 20 years.

Here today let’s talk about Meltdown & Spectre. I’m not an expert of the micro architecture, but I’ll try to share what I’ve known from an engineer’s perspective.

话题列表 Agenda

首先，我们需要理解漏洞所利用到的两个基础技术点，CPU的乱序执行和预测执行。

这两个概念（功能）对于现代高性能微处理器十分重要，但在今天却被Meltdown和Spectre漏洞所利用来实施攻击行为。

接下来会提到 缓存边信道攻击。我们会举一个简单的例子来解释什么是边信道攻击以及缓存计时攻击。

之后就是，漏洞的核心指令和一个利于理解的例子来理解漏洞的核心思想。

最后会有一个简单的POC。

First, I’d like to talk about 2 building blocks of these two vulnerabilities. The out-of-order execution and speculative execution, are two important concepts and features of a modern high-performance microprocessor, which however, are leveraged by meltdown and spectre to do the bad things.

Second one is the cache side channel attack. We will have a brief explanation of side channel attack and a detailed explanation of cache timing attack.

Then, it would be the core instruction and some toy examples for us to understand the core idea of these two vulnerabilities.

I’ve also got a simple POC for you, which leveraged some core techniques of these two vulnerabilities and is simple enough for us to understand.

CPU指令

从计算机指令开始。

指令是处理器接受的基础执行单元，比如 ADD AX，BX，意思是让处理器把AX寄存器的值放进BX寄存器，之后把结果保存进AX寄存器。

高级编程语言（C/C++，Java）允许我们用更抽象更容易理解的方式编写指令。但最终每个程序都需要被转化成处理器指令才能被计算机执行。

接下来我们就进入微处理架构来理解什么是乱序执行和预测执行。

Let’s start with instructions. 

Instructions are the basic actions the processor can take. Like the example, ADD AX, BX, means we are asking the processor to add the value in the AX register to BX and then stored the result into AX register.

High level programming languages (C/C++, Java) allow many processor instructions to be written simply. But every program must be converted to the processor instructions of the computer it will be run on.

Let’s go deep into the microarchitecture to understand what out-of-order execution is all about. 

大致上，每个指令都包含一系列操作。

简单的说，有四个主要的操作。Fetch，Decode，Execute和Write。

就像是，在现实世界中，如果你去一家餐厅，你会先点一些菜，等上菜，享用美食，最后付钱离开。然而，这个餐厅不会只服务你一位顾客。一旦你点完了菜，服务员就会离开，然后去服务其他客人。

对于微处理器来说，也是类似的。

In general, processing one instruction involves a number of operations. 

To make it easy to explain, there are four main basic operations. Fetch, Decode, Execute and Write.

In the real world, if you go to a restaurant, you have to order some dishes, wait for them， have your food and pay the bill. The whole restaurant will not just serve you only one customer. Once you order some dishes, the waiter would go away and serve other people. 
For the microprocessor, it’s similar. 

这里有一个来自《深入理解计算机系统》的例子。

在现实中，由于平台的不同，各大CPU厂商之间的规范不同，操作可能不止四个，但其核心思想都是相似的。

图里的这条指令在寄存器A和B之间做了一些操作。

第一个操作是Fetch，我们需要找到下一条需要执行的指令在哪儿。

第二个操作是Decode，我们需要弄清楚这条指令是什么，并进行预处理。比如，从寄存器中提前读取数据出来。

第三个操作是Execute，使用算法逻辑单元来计算结果。

最后一个操作是Write back，将计算完成的结果写回去。

Here is an example from the book Computer Systems – A Programmer’s Perspective, which is about how an instruction is organized into several operations.

In the real world, things could be very different. The operations on different platform, from different CPU vendors, might be different, but the core idea is the same.

This instruction is about some OPs with two registers A and B.
The first operation is fetch, we need to figure out which instruction is the next one to execute.
The second operation is called Decode, during this period, we need to figure out what is this instruction, and prepare for the values, like reading values from registers for further exection as shown above.
The third one is execute. Calling some algorithm logic units to calculate the result.
The last one is write back. We have to store the result we got.

在某些古老的CPU上会使用到顺序执行。即，把指令中的操作按照顺序依次执行。

Fetch…Decode…Execute…Write… Fetch…Decode…Execute…Write…

非常简单直白。

但事实上，在CPU中，不同的操作是被不同的CPU单元执行的。比如Decode单元和Fetch单元能独立工作。那么当Fetch单元空闲的时候，为什么Fetch单元还需要等待其他指令的完成呢？

这个对于高性能CPU来说，这种工作模式不能算是高效。就像是，你去一家餐厅，你发现你居然不能点菜，只是因为前一个顾客还没有吃完付钱？！

Here is a basic idea of sequential processing.

We execute the instruction one by one. 

Fetch..Decode..Execute..Write.. Fetch..Decode..Execute..Write..

Pretty simple and straightforward.

But actually, in a CPU, the operations are executed by different Units. The Decode unit and the Fetch unit can work separately. Just like in the previous picture. So why does the Fetch Unit have to wait for the whole instruction being executed? 

It is an absurd way for a high performance CPU to do this job. Like people could be really angry if you go a restaurant and you are not allowed to order any food because the previous customer has not finished his lunch.

So…

乱序执行 Out-of-Order Execution

所以这个就是乱序执行的基础思想。

在这张图里，我们一共有9个时钟周期和6个被执行的指令。而这一切几乎同时发生，这个对于微处理器来说，进步意义巨大。这就是流水线的威力。

So that is basically the idea of out-of-order execution.

As you can see here in the picture, we have 9 clock cycle and 6 instructions executed in mostly the same time, which is huge step forward in the microarchitecture. And that’s the power of pipeline.

对应到高级编程语言，我们用这段伪代码来举个例子。

第一行是，将42赋值给变量x。

第二行是，发起网络请求。一步可能会花费很多时间。

第三行是，做一些关于x的计算。

在CPU执行第二行的时候，可能有大部分时间都在等待网络请求的返回结果。由于乱序执行，CPU有可能就会想，“嘿，我们有一些单元是空闲的，让我们来康康（看看）下一步要执行个啥”。所以当第二行结束的时候，有可能第三行的部分操作已经被执行了。

Let’s pull out of the dive. Take a look at this piece of high level pseudocode written in C.

The first line is about assign 42 to variable x.
The second line is reading the network, which could be really really slow.
The third line is about calculate something related to x.

While CPU is executing the 2nd line, when it is waiting from the network. It may think, “hey some units are spare, let me take a look at what I should do next”. So when the 2rd line is finished, some parts of the 3rd line could have been executed.

Just like what the picture is showing to us.

预测执行 Speculative Execution

接下来，我们再来看一下另一块Meltdown和Spectre利用到的基础技术 – 预测执行。

预测执行是一个优化技术，可以使得CPU提前完成某些可能不被需要的任务。

假设我们有一块超强的CPU，其可以在很短的时间执行很多指令。但最终，这块CPU也会需要从内存中读取或者写入一些数据。对于CPU来说，读写IO是很慢的操作。

所以人们就在思考另一种压榨CPU性能的方法，“为什么不让CPU做预测执行呢？”

在这段代码中，我们有一个IF分支语句会依赖一块很慢的内存读取。

这个IF分支语句一共有两个分支，当内存地址233的值等于333时，执行括号内的Calculate函数。如果不等于，就跳过不执行。

但是，如果我们让CPU具备预测的能力呢？CPU也许会推测，从内存地址233读到的值大概率就是333，反正闲着也是闲着，那我就先预执行一点括号内的内容吧。于是CPU便决定在等待内存读取的同时，执行Calculate函数。

这就是预测执行。

到这里你可能会想，“如果内存地址233的值不等于333，但是由于预测执行，Calculate函数却被执行了，那不就和产生不一致了吗？”

其实乱序之行也有可能会有这种问题。比如下一条指令的部分被执行了，但是代码逻辑却被其他中断强行终止了，比如异常。这个其实也会引起不一致的问题。

OK. Let’s take a look at another building block of Meltdown & Spectre.
Speculative execution is an optimization technique where a computer system performs some task that may not be needed.

Let’s say we have a super fast CPU, which can execute a bunch of instructions in a very short time.
But ultimately, it will need to read something from the memory, which, comparing to instruction executing, is pretty slow.

So people figured that “Why don’t we make our CPU do speculative execution?”, which is another way to boot the performance of our CPU.

In this code piece, we have a IF sentence that depends on a slow memory reading. There would be two options here, the value in the address 233 if memory is equal to 333 which results in the Calculate method being executed in that case, or it is not. Nothing happened and CPU would start doing something Else.

So what if you just guess? You think that the most likely thing to happen here is reading a 333 from the memory and executing the calculate method. So you decide to execute the Calculate method while waiting for the response from memory.
And That is speculative execution.

Here, you may have a question, like “What if 233 of the memory is not actually equal to 333 and the calculate method has already been done, isn’t it wrong because CPU are executing something that should never happened.” For out-of-order execution, if some part of the next instruction are executed and that code piece is terminated by some other interrupt, like exceptions, thing are totally wrong.

原因是这样的。

虽然某些指令本不应该被执行，但在背后被执行了，而最终却没有引发问题，是因为其计算结果并没有被写回去。所有预测执行和乱序执行的计算结果都是放进了缓存中。

如果CPU发现某条指令被提前执行了，那么CPU就直接从缓存中把计算结果拿出来。

如果CPU发现某条指令不应该被执行，却被执行了，那么CPU就直接重置寄存器的数值，缓存中的数据就是无效数据。

The reason is that the result is not written back, although some instructions are executed on the background which should not be. The results of the speculative execution and out-of-order execution are put into cache.

If CPU found the instruction had been executed before, it would grab the results from the cache instead of executing it again.
Or if CPU found the instruction should not be executed, it would just reset the state of registers and the calculated data in cache is useless.

边信道攻击 Side Channel Attack

0

下一个被Meltdown和Spectre利用的基础技术就是边信道攻击。

边信道攻击，和传统攻击不同，是一种基于软件或硬件运行时产生的各种物理泄漏信息来实施攻击的方式。

比如，时间：某个操作运行了多久。

The next building block of meltdown and spectre is side-channel attack.

A side-channel attack is any attack based on information gained from the physical implementation of a computer system, rather than weaknesses in the implemented algorithm itself.
Like, the time: how long your task is executed. 

缓存计时攻击 Cache Timing Attack

1

缓存计时攻击，则是基于上面提到的缓存机制的边信道攻击。

由于CPU读取缓存比读取内存的速度快得多，由此攻击者可以根据两者的时间差异来判断CPU是从缓存还是内存中读取的数据。

这个便是Meltdown和Spectre用于泄漏数据的核心思想。

A cache timing attack is based on the cache mechanism we mentioned just now. Attackers are leveraging the fact that CPU retrieve data from cache is much faster than retrieving from the main memory.

This is main idea of leaking data out of memory in meltdown and spectre attacks. So let’s go deeper inside of this attack.

2

这里又一个简单的例子。

假设这里有一小段内存，一部分是内核态内存（kernel memory），另一部分是用户态内存（user memory）。

内核态内存里的数据是不允许未授权的用户进程访问。

在这张图里，内存地址1-7是用户管理的内存区域，而从地址8-10是内核管理的内存区域。

Here I prepared a simplified example.

Let’s say we have a piece of memory here. In process of the computer system, memory is split into kernel memory and user memory.

Kernel memory is not allowed to read from an unprivileged user process.

In the picture, memory 1 to 7 is managed by user. Memory 8 to 10 is managed by kernel.

3

我们用右边的白盒子代表CPU和CPU中的缓存空间，在左边是用户态内存和内核态内存。我们的攻击目标就是，通过未授权的用户进程访问到位于地址9的内核态数据。

攻击的第一步，是使用类似CLFLUSH指令去强制清理缓存。

第二步，使用特定的数据写入用户态内存。这里我们使用ABCDEFG分别写入地址1到7的空间。

第三步，利用乱序执行和预测执行的特性，我们将 memory[memory[9]] 的内容写入到缓存中。

最后，我们在把用户态内存所有数据依次读取一遍，并且计算出每次读取操作花费的时间。

对于地址1到7的读取操作，大部分操作消耗的时间都非常接近，然而却有一个例外，即读取位于地址5的内存空间。因为memory[9]=5，而memory[memory[9]] => memory[5]这个值被我们提前装载进了内存。

虽然我们不知道memory[9]的值，但是当我们发现CPU在读取memory[5]的速度快过其他操作时，我们就基本能判断memory[9]=5。

以此，我们达到了从未经授权的用户态访问内核数据的目标。利用这个攻击原理，我们便可以尝试从内核数据中偷取更多数据。每偷取一个字节的数据，我们只需要循环256次即可。

As you can see here, we have a white box representing the CPU cache and on the left side we have a piece of memory of user space and kernel space. Our goal is reading the data in the memory at address 9.

The first thing we have to do, is flush the cache, like using CLFLUSH instruction or some other method.

The second is prime the memory. Like here, we assign values ABCDEFG into these memory.

Then, we try to let CPU load memory[memory[9]] into cache.

The last thing we need to do is retrieving memory from 1 to 7 and record the time consumptions.

Time consumption of most of the memory retrieving are pretty close. But there should be one that returns very fast because it is loaded into cache. The CPU does not need to read it from the memory.
So by taking this approach, we found our way to retrieve kernel memory.

If you want to dump a byte from the memory, then you would need a probe array of 256 bytes. This is just a simplified example.

4

这个也是Prime+Probe攻击的基本思路。Meltdown使用了一种Prime+Probe的变形，即Flush+Reload。

到这里你有可能会想，“诶，这个第三步有问题。我们是不可能直接读取到memory[9]的内核数据，内核会直接硬中断掉你的操作”。

没错，这个想法曾经被认为是正确的。当你尝试从用户态直接访问内核态数据时，内核就会跳出来，直接杀掉你的进程。这个理所应当很安全，觉得利用这个不可能有攻击发生。直到Meltdown和Spectre出现…

That is the basic idea of Prime+Probe attack. Meltdown uses a variation of Prime+Probe, which is called Flush+Reload.

But here, you may think, “The step 3 is impossible. It tried to read data in kernel memory, which is not allowed by kernel and should be stopped by some hard interuption.”
Yeah, that’s true. When you are trying to retrieve data in kernel memory, kernel would jump out and say “Hey! This is not allowed. I will kill this process.” Many people are thinking like this, think some attacks like this can not come true. It’s no big deal. Until Meltdown and spectre come out…

5

这里有个Meltdown的小例子，来自谷歌Project Zero组的文章。

这里的probe_array就是我之前提到的，被攻击者控制的用户态内存空间。

在第一行，这段代码直接抛出异常。但是这时，由于乱序执行的特性，第二行的代码其实已经被部分执行了，即从probe_array中读取数据，并放进到缓存中。

这段代码没有展示的就是，之前提到的第三步，实施攻击。

Here is a toy example of Meltdown from the paper published by Project Zero team from Google. 
The probe_array is something like user space data I just showed in previous slices, and the data variable is controlled by attackers.
The line 1, it raises an exception. But due to out-of-order execution, the instruction in the line 2 is partially executed by the CPU, especially the step of loading value of probe_array into the cache.
And this is the missing step 3 in the previous attack. The step is supposed to be impossible.

Meltdown攻击

6

这里是一段Meltdown攻击的核心指令。

内核数据被转载进寄存器，并抛出异常。但是后续指令在异常抛出前，已经被部分执行，导致泄漏了内存空间数据。

This is the core instruction sequence of Meltdown.
An inaccessible kernel address is moved to a register, raising an exception. The subsequent instructions are already executed out of order before the exception is raised, leaking the content of the kernel address through the indirect memory access.

I’m not going to discuss about every line in this picture.

Spectre攻击

7

接下来是Spectre攻击。

和乱序执行不同，它利用了预测执行来实施攻击。

array2和之前的probe数组类似，是被攻击者控制并恶意填充的用户态空间。

攻击者会提供整数x，同时将array1_size和array2清理出缓存，并使得CPU的预测功能组件认为这个IF语句很可能为真。

攻击者会给x赋一个巨大的值，由于array1_size不在内存中，并且CPU会预测这个IF语句为真，那么在CPU从内存读取array1_size的值时，它就会去提前执行IF中的分支语句，即将array2[array1[x]*256]装载进缓存。

接下来就是实施边信道攻击。

我们只需要尝试依次读取probe数组的值，并且记录每一次的时间开销，就能把array1[x]的值从内核态读取出来。

我们并没有直接读取内核态数据，是CPU的提前执行代替我们完成了攻击。

And for the spectre attack.

Instead of out-of-order execution, it leverages the speculative execution.
The array2 is a probe array that is primed by the attacker.
This piece of code receives an integer x from attacker. And the attacker makes array1 size and array2 not present in the processor’s cache and makes branch predictor assume that the if is likely to be true.

Here is the attack.

The attacker assigns an x that is way much bigger than array1 underline size.
As the array1 underline size is not in the memory, the CPU would predict that the if is likely to be and execute the code block in the if sentence while fetching the value of array1 underline size from memory.
So as a result of speculative execution, the value of array2 of x of array1 multiply 256 is loaded into memory.

And this is the missing step 3 in the previous attack. The step is also supposed to be impossible.

What we have to do then, is just retrieving the probe array and record the time consumption of each value retrieving. So that we can get the value of x of array1 out from some other memory.
It is the CPU that is performing the value fetching.

总结

8

简单的总结，这两个攻击利用的核心技术都是对于CPU性能非常敏感的技术缺陷。

乱序执行和预测执行是两个非常重要加速CPU的底层技术。这就是为什么通过打补丁修复这两个漏洞的方式会非常影响CPU的性能。

如果我们需要使用和其他人共享主机的云服务，我们最好确认我们使用的云服务提供商的物理机器已经被打上了补丁。否则，我们的数据就会有泄漏的风险。

这两个攻击同时也使用到了很经典的边信道攻击手段。今后，可能类似的攻击会变得更多。

The techniques these two vulnerabilities leveraged are both sensitive to the performance of CPU. The out-of-order execution and the speculative execution, are main basic techniques to boost the performance of CPU. And that’s why patches for these vulnerabilities have a big impact on the performance of CPU.

If you are using a shared service with someone else, you have to make sure the machines provided by your cloud service provider are patched, or someone would leak your sensitive data out from your kernel memory.

Those are classic usage of side channel attack. And we should keep an eye on such attacks like this.