I made the very model, but the model was too general¶

Modeling every cyber vegetable, animal, and mineral

Agenda¶

Introduction / About Sandia
Motivation
Overview of Sandia Emulytics
Research challenges
Shameless recruiting plug

About Sandia¶

Nuclear weapons laboratory, began as the Z division of Los Alamos at the end of the Manhattan Project
Ordnance design, testing, and assembly
Gained expertise in red-teaming NW systems
Carried that expertise into new domains as the lab grew, including cyber
Lots of cyber focus areas, including modeling and simulation for critical infrastructure, etc.

¶

The idea¶

We're working to boot millions of IT endpoints
And couple that with millions of non-traditional endpoints
- Mobile devices
- IoT
- SCADA/ICS
And couple that with behavioral models
- Sociology, cognitive science, ...
Focus on national scale problems
- Interdependency studies of electric power, IT, OT, telephony, IoT...
- Cyber and cyber-physical domains are all fair game

Emulytics¶

Emulation + Analytics
Blend hardware in the loop, simulation, and emulation
Also support human in the loop

Why model?¶

As a National Laboratory, we get to study lots of interesting problems

DevOps: Can we pre-flight new hardware, software, services, to ensure operation in high consequence environments? Can we conduct predictive analysis to detect malfunctions, misconfigurations, malicious consequences?
Malware: Can we gain new understanding of malware through pseudo-in situ execution? How will these 1 million samples impact my network specifically?
ICS/SCADA: What are the best countermeasures for my IT-connected ICS systems, despite not having certainty about the threat? Can I detect attacks on ICS systems from the IT-connected perspective? Can I prove resiliency solutions for IT-control over entire power grids?
Nuclear Weapons: Can we ensure the President will always be able to communicate with a weapon regardless of network state and threat?

Research and Development in Emulytics¶

These are great research questions
- And we won't be looking at any of them today
- But they do prompt a number of R&D activities in Emulytics itself

So where do we start¶

A few interesting things happened in 2007
- Cyberattacks in Estonia
- KVM gets merged into Linux 2.6.20
- iPhone is released
- Worst European heat wave in a century (probably unrelated)
Fast forward to 2008
- We boot 4 million KVM VMs on Jaguar at Oak Ridge National Laboratory
- Lessons learned: Switch forwarding tables are still very much vulnerable to MAC flooding

VM-based sandboxes¶

Can we make use of VM-based sandboxes to model enough of a country to study national-scale attacks?

Is my house on fire or is Atlanta on fire?
How much detail needs to be in the model?

miniweb

A few notes on scale¶

One possibility is to use HPC resources
- Titan - ~300k core / ~18k node AMD cluster
- Sequoia - ~1.5M core / 98k node PPC cluster
- And all those little 10k core systems out there
They all run Linux
We have access!

100 nodes¶

10,000 nodes¶

A typical supercomputer

1M nodes¶

We've run out of pixels

10M nodes¶

Diffraction pattern

Why don't we use the public internet?¶

That's the supercomputer
The bad guys do
The internet has hundreds (of millions) of nodes
As a legal entity, Sandia would have some difficulty pursuing this approach :)

Enter minimega¶

Open source, publicly available, active development
Launch and manage VM-based experiments
Setup complex network topologies
Integrate real hardware with virtual experiments
- Or simulators
- Or humans
Fast!
Repeatable (for some definition of repeatable)
One tool in the toolset; the substrate for a number of programs
- One of your grad students is responsible for the another major tool

R&D in Emulytics¶

The research questions from before inform all sorts of R&D in the platform
And they're all important!
- Scale / density
- VM placement / scheduling
- Experiment description and discovery
- Behavioral/threat modeling
- Non-IT comms
- Measuring humans-in-the-loop
- Verification & Validation
- ...
We can walk through a few of these

Experiment description and discovery¶

Can we automate the description (and generation) of models of large, real networks?

Map-to-model isn't a new problem
But it's novel for getting to a bootable model
Operators often don't know what is actually on their network
Result is significant hand-tooling, human in the loop
- Humans don't scale when building national scale models...

Map-to-Model¶

Ingest router configs, PCAP, active scans, exotic datasets, ...
And simply build a graph
Allow operator-defined templating of "actions" on nodes in the graph
Template system allows for multi-pass model generation
Iterative workflow
Infinite models from one intermediate representation

{{ if .Node.D.name }}
    vm launch container {{ .Node.D.name }}
{{ else }}
    {{ $name := printf "discovery-node-%v" .Node.NID }}
    vm launch container {{ $name }}
    {{ setData .Node "name" $name }}
{{ end }}

Example: SCinet¶

High capacity network
- 9x 100Gbps links
Exists for 2 weeks a year
Supports the SC conference
- 10k+ attendees

SC16¶

SC17¶

Animation!

Non-IT comms (representing IoT devices)¶

How do we plumb non-network based connectivity?

Vehicle radars
physics - sound, heat, light, rabid bunnies
cyber-physical interactions of all kinds

Plumbing¶

Enter miniplumber

A networkless, out-of-band, inter-process communication layer
Quick specification of communication pathways (pipelines)
uni- or multi-cast experiment communication
Supports any number of clients (scales along with the rest of minimega)
Similar to unix pipelines (though not limited to linear pipelines)
Borrows concepts from the plan 9 plumber
Newline delimited messages (as opposed to unix byte streams)
Works on host, in minimega, and any miniccc client (ARM, x86-64 / *bsd, linux, windows)

Example: Setting fire to your datacenter¶

Temperature sensors in datacenter
More sensors outside of the room
Thermal alerts at 80F, critical at 100F (shutdown)
Room temperature simulated as a simple value injected by the user

Plumbing¶

In-room sensors see slightly different values (via)
Out-of-room sensors see different distribution (truncated function of room temperature)
Critical thermal event triggers shutdown of in-room machines
in/out-of-room defined simply by pipe source (no need for geographic annotation)

Plumbing¶

fire.mm

clear namespace wan

vm config filesystem /root/containerfs

# machines in the machine room
vm launch container server[1-10]

# sensors in the machine room
vm launch container sensor_room_[1-4]

# sensors in the hallway
vm launch container sensor_hall_[1-3]

# launch faulty sensors
vm launch container sensor_faulty_[1-2]

vm start all

# setup vias
pipe Troom via bin/normal -stddev 5.0
pipe Thall via bin/normal -stddev 5.0

# connect the hall to the room temperature
plumb Troom "bin/mult .66" Thall

# launch room sensors
cc filter hostname=sensor_room_1
cc exec stdin=Troom stdout=sensor_room_1 /sensor
cc filter hostname=sensor_room_2
cc exec stdin=Troom stdout=sensor_room_2 /sensor
cc filter hostname=sensor_room_3
cc exec stdin=Troom stdout=sensor_room_3 /sensor
cc filter hostname=sensor_room_4
cc exec stdin=Troom stdout=sensor_room_4 /sensor

# launch hall sensors
cc filter hostname=sensor_hall_1
cc exec stdin=Thall stdout=sensor_hall_1 /sensor
cc filter hostname=sensor_hall_2
cc exec stdin=Thall stdout=sensor_hall_2 /sensor
cc filter hostname=sensor_hall_3
cc exec stdin=Thall stdout=sensor_hall_3 /sensor

# connect critical thresholds
plumb Troom "bin/crit 100.0"

Download this example

Plumbing¶

fire2.mm

# inject a temperature value
pipe Troom 70.0

Download this example

Example: Digital logic¶

Not really a cyber-physical example, but it shows the power of the communication framework
Use plumbing as nets in a circuit
VMs as discrete logic components (and, or, not, xor provided)
5-bit unsigned array multiplier
145 gates (VMs)
166 nets (pipes)

Plumbing¶

array.mm

vm kill all
vm flush
clear plumb
clear pipe

vm config filesystem /root/containerfs
vm launch container vm0
cc filter hostname=vm0
cc background /and net1 a0 x0
plumb net1 p0
vm launch container vm2
cc filter hostname=vm2
cc background /and net3 a1 x0
vm launch container vm4
cc filter hostname=vm4
cc background /and net5 a0 x1
vm launch container vm6
cc filter hostname=vm6
cc background /xor net7 net5 zero
vm launch container vm8
cc filter hostname=vm8
cc background /xor net9 net3 net7
vm launch container vm10
cc filter hostname=vm10
cc background /and net11 net3 net5
vm launch container vm12
cc filter hostname=vm12
cc background /or net13 net3 net5
vm launch container vm14
cc filter hostname=vm14
cc background /and net15 zero net13
vm launch container vm16
cc filter hostname=vm16
cc background /or net17 net11 net15
plumb net9 p1
vm launch container vm18
cc filter hostname=vm18
cc background /and net19 a2 x0
vm launch container vm20
cc filter hostname=vm20
cc background /and net21 a1 x1
vm launch container vm22
cc filter hostname=vm22
cc background /xor net23 net21 zero
vm launch container vm24
cc filter hostname=vm24
cc background /xor net25 net19 net23
vm launch container vm26
cc filter hostname=vm26
cc background /and net27 net19 net21
vm launch container vm28
cc filter hostname=vm28
cc background /or net29 net19 net21
vm launch container vm30
cc filter hostname=vm30
cc background /and net31 zero net29
vm launch container vm32
cc filter hostname=vm32
cc background /or net33 net27 net31
vm launch container vm34
cc filter hostname=vm34
cc background /and net35 a0 x2
vm launch container vm36
cc filter hostname=vm36
cc background /xor net37 net35 net17
vm launch container vm38
cc filter hostname=vm38
cc background /xor net39 net25 net37
vm launch container vm40
cc filter hostname=vm40
cc background /and net41 net25 net35
vm launch container vm42
cc filter hostname=vm42
cc background /or net43 net25 net35
vm launch container vm44
cc filter hostname=vm44
cc background /and net45 net17 net43
vm launch container vm46
cc filter hostname=vm46
cc background /or net47 net41 net45
plumb net39 p2
vm launch container vm48
cc filter hostname=vm48
cc background /and net49 a3 x0
vm launch container vm50
cc filter hostname=vm50
cc background /and net51 a2 x1
vm launch container vm52
cc filter hostname=vm52
cc background /xor net53 net51 zero
vm launch container vm54
cc filter hostname=vm54
cc background /xor net55 net49 net53
vm launch container vm56
cc filter hostname=vm56
cc background /and net57 net49 net51
vm launch container vm58
cc filter hostname=vm58
cc background /or net59 net49 net51
vm launch container vm60
cc filter hostname=vm60
cc background /and net61 zero net59
vm launch container vm62
cc filter hostname=vm62
cc background /or net63 net57 net61
vm launch container vm64
cc filter hostname=vm64
cc background /and net65 a1 x2
vm launch container vm66
cc filter hostname=vm66
cc background /xor net67 net65 net33
vm launch container vm68
cc filter hostname=vm68
cc background /xor net69 net55 net67
vm launch container vm70
cc filter hostname=vm70
cc background /and net71 net55 net65
vm launch container vm72
cc filter hostname=vm72
cc background /or net73 net55 net65
vm launch container vm74
cc filter hostname=vm74
cc background /and net75 net33 net73
vm launch container vm76
cc filter hostname=vm76
cc background /or net77 net71 net75
vm launch container vm78
cc filter hostname=vm78
cc background /and net79 a0 x3
vm launch container vm80
cc filter hostname=vm80
cc background /xor net81 net79 net47
vm launch container vm82
cc filter hostname=vm82
cc background /xor net83 net69 net81
vm launch container vm84
cc filter hostname=vm84
cc background /and net85 net69 net79
vm launch container vm86
cc filter hostname=vm86
cc background /or net87 net69 net79
vm launch container vm88
cc filter hostname=vm88
cc background /and net89 net47 net87
vm launch container vm90
cc filter hostname=vm90
cc background /or net91 net85 net89
plumb net83 p3
vm launch container vm92
cc filter hostname=vm92
cc background /and net93 a4 x0
vm launch container vm94
cc filter hostname=vm94
cc background /and net95 a3 x1
vm launch container vm96
cc filter hostname=vm96
cc background /xor net97 net95 zero
vm launch container vm98
cc filter hostname=vm98
cc background /xor net99 net93 net97
vm launch container vm100
cc filter hostname=vm100
cc background /and net101 net93 net95
vm launch container vm102
cc filter hostname=vm102
cc background /or net103 net93 net95
vm launch container vm104
cc filter hostname=vm104
cc background /and net105 zero net103
vm launch container vm106
cc filter hostname=vm106
cc background /or net107 net101 net105
vm launch container vm108
cc filter hostname=vm108
cc background /and net109 a2 x2
vm launch container vm110
cc filter hostname=vm110
cc background /xor net111 net109 net63
vm launch container vm112
cc filter hostname=vm112
cc background /xor net113 net99 net111
vm launch container vm114
cc filter hostname=vm114
cc background /and net115 net99 net109
vm launch container vm116
cc filter hostname=vm116
cc background /or net117 net99 net109
vm launch container vm118
cc filter hostname=vm118
cc background /and net119 net63 net117
vm launch container vm120
cc filter hostname=vm120
cc background /or net121 net115 net119
vm launch container vm122
cc filter hostname=vm122
cc background /and net123 a1 x3
vm launch container vm124
cc filter hostname=vm124
cc background /xor net125 net123 net77
vm launch container vm126
cc filter hostname=vm126
cc background /xor net127 net113 net125
vm launch container vm128
cc filter hostname=vm128
cc background /and net129 net113 net123
vm launch container vm130
cc filter hostname=vm130
cc background /or net131 net113 net123
vm launch container vm132
cc filter hostname=vm132
cc background /and net133 net77 net131
vm launch container vm134
cc filter hostname=vm134
cc background /or net135 net129 net133
vm launch container vm136
cc filter hostname=vm136
cc background /and net137 a0 x4
vm launch container vm138
cc filter hostname=vm138
cc background /xor net139 net137 net91
vm launch container vm140
cc filter hostname=vm140
cc background /xor net141 net127 net139
vm launch container vm142
cc filter hostname=vm142
cc background /and net143 net127 net137
vm launch container vm144
cc filter hostname=vm144
cc background /or net145 net127 net137
vm launch container vm146
cc filter hostname=vm146
cc background /and net147 net91 net145
vm launch container vm148
cc filter hostname=vm148
cc background /or net149 net143 net147
plumb net141 p4
vm launch container vm150
cc filter hostname=vm150
cc background /and net151 a4 x1
vm launch container vm152
cc filter hostname=vm152
cc background /and net153 a3 x2
vm launch container vm154
cc filter hostname=vm154
cc background /xor net155 net153 net107
vm launch container vm156
cc filter hostname=vm156
cc background /xor net157 net151 net155
vm launch container vm158
cc filter hostname=vm158
cc background /and net159 net151 net153
vm launch container vm160
cc filter hostname=vm160
cc background /or net161 net151 net153
vm launch container vm162
cc filter hostname=vm162
cc background /and net163 net107 net161
vm launch container vm164
cc filter hostname=vm164
cc background /or net165 net159 net163
vm launch container vm166
cc filter hostname=vm166
cc background /and net167 a2 x3
vm launch container vm168
cc filter hostname=vm168
cc background /xor net169 net167 net121
vm launch container vm170
cc filter hostname=vm170
cc background /xor net171 net157 net169
vm launch container vm172
cc filter hostname=vm172
cc background /and net173 net157 net167
vm launch container vm174
cc filter hostname=vm174
cc background /or net175 net157 net167
vm launch container vm176
cc filter hostname=vm176
cc background /and net177 net121 net175
vm launch container vm178
cc filter hostname=vm178
cc background /or net179 net173 net177
vm launch container vm180
cc filter hostname=vm180
cc background /and net181 a1 x4
vm launch container vm182
cc filter hostname=vm182
cc background /xor net183 net181 net135
vm launch container vm184
cc filter hostname=vm184
cc background /xor net185 net171 net183
vm launch container vm186
cc filter hostname=vm186
cc background /and net187 net171 net181
vm launch container vm188
cc filter hostname=vm188
cc background /or net189 net171 net181
vm launch container vm190
cc filter hostname=vm190
cc background /and net191 net135 net189
vm launch container vm192
cc filter hostname=vm192
cc background /or net193 net187 net191
vm launch container vm194
cc filter hostname=vm194
cc background /xor net195 zero net149
vm launch container vm196
cc filter hostname=vm196
cc background /xor net197 net185 net195
vm launch container vm198
cc filter hostname=vm198
cc background /and net199 net185 zero
vm launch container vm200
cc filter hostname=vm200
cc background /or net201 net185 zero
vm launch container vm202
cc filter hostname=vm202
cc background /and net203 net149 net201
vm launch container vm204
cc filter hostname=vm204
cc background /or net205 net199 net203
plumb net197 p5
vm launch container vm206
cc filter hostname=vm206
cc background /and net207 a4 x2
vm launch container vm208
cc filter hostname=vm208
cc background /and net209 a3 x3
vm launch container vm210
cc filter hostname=vm210
cc background /xor net211 net209 net165
vm launch container vm212
cc filter hostname=vm212
cc background /xor net213 net207 net211
vm launch container vm214
cc filter hostname=vm214
cc background /and net215 net207 net209
vm launch container vm216
cc filter hostname=vm216
cc background /or net217 net207 net209
vm launch container vm218
cc filter hostname=vm218
cc background /and net219 net165 net217
vm launch container vm220
cc filter hostname=vm220
cc background /or net221 net215 net219
vm launch container vm222
cc filter hostname=vm222
cc background /and net223 a2 x4
vm launch container vm224
cc filter hostname=vm224
cc background /xor net225 net223 net179
vm launch container vm226
cc filter hostname=vm226
cc background /xor net227 net213 net225
vm launch container vm228
cc filter hostname=vm228
cc background /and net229 net213 net223
vm launch container vm230
cc filter hostname=vm230
cc background /or net231 net213 net223
vm launch container vm232
cc filter hostname=vm232
cc background /and net233 net179 net231
vm launch container vm234
cc filter hostname=vm234
cc background /or net235 net229 net233
vm launch container vm236
cc filter hostname=vm236
cc background /xor net237 net205 net193
vm launch container vm238
cc filter hostname=vm238
cc background /xor net239 net227 net237
vm launch container vm240
cc filter hostname=vm240
cc background /and net241 net227 net205
vm launch container vm242
cc filter hostname=vm242
cc background /or net243 net227 net205
vm launch container vm244
cc filter hostname=vm244
cc background /and net245 net193 net243
vm launch container vm246
cc filter hostname=vm246
cc background /or net247 net241 net245
plumb net239 p6
vm launch container vm248
cc filter hostname=vm248
cc background /and net249 a4 x3
vm launch container vm250
cc filter hostname=vm250
cc background /and net251 a3 x4
vm launch container vm252
cc filter hostname=vm252
cc background /xor net253 net251 net221
vm launch container vm254
cc filter hostname=vm254
cc background /xor net255 net249 net253
vm launch container vm256
cc filter hostname=vm256
cc background /and net257 net249 net251
vm launch container vm258
cc filter hostname=vm258
cc background /or net259 net249 net251
vm launch container vm260
cc filter hostname=vm260
cc background /and net261 net221 net259
vm launch container vm262
cc filter hostname=vm262
cc background /or net263 net257 net261
vm launch container vm264
cc filter hostname=vm264
cc background /xor net265 net247 net235
vm launch container vm266
cc filter hostname=vm266
cc background /xor net267 net255 net265
vm launch container vm268
cc filter hostname=vm268
cc background /and net269 net255 net247
vm launch container vm270
cc filter hostname=vm270
cc background /or net271 net255 net247
vm launch container vm272
cc filter hostname=vm272
cc background /and net273 net235 net271
vm launch container vm274
cc filter hostname=vm274
cc background /or net275 net269 net273
plumb net267 p7
vm launch container vm276
cc filter hostname=vm276
cc background /and net277 a4 x4
vm launch container vm278
cc filter hostname=vm278
cc background /xor net279 net275 net263
vm launch container vm280
cc filter hostname=vm280
cc background /xor net281 net277 net279
vm launch container vm282
cc filter hostname=vm282
cc background /and net283 net277 net275
vm launch container vm284
cc filter hostname=vm284
cc background /or net285 net277 net275
vm launch container vm286
cc filter hostname=vm286
cc background /and net287 net263 net285
vm launch container vm288
cc filter hostname=vm288
cc background /or net289 net283 net287
plumb net281 p8
plumb net289 p9
vm start all

Download this example

Multiplication¶

array2.mm

# 5 x 2
pipe a4 0
pipe a3 0
pipe a2 1
pipe a1 0
pipe a0 1

pipe x4 0
pipe x3 0
pipe x2 0
pipe x1 1
pipe x0 0

shell sleep 1

.columns name,last pipe

Download this example

Verification and Validation¶

We don't know how much to trust the results from our emulation-based models

Virtual machines, virtual networks, emulators, simulators...
Real software, real network packets, real behavior

V&V¶

Short answer: It depends
Case study: DNS amplification attack modeling
- We know that disabling logging in DNS servers can exacerbate the effect of the attack
- Because DNSd is suddenly more performant
Can we illustrate this in a model?

DNS bare metal performance¶

DNS emulated performance¶

Virtual artifacts¶

We're starting to make a 'cookbook' of virtual artifacts
Goal is to provide the experiment designer with measurement boundaries

And why not...¶

In a talk about modeling everything, why not play out the game of life?

Ongoing research¶

Threat modeling¶

Cyber doesn't exist in the absence of an adversary

Or from a modeling perspective...
The quality of our cybersecurity modeling is bound to the quality of the represented threat
So how do we model threat in a holistic way?

Threat modeling¶

Two major pushes
- Threat parsing
- Threat/actor modeling
The latter is a traffic generation problem
- Represent all the high-order things an attacker does
- Dropped files, timing, laziness...
Academic alliance with Purdue, GT, NM Tech
- Dr. Manos Antonakakis

A plug¶

We need help!

Academic partnerships
Internships
- Undergrad through post-doc
Full-time R&D staff
- Masters and Ph.D.