Command and control¶
How to use miniccc - minimega's command and control solution
Overview¶
In addition to orchestrating VMs, minimega includes a command and control (cc or c2) layer for VMs. The c2 layer allows users to:
- Configure VM post-boot
- Execute/Background programs
- Push/pull files
The cc API in minimega provides a mechanism to programmatically execute programs and send and receive files to VMs launched by minimega.
In addition, the cc API allows creating TCP tunnels (as well as reverse tunnels) from the host machine to/from any VM.
miniccc - the cc agent¶
The cc API reles on having an agent run on the VM called miniccc. minicc supports two modes of communication with VMs:
-
Virtio-serial (KVM), or UNIX domain socket (containers)
- simple i/o layer, similar to a UNIX pipe between the host and VM.
- both host and VM must have virtio-serial support
-
TCP
- The VMs must be able to communicate directly with the host they are launched on.
- See module 05 on creating a tap to the host for one possible route for this.
Most experiments communicate via the serial interface to avoid introducing orchestration traffic to the experiment network.
Starting miniccc¶
The miniccc client uses several command line switches to control how to connect to minimega, as well as where to store files received for the client.
To use virtio-serial, use the -serial flag, with the path to the virtio-serial file.
NOTE: In linux, the default virtio-serial cc port is /dev/virtio-ports/cc. In Windows, the path is \.\Global\cc.
To connect over TCP, provide the -parent flag with the host/ip of host running minimega.
By default, miniccc will create the directory /tmp/miniccc to store state and files in. Files sent to the client will be stored in /tmp/miniccc/files. You can change this directory with the -base flag.
cc clients API¶
Executed by itself, cc give you a count of connected clients:
cc clients can be used to see more information about those clients:
Clients report their UUID, hostname, OS, architecture, IP and MAC addresses to minimega. This information is updated periodically, so if an IP changes, minimega will see the change.
Client information is stored by UUID in minimega. When a client responds to a command, the response is logged by minimega in a subdirectory named after the UUID for that client. We'll discuss responses later.
cc exec API¶
The simplest way to run commands is with the cc exec API.
Note: You may need to wrap your command in quotes or escape special characters.
# This will run 'hostname' on all connected clients:
cc exec hostname
# Get a directory listing of the client's root:
cc exec ls /
Asynchronous communication¶
minimega's c2 layer is asynchronous so you will have to wait a few seconds for the following actions to occur:
- minimega pushes commands to miniccc
- miniccc runs the command and stores the response
- miniccc reports the results during the next heartbeat
Pitfall: cc commands¶
There are two things to note at this point:
-
First, commands don't go away until you delete them with
cc deletecommand.- This means that if you were to reset a VM or start new VMs, they would all see and execute these commands.
-
Second, the response from the client isn't printed to screen.
- Instead, responses are logged in a special directory structure in minimega's base path.
- You can browse to the responses yourself, or use the
cc responsescommand to view responses from clients.
More on both of these API commands later.
cc commands API¶
You can inspect current in-flight commands with cc command, which shows the contents of the command, any applied filters (more on that later), and how many clients have responded.
For now, we will focus on the command and responses columns.
Once we have responses, we can check them with cc responses:
# list all commands
cc commands
# print the response of the previous command
cc responses 1
# check all responses
cc responses all
When the client responds to a cc exec command, standard out and error are stored in the files stdout and stderr respectively.
Pitfall: cc exec API¶
Many users will try to run long-running commands via cc exec. Unfortunately,
miniccc will wait until the process exits before proceeding -- causing it to
hang.
Instead, long-running commands should be started with cc background.
miniccc reports running processes back to minimega which can be viewed with cc process list.
minimega can even kill these processes.
# intead of blocking on this command, run in the background:
cc background protonuke -serve -dns 100.0.2.2
# show all running processes
cc process list all
# you can kill a process by PID
cc process kill <pid>
# or all at once, by name
cc process killall protonuke
cc filter API¶
So far, all the commands that we have looked at would have run on every VM
connected to minimega. Most of time, we only want to run commands on a subset
of VMs. This is accomplished with the cc filter API:
cc filter ip=8.8.8.8
cc background protonuke -serve -dns 100.0.2.2
# you can filter on more than one property:
cc filter name=foo os=windows
There are many ways to filter VMs:
- IP address/CIDR
- Operating system
- Tags
Filters remain in affect until replaced or cleared.
File I/O - sending¶
In addition to running commands, the c2 layer can send and receive files.
In order to send files to a client, the files must be rooted in the files subdirectory in the minimega base directory. By default minimega uses /tmp/minimega/files.
# upload the protonuke file to all VMs matching the current cc filter.
# protonuke is located in /tmp/minimega/files/
cc send protonuke
# Send all files (globs) with the * operator
cc send /somedirectory/*
# use commands to check recent file operations, just as for commands
cc commands
Files will be transfered in the next heartbeat, and will appear in the files subdirectory in the client's base directory. By default, this is /tmp/miniccc/files.
File I/O - receiving¶
Receiving files is just like sending files, except that you can specify any path on the client to receive files from.
Globs (wildcards) work with receiving files too, so you can receive entire directories of files.
Clients will fetch the files from minimega before moving on to any other commands.
# create a file bar.bash in /tmp/minimega/files/ containing the following:
# #!/bin/bash
# mkdir /foo
# echo "hello cc!" >> /foo/bar.out
#
# then run the following:
cc send bar.bash
cc exec bash /tmp/miniccc/files/bar.bash
cc recv /foo/bar.out
Mounting filesystems¶
Another way to interact with VMs is by mounting their filesystems on the host
using the cc mount API. Users can then read, write, create, and delete files in /mnt/foo.
cc mount is fully integrated with namespaces -- you may mount a VM's filesystem to the head node, regardless of which host is actually running the VM.
To list existing mounts, run it with no arguments. You can unmount a specific VM or mount by path. Without an argument, clear cc mount clears all mounts. (This also occurs when you call clear cc.)
# Mount the filesystem for foo
cc mount foo /mnt/foo
# List existing mounts
cc mount
# unmount a specific VM
clear cc mount [uuid or name or path]
# unmount all
clear cc mount
Tunneling¶
Create forward and reverse TCP tunnels over the cc connection, including over virtio-serial connections. To create a forward tunnel, that is, a listening port on the minimega host that is tunneled to a destination host and port from the perspective of the client, use cc tunnel. When creating a forward tunnel the UUID of the client must be specified. The destination host can be localhost or any other host reachable from the client.
Similarly, a reverse tunnel, a listening port on the client tunneled to a host and port reachable from the minimega host, can be created by using cc rtunnel. Reverse tunnels do not require a UUID to be specified, and instead use the current client filter to restrict which clients create the tunnel. That means you can tunnel a port for every client to a resource outside of the experiment.
For example, to tunnel local port 4444 on each client to a web server reachable from the minimega host:
Other Commands¶
cc delete to delete commands and responses
The miniccc client can add tags to the info for the VM the client is running on. This enables third party tools to upstream information about a VM to minimega via miniccc. Tags are key/value pairs, and are added simply by using the -tag switch on a running miniccc instance.
Pitfall: version mismatches¶
miniccc and minimega must come from the same minimega release. The c2 protocol between them can change between versions, so mixing versions can cause clients to fail to connect, silently drop commands, or misreport responses.
When a client's version doesn't match, minimega logs a warning at handshake time:
Check minimega's log (-level info or higher) if clients seem to connect but never respond as expected.
If you see this warning, rebuild your VM image's miniccc binary any time you upgrade minimega.