William Liu

AWS Security Training

AWS Overview

AWS is broken down into (largest to smallest pieces):

Most services are regional. You mainly deal with AZs in regard to VPC related service decisions.

Different Regions

us-east-1

us-west-1

AWS Services

API calls are getting increasingly complex.

Service # of functions EC2 352 IOT 167 …

Global Services

Most services are regional, but some are global, like:

Commonly confused terms

An aws account contains IAM users

AWS “Shared” Responsibility Model

Summary: Anything you can secure, is your responsibility to secure.

Customer - responsibility for security ‘in’ the cloud Customer Data Platform, Applications, Identity and Access Management Operating System, Network etc.

AWS’s problem:

Your problem:

Just because something is hosted on AWS, does not mean it is AWS’ problem

Basically, the AWS security responsibility is not perfect security.

Monoculture

AWS provides the same security all accounts (unless you pay to have your own isolated partitions like the US gov, ~$600M). No one gets to visit their data centers. Everyone on AWS is running in the same ‘version’ of AWS (same APIs, same software)

AWS does some additional security benefits

AWS Rules

Resources for keeping up with AWS news

Last Week in AWS - https://lastweekinaws.com CloudSecList - https://cloudseclist.com og-aws: https://og-aws-slack.lexikon.io @awswhatsnew (posts announcements) @AWSSecurityInfo @jeffbarr (AWS Evangelist) ‘Securing DevOps’ book by Julien Vehent

How AWS contacts you

AWS communicates only to the root account email address

Account Recovery

MFA (Multi-Factor Authentication) Options

Supported MFA solution  Pros                                    Cons
Hardware Token          Forces physical access                  Phishable, timing gets out of sync, lost
QR code                 Free                                    Phishable, can be copied to anywhere, 2FA -> 1FA
U2F                     Forces physical access, not phisable    May be lost

Remember that the QR code is just a representation of data (it is just text)

Backups

Do not put your backups in the same place as the data being backed up

Case Study:

Disaster Recovery

S3 durability is 99.999999999%

S3 allows cross-region replication and life cycle policies

Not all resources can be backed up

AWS Backup

AWS Backup is a service to automate common backup tasks (e.g. RDS, DynamoDB)

Consider how long it takes to restore a backup.

Thundering Herd

If everyone evacuates one region, other regions will experience trouble (e.g. everyone will start moving off the failure region and try another region, which will cause say EC2 times a while to start)

How can I tell if something is down?

AWS status page shows service status, but usually everything is always green (e.g. if S3 down, this page will not update) Instead, use the Personal Health Dashboard A lot of monitoring companies will be quiet when AWS is down.

SLAs

S3 SLA is for access not durability

SLA Math

You need to multiple when at least one of your apps is down

ELB = 99.99% * EC2 = 99.99% * EBS 99.99% * RDS 99.95% = 99.92% Total Time

AWS service baseline

Remember that AWS works in the max 2 pizza size team rule. There really isn’t a PM that works across teams. A feature might be released and not have ‘simple’ cross functional services (e.g. no access logs)

Basically, you want to wait a while before using a new service that AWS releases.

VPC Network Communication

Encryption

Many services have an ability to encrypt data at rest, but others you have to opt in. You cannot really tell after you hit ‘encrypt’ option because AWS does not tell you much about what it is doing for encryption. There is no way to prove the data really is encrypted, how keys are managed, etc.

Key takeaways

Public S3 buckets

You can see in S3 ‘Access’, there is ‘Public’. You can have bucket permissions through:

Services that work with Resource Based policies (e.g. EC2s)

Resources

Why are so many S3 buckets been made public?

S3 Public Block

Denies S3 buckets account-wide or select buckets from being made public Beware that AWS’s definition of Public is not yours If the bucket allows access from a single IP, it is not public. Also not public if it is a /1 If the bucket allows access from another account, it is not public.

Key takeaways

Logs

            What        Delay           Storage CloudTrail      API calls   15 minutes      S3 VPC Flow Logs   etc

VPC Flow Logs

VPC Flow Logs are awkward and much less helpful than you assume

VPC Traffic Mirroring

Full packet capture If traffic is encrypted, will not magically decrypt traffic Specifies a listener and sends packets to UDP port 4879

CloudTrail

What is it?

Cloud Trail tells us:

Log Options include:

There are things CloudTrail records where there is no API calls (e.g. console logins) CloudTrail does not record things like sshing into a server There is not a strict 1:1 ratio of IAM privileges to API calls (e.g. dax.CreateCluster requires 11 IAM privileges) IAM vs API vs CloudTrail frequently has names that do not match with the service names CloudTrail records e.g. CLI command: aws s3 ls = IAM Privilege: ListAllMyBuckets = API Action: ListBuckets

IAM

Use the aws site for generating IAM policies

https://awspolicygen.s3.amazonaws.com/policygen.html

More data is available by parsing the docs

API Data

Look at botocore

Best Practices for CloudTrail

Your Prod AWS account is separate from your CloudTrail AWS account (that way if something is compromised on your prod account, a CloudTrail AWS account can still see access)

GuardDuty

Best monitoring system for detecting compromised systems in AWS. Data sources include:

Uses ‘machine learning’, but probably “This hasn’t been seen for 30 days” Uses ‘Threat Intel’, which is mainly “Here’s a list of bad IP Addresses that we see being used”

GuardDuty is regional (need to turn on per region) Usually less than 1% of your total AWS bill GuardDuty has a master/member concept for multiple accounts, but not necessary and cannot aggregate between regions.

Master/Member Accounts - look up, but not recommended to use; see CloudWatch Events Aggregation

Basically, you should use GuardDuty. It detects a lot of ways that a system might be compromised and this extends to say ‘Access Denied’ from services, changes not done through Terraform, etc. Can also alert on high impact areas (e.g. DeleteVPC, DeleteDBCluster) or known rare calls (e.g.iam:Create, accessing backups, modifydbinstance password)

CloudWatch Events aggregation

Turn on cloud watch event rules, that then goes to your ‘security’ aws account which sends to kinesis stream and create a rule to send the events to. Have a lambda send the events in the kinesis stream to another account (say us-east-1)

Then on us-east-1, have the events sent to StreamAlert which analyzes and does some rules (e.g. alert)

Alerts

You can set up your own detections to see if there are any bad security practices:

Security Alerts

Alert Levels

Ticket your alerts, will help you learn:

Actionable Alerts

Good developers document their code. Good defenders document their alert rules.

Querying Logs

You can:

JQ

Few ways to use JQ:

$cat *.json | jq '.Records[]|eventName'

# show in tab
jq -r '.Records[]'

# Can import jq with Python
import pyjq

How to respond to Alerts

Netflix has Diffy which can help check if any resources are compromised during an incident https://github.com/Netflix-Skunkworks/diffy

Key takeaways

IAM

Access keys

Session Keys

Acquired when you assume a role or use an IAM role by a service

Secret Key

Only the secret key needs to be kept secret. The access key and session token will show up in CloudTrail logs, but not the secret key.

boto

Python SDK used by the AWS CLI and many tools for interacting with AWS is with boto

aws-vault

https://github.com/99designs/aws-vault

Tool used to securely store and access AWS credentials in a development environment. Uses the macOS keychain or an encrypted file, keeps secrets out of ~ Has some issues with needing to repeatedly type in your password

Configuring IAM roles

In ~/.aws/config, you can configure role assumptions

[default]
output=json
region=us-east-1

[profile prod]
role_arn=arn...

[profile stag]
role_arn=arn...

SSO

You should use SSO, but not AWS SSO. Use Segment’s version of AWS OKTA

https://github.com/segmentio/aws-okta

Magical IP Address

The IP Address 169.254.169.254 is a magical IP in the cloud world (for AWS, Azure, DigitalOcean, Google) that allows cloud resources to find out metadata about themselves.

IP Addresses:

IAM differences

Users vs Roles

Why use IAM instead of the root creds for everything?

IAM Policies

Different actions support different conditions

Consider using an IAM linting library to check if policies are valid or if there are bad policy patterns

Attribute Based Access Control (ABAC)

Restricts people to only certain application pipelines using tags

IAM Limits

Ideally you would specify exactly

But managed policies are limited to 6144 characters

AWS Managed Policies

Best Practice

Allow user to add their own MFA device and restrict access for other actions unless MFA is active

IAM Permissions Boundaries

An IAM permissionsboundary is used to set the boundary for an IAM entity (user or role). This limits the maximum permissions for the user or role.

Resource Policies

Resource Policies control how the specified principal can access the resource to which the policy is attached

Organization SCPs

Service Control Policies (SCPs) are the only way that even the root user can be stopped from doing something Master account cannot be blocked via SCP on itself.

Session Policies

When you assume a role, you can assume a session policy to reduce the scope of the privileges you would have. E.g. I want to delete a bucket, but be careful that I do not delete these other critical buckets, I might assume a role.

Best Practices

VPC endpoints

Honey Tokens

SpaceCrab - Atlassian generates an access key for honey tokens CanaryTokens - Create fake AWS keys, email addresses, etc Rhino Security Labs - honeytoken usage detected via Cloudtrail

Key takeaways

Open Source AWS Security Tools

https://github.com/toniblyx/my-arsenal-of-aws-security-tools

S3 bucket defense ideas

Auditing Tools

Tool Author ScoutSuite nccgroup - focused on pentests Security Monkey Netflix - EOL, first regular scanning tool for security teams AWS Config Rules AWS - easy to setup, but may be difficult to configure to your needs Cloud Custodian Capital One - first aws-first tool, auto-remediation, difficult to configure Prowler CIS (Center for Information Security Benchmark) - easy to use bash tool CloudMapper Duo Labs PacBot T-Mobile (but costs $1000 a month to deploy) StreamAlert Airbnb - Use terraform to set up an application based on Kinesis Streams and Lambda to receive and analyze logs Uses Rules as code so it is pretty awesome

No tool is a clear winner, each tool has a dozen unique checks Be careful with scanning accounts regularly - can cause CloudTrail logs 10x in size, AWS can rate-limit you for too many queries

AWS Config Rules

AWS Trusted Advisor

First day of using any auditing tool

Key takeaways

Pentests

Most activities no longer require informing AWS about pentesting. Just do not Dos or brute-force things.

https://aws.amazon.com/security/penetration-testing/

Assessments

  1. Ask your finance team: Who is paying?
  2. Ask your TAM: What accounts use your domain for their email?
  3. Ask around: Find accounts tied to personal emails using the free tier
  4. Search company emails: Subject “Welcome to Amazon Web Services”
  5. Search network logs: DNS to console.aws.amazon.com
  6. Identify account relationships: Use CloudMapper ‘weboftrust’
  7. Perform recon (find subdomains, etc)

Important APIs

aws iam generate-credential-report
aws iam get-credential-report

Instances of hacks

Code Spaces Instagram Million Dollar Bug

Limiting Access

Security Groups

NACLS

KMS

Key Management Service Ensures your encryption key is never accessible Pricing is usage based Every key rotation increases the monthly cost

CloudHSM

Dedicated hardware Pricing is per hour with unlimited usage

Inspector: Network Reachability

No agent needed, uses automated reasoning, only works on EC2

Global Accelerator

Allows you to create direct connectivity to resources, including in a private subnet

AWS Systems Manager

Attempt at replacing Chef, Ansible, etc. (also has a service called OpsWorks) Have to install an agent onto EC2s

AWS Secrets Manager

Store secrets (API keys, passwords, etc)

AWS WAF (Web Application Firewall)

Integrates with CloudFront (caching) or ALB (modern HTTP focused version of ELB) Pattern matching to block IP Addresses, HTTP headers, HTTP body, or UI strings and also rate-limiting

AWS Firewall Manager

Manages WAF and Security Groups on multiple accounts

AWS Shield

Managed DDoS protection Every AWS account has this on You can pay for Shield Advanced ($3k/mo + usage)

Service Quotas

Tells you when you’re getting close to limits for AWS (e.g. default EC2 limit is 20/region) Allows you to request limit increases via API as opposed to support tickets

Roadmap to Securing your Infra

  1. Inventory What is in the accounts? Who is the point of contact? Who pays for it? Identify a Security Account Move the account into an Organization
  2. Backups Ensure you have backups, test your backups, and that Recovery Time Objectives (RTO) and Recovery Point Objectives (RPO) meet your requirements Have backups in a separate account S3 Object Lock and Glacier Vaults can ensure data cannot be deleted
  3. Visibility and initial remediation Turn on CloudTrail logs for all accounts, send to a central location AWS Organization Trail makes this easy Make sure you allow each account to see their own logs somehow Create an IAM role in every account to give Security view access Create an account initialization process
  4. Detection Turn on GuardDuty in all active regions Detect issues from logs in near real-time Perform regular scanning of your accounts for security issues Document your security guidelines
  5. Secure IAM access Use SSO for access Remove all IAM users Reduce the privileges of roles to necessary services (use Access Advisor) Consider using github.com/Yelp/detect-secrets to scan for env variables
  6. Network attack surface reduction Have no publicly facing EC2s or S3 buckets Put EC2 behind load-balancers S3 buckets can be behind CloudFront Move all non-public network resource sinto private subnets and proxy
  7. Reproducibility and supply chain management Control AMI and package sourcing Option 1: Use Salt/Puppet/Ansible/Chef to maintain configurations on your EC2s Option 2: Build your own AMIs, make the filesystem read-only Do not ssh into instances to make changes Host your own repo of libraries and software (do not npm/pip/yum/apt-get repos from every EC2) Use infrastructure as code
  8. Enforce protections Apply SCP restrictions Block unwanted regions Protect defenses (e.g. cannot uninstall GuardDuty, the IAM role for Security) Automated remediation (remove unused IAM users) Refine IAM policies (refine the actions, resources, and conditions on IAM policies)
  9. Advanced defense Restrict metadata access Setup honeytokens
  10. Incident preparation Limit the blast radius of incidents Segment accounts further? Segment applications further? Practice responding to incidents (e.g. if an EC2 is compromised)
  11. Tagging Strategy Important for billing, identifying owners, if something is ‘Public’9. Advanced defense Restrict metadata access Setup honeytokens
  12. Further restrict who can do what E.g. Deny ability to create IAM users and access keys Deny the ability for other users to setup any networking Have additional sign off (so if a laptop is compromised, can’t merge code in)
  13. Have different policies for different environments

William Liu

github.com/williamqliu