¶ AWS
¶ Architecture
All services are hosted in a single AWS account, all in ca-central-1. At present, one server (dcerberus_com_web1) hosts all of dcerberus.com. This instance is a t3a.medium on an EC2 compute saving plan.
A single VPC with public and private subnets has been created, though most are not in use at present. As the site grows over time, new services and Lambda functions may be deployed in these subnets.
No NAT gateway or EC2 elastic load balancers are in use, and an
Arecord points directly at the elastic IP address associated with this EC2 instance.
My goals are to have this site be inexpensive and easy
¶ IAM
Native services are used for as much of the platform as possible (shy of using ECS or EKS), and reliance on instance profiles is preferred wherever posssible. However, some IAM users are created for specific purposes.
¶ Users
| Name | Role | Notes |
|---|---|---|
| rich | administrator | Me! |
| ses-smtp-user.20221104-191257 | none | Used for sending messages into SES. Uses the inline SES policy below. |
| dcerberus-uploads | none | uses persistent keys stored in .env.production. Uses the S3 inline policy below. |
S3 inline policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObjectAcl",
"s3:GetObject",
"s3:ListBucket",
"s3:DeleteObject",
"s3:GetBucketAcl",
"s3:PutObjectAcl"
],
"Resource": [
"arn:aws:s3:::BUCKET",
"arn:aws:s3:::BUCKET/*"
]
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": "s3:ListAllMyBuckets",
"Resource": "*"
}
]
}
SES inline policy below:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "ses:SendRawEmail",
"Resource": "*"
}
]
}
¶ Roles
| Role | Permissions |
|---|---|
| dcerberus_com_web | CloudWatchAgentAdminPolicy, CloudWatchAgentServerPolicy, AmazonSSMManagedInstanceCore, plus extra permissions |
Extra permissions policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"logs:PutRetentionPolicy",
"cloudwatch:GetMetricWidgetImage"
],
"Resource": "*"
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject",
"s3:PutObjectAcl"
],
"Resource": [
"arn:aws:s3:::BUCKET/metrics.png",
"arn:aws:s3:::BUCKET/*"
]
}
]
}
This
arn:aws:s3:::BUCKET/metrics.pngobject relates to the CloudWatch metric script below.
¶ CloudWatch dashboards
I like to keep things in a single screen with all the important data above the fold.
¶ CloudWatch agent
All servers run the CloudWatch Agent. The configuration is stored in SSM Parameter Store in the AmazonCloudWatch-linux key.
{
"agent": {
"metrics_collection_interval": 60,
"run_as_user": "root"
},
"logs": {
"logs_collected": {
"files": {
"collect_list": [
{
"file_path": "/var/log/mail.log",
"log_group_name": "mail.log",
"log_stream_name": "{instance_id}",
"retention_in_days": 30
},
{
"file_path": "/var/log/fail2ban.log",
"log_group_name": "fail2ban.log",
"log_stream_name": "{instance_id}",
"retention_in_days": 30
},
{
"file_path": "/var/log/auth.log",
"log_group_name": "auth.log",
"log_stream_name": "{instance_id}",
"retention_in_days": 30
},
{
"file_path": "/var/log/letsencrypt/letsencrypt.log",
"log_group_name": "letsencrypt.log",
"log_stream_name": "{instance_id}",
"retention_in_days": 30
},
{
"file_path": "/var/log/syslog",
"log_group_name": "syslog",
"log_stream_name": "{instance_id}",
"retention_in_days": 30
},
{
"file_path": "/var/log/nginx/access.log",
"log_group_name": "nginx",
"log_stream_name": "{instance_id}",
"retention_in_days": 30
},
{
"file_path": "/var/lib/docker/containers/*/*.log",
"log_group_name": "docker",
"log_stream_name": "{instance_id}",
"retention_in_days": 30
}
]
}
}
},
"metrics": {
"aggregation_dimensions": [
[
"InstanceId"
]
],
"append_dimensions": {
"AutoScalingGroupName": "${aws:AutoScalingGroupName}",
"ImageId": "${aws:ImageId}",
"InstanceId": "${aws:InstanceId}",
"InstanceType": "${aws:InstanceType}"
},
"metrics_collected": {
"disk": {
"measurement": [
"used_percent"
],
"metrics_collection_interval": 60,
"resources": [
"/"
]
},
"mem": {
"measurement": [
"mem_used_percent"
],
"metrics_collection_interval": 60
}
}
}
}
So far this approach has kept the use of CloudWatch under the free tier limit, even for logs. If this changes though then forwaring Nginx logs can be removed.
If you follow this approach, be mindful of two things: 1) CloudWatch Logs charge $0.50/GB beyond free tier, and 2) using Contributor Insights is AMAZINGLY useful - but does require CloudWatch Logs.
¶ CloudWatch metric script
I run a pair of scripts every hour that creates a static .png showing the server's CPU utilzation, credit balance, disk and memory consumption. This is pretty easy stuff to do, though it's not common knowledge that you can create static image widgets from CloudWatch this way.
Having this image available lets me pull it from my phone whenever I want to! Note that I pull it off of another domain, just in case something goes sideway with dcerberus.com.
First, here's the Python script that creates the image using the GetMetricImageWidget API:
#!/usr/bin/env python
import boto3
METRICS="""{
"sparkline": true,
"metrics": [
[ { "expression": "576-m4", "label": "[last: ${LAST}, max: ${MAX}, avg: ${AVG}] CPU credits used", "id": "e1", "period": 60 } ],
[ "CWAgent", "mem_used_percent", "InstanceId", "i-REMOVED", { "id": "m1", "label": "[last: ${LAST}, max: ${MAX}, avg: ${AVG}] Memory" } ],
[ ".", "disk_used_percent", ".", ".", { "id": "m2", "label": "[last: ${LAST}, max: ${MAX}, avg: ${AVG}] / disk" } ],
[ "AWS/EC2", "CPUUtilization", ".", ".", { "id": "m3", "label": "[last: ${LAST}, max: ${MAX}, avg: ${AVG}] CPU utilization" } ],
[ ".", "CPUCreditBalance", ".", ".", { "visible": false, "id": "m4" } ]
],
"view": "timeSeries",
"stacked": false,
"stat": "Average",
"period": 60,
"yAxis": {
"left": {
"min": 0,
"showUnits": false
}
},
"setPeriodToTimeRange": true,
"title": "Server metrics",
"legend": {
"position": "right"
},
"width": 1219,
"height": 349,
"start": "-PT3H",
"end": "P0D"
}"""
client = boto3.client('cloudwatch')
result = client.get_metric_widget_image(
MetricWidget=METRICS
)
handle = open('metrics.png', 'wb')
handle.write(result['MetricWidgetImage'])
handle.close()
I then call this from a shell script. Yes, calling Python from bash is superfluous - I may have more in the shell script in the future though, hence the two-script action.
#!/bin/bash
# Setup our Python env properly
. /root/venv/bin/activate
# Create our latest PNG file
if [ -e metrics.png ] ; then rm metrics.png ; fi
AWS_DEFAULT_REGION=ca-central-1 ./get_cw_metrics.py
# Upload to S3
aws s3 cp metrics.png s3://BUCKET/metrics.png
# Make it public
aws s3api put-object-acl --bucket BUCKET --key metrics.png --acl public-read
if [ -e metrics.png ] ; then rm metrics.png ; fi
And finally I call this from a crontab.
13 * * * * root /root/get_cw_metrics.sh >/dev/null
¶ CloudWatch alarms
The following have alarms created in CloudWatch, with notifications going to a single SNS topic to which my email has subscribed:
| Name | Threshold |
|---|---|
| Used memory | >85%,1 period |
| CPU utilization | >50%, 2 periods |
| CPU credit balance | =<100, 1 period |
| Root disk free | >75%, 1 period |
| EC2 status check | >0, 1 period |
There are many free or nearly free paging and operations tools I can use that are more feature-rich than simple email from a CloudWatch alarm. However, my goals are simple and inexpensive hosting.
¶ CloudFront
It is always safest to use a CDN for hosting assets, and CloudFront has a generous free tier. I have a single distribution that is linked to in the .env.production file, so my static assets are always fast and cheap.
¶ SES
The following are authorized sending addresses for this account:
| Name | Type |
|---|---|
| dcerberus.com | Domain |
| admin@dcerberus.com | |
| root@dcerberus.com | |
| notifications@dcerberus.com |
All messages are delivered to email-smtp.ca-central-1.amazonaws.com using TLS.
See Mail for local server/Postfix configuration details.
¶ Backup
There are two separate backup mechanisms used. First are the application backups that are listed in the backup page. However, EC2 Lifecycle Manager also creates an AMI for me every day. This way, I have "suspenders and a belt" should things go wrong 
¶ S3
Buckets for this project (actual names removed):
| Name | Purpose | Notes |
|---|---|---|
backups |
Daily backups for Mastodon | This bucket has a policy that expires objects after 30 days |
logs |
CloudFront logs | Not currently in use |
uploads |
Content (cache and same-site uplods) for Mastodon | This will always be the biggest bucket in the account |
Below is the policy for the uploads bucket:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Statement1",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::123456789012:user/UPLOAD"
},
"Action": [
"s3:PutObject",
"s3:PutObjectAcl",
"s3:GetObject",
"s3:GetObjectAcl",
"s3:DeleteObject"
],
"Resource": "arn:aws:s3:::UPLOAD/*"
},
{
"Sid": "statement2",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::123456789012:user/UPLOAD"
},
"Action": "s3:ListBucket",
"Resource": "arn:aws:s3:::UPLOAD"
}
]
}