How we assured the reliability of our CryptX 2.0 CTF platform
Way we ensured a safe, reliable, fair Gameplay
Malin Dhamsara, Waruna Udara
Keeping a CTF Platform Online for 80+ Teams
More than 80 teams, four people per team, and AI-assisted solvers all hitting the scoreboard at once. We had one job: keep the platform useful while players, scripts, scanners, and challenge traffic arrived together. This is how a student team prepared the infrastructure, what held, where latency appeared, and which assumptions we would challenge before the next event.
The short version: We ran the event platform on a 16 GB DigitalOcean droplet, put Cloudflare and BunkerWeb in front of it, watched the host and services with Prometheus and Grafana, and load-tested with Grafana Cloud k6 plus browser and connection tests. Latency rose under pressure. The reliability report recorded no HTTP 502, 503, or 504 responses during its run. An operating-system update also stopped running processes and required hands-on recovery. Those details belong in the reliability story too.
A CTF is a production system with a scoreboard
A capture-the-flag platform has a traffic pattern most web apps never see. The quiet period before the start is followed by a crowd signing in, refreshing the scoreboard, opening challenges, and submitting flags in the same few minutes. Add browser automation, command-line scripts, security scanners, and AI-assisted solving workflows, and an ordinary application starts behaving like a very enthusiastic queueing demo.
CryptX CTF drew more than 80 teams, with four members per team. That meant capacity for at least 320 competitors before counting organizers and observers. Our team also carried the names of our university, faculty, and society. Reliability was a responsibility, not a slide with a green checkmark.
Site reliability engineering gave us a useful frame. Instead of asking only “does the homepage open?”, we asked what could fail, how we would notice, what players would experience, and what evidence would show whether a change helped.
We had read Medium infrastructure writeups as a team and compared their approaches with reports from larger competitions. One useful pattern: the platform, its database, and per-team challenge instances have different scaling needs. Kubernetes-based challenge spawning and infrastructure as code can help with isolated instances, while also adding operational complexity and new failure modes. We borrowed questions and design ideas, not another event’s architecture as a blueprint. Useful references include DownUnderCTF’s writeups on scalable CTF infrastructure and surviving the first five minutes. Our event was much smaller, so we used that material to inform decisions rather than copy its capacity claims or topology.
The people behind the platform
One student led much of the infrastructure provisioning and platform customization, bringing several years of software development and cloud experience. That work connected the customized CTFd deployment to the challenge-management workflow, rather than treating the platform as an untouched container image. Another student took the lead on security, putting the web application firewall layers, rate limits, and related protective measures in place. The rest of us contributed testing, monitoring, review, and event operations. The infrastructure was a team responsibility even when individual parts had clear owners.
That division of work helped us move in parallel: one person could improve the deployment path while another examined how traffic reached the application. It also made review important. In a security event, a configuration change can affect every player, so the team needed to understand not only what a setting did, but what a false positive would look like.
One player path, separate kinds of state
The simplest way to understand the system is to follow a request. It passes through edge protection, a second inspection layer, the web proxy, and the CTFd application. CTFd then reads or updates persistent competition data and uses a cache service for short-lived or frequently accessed information. Challenge containers have a different job from the scoreboard and flag-submission path.
The platform repository describes a Docker Compose deployment with CTFd, Nginx, MariaDB, and Redis. The CTFd application handles logins, teams, challenge pages, scoring, and submissions. MariaDB holds durable state such as users, teams, and submissions. Redis provides a fast service for caching and other short-lived coordination. Uploaded files also need durable storage and a sharing plan if application replicas are introduced.
Kubernetes belongs in the picture for challenge instances, not as a claim that the whole event platform ran in Kubernetes. The platform code includes a DigitalOcean Kubernetes Service configuration and challenge-manager material. That is a separate scaling problem: challenge containers can be isolated and created on demand, while the web platform and its shared records remain the control plane. The important architectural point is that a burst of challenge-instance work does not have to share the same scaling unit as logins, scoring, and flag submissions.
Why we scaled the main host vertically
We hosted the event’s main platform on DigitalOcean and vertically scaled the server to 16 GB of RAM. Vertical scaling means giving the existing host more resources. It let us keep the deployment shape familiar while adding headroom before a time-critical event.
Our platform has stateful workflows. A flag submission changes competition data; teams, scores, sessions, uploads, and challenge state must remain consistent. That shaped our risk analysis, but it does not mean CTFd cannot scale horizontally. CTFd’s own documentation describes running duplicate application servers against the same MySQL and Redis configuration. Horizontal scaling is possible when the shared database, session configuration, uploads, proxy, and deployment process are designed for it.
For this event, we chose a larger known host over introducing and validating multiple web nodes at the last minute. That was a scoped operational choice, not a rule that stateful applications must stay on one server. The tradeoff is clear: a single host is simpler to reason about, but it remains a single failure domain. Vertical scaling increases capacity; it does not provide host redundancy.
Provisioning that can follow the team
Terraform made the cloud setup more repeatable. Rather than reconstructing a server by hand, the repository carries provider-specific configurations for DigitalOcean and Azure, along with a DigitalOcean Kubernetes path for challenge infrastructure. The deployment workflow accepts provider choices and reads the corresponding cloud credentials from separate GitHub secrets. That means the same team can provision into different cloud accounts by selecting the intended provider configuration and credentials.
There is an important nuance: Terraform is not a magic button that makes every cloud identical. Each provider has its own resources and configuration. The benefit was a repeatable workflow and reusable operational knowledge, with separate provider-specific code where the clouds differ. Terraform’s provider model supports multiple provider configurations, including aliases, but each target still needs valid credentials, state management, and a tested plan.
Vagrant served a different purpose. The checked-in Vagrantfile creates a local virtual machine for development and bootstraps a CTFd environment. That gave contributors a more consistent place to work without treating a laptop as the production cloud. Terraform provisioned remote infrastructure; Vagrant reproduced a development machine. They complemented each other rather than doing the same job.
This work was led by a student with several years of software development and cloud experience. It turned “I can make a server work” into “the team can provision a known environment again, inspect changes, and choose a supported target without starting from scratch.”
Two security layers, with real players behind them
We placed Cloudflare at the edge and BunkerWeb in front of the application as a second inspection layer. Cloudflare could challenge or limit suspicious traffic before it reached the origin. BunkerWeb added another web application firewall and rate-control point closer to the application. We also used supporting security measures around the platform. A CTF attracts people who are good at finding unexpected behavior, so the origin was never a place to assume that every request was friendly.
Layering is useful, but two filters can also create two places to misconfigure. We needed to understand the real client IP forwarded between layers, keep the origin reachable only through intended paths where possible, and examine false positives alongside blocked traffic. A rate limit that protects the server but blocks a team after a legitimate solve is still a player-facing incident. Cloudflare’s rate-limiting rules can match traffic and apply actions such as blocking or challenging; BunkerWeb also documents request and connection limiting. We should keep exact rule thresholds and allowlists private unless publishing them is safe.
The repository’s SRE report observed Cloudflare challenge responses and HTTP 429 responses during testing. It does not provide an independent count for each WAF, so we do not claim a specific number of requests blocked by BunkerWeb.
Load testing from more than one laptop
We used Grafana Cloud k6 for burst testing and ran load tests from several team members’ personal computers at the same time. That gave the exercise multiple independent traffic sources instead of making one laptop responsible for every request. We also used browser automation to exercise rendered pages and a Go tool to apply connection pressure. Prometheus and Grafana let us watch the target while the tests ran.
The checked-in test suite describes a staged k6 profile that reaches 2,000 virtual users in about four and a half minutes. A separate authenticated script defines a longer profile that can reach 5,000 virtual users. Those are test configurations, not proof that every configured level completed successfully. The report we reviewed recorded approximately 2.1 seconds average response time at baseline and observations around 3.0 to 3.9 seconds under heavier pressure. It reported zero observed 502, 503, and 504 responses, alongside challenges and 429 rate limits. Its final total-request field was pending, and raw k6 results were not present in the checkout.
The hosted Grafana environment gave us both observability and a managed k6 option within the free allowance available to our team at the time. That reduced the amount of test infrastructure we had to operate ourselves. Free-plan quotas can change, so the reusable lesson is to calculate test volume against the account’s current allowance before a large run. Grafana documents performance-test limits in virtual-user-hours and per-test VU and duration quotas.
One more important limit: the SRE scripts include a basic k6 request profile as well as an authenticated profile. A virtual user is not automatically equivalent to a real competitor. For the next run, we would model more of the actual journey: login, challenge listing, challenge files, flag submissions, scoreboard refreshes, and the pauses between those actions. We would also report the number of load generators, the combined traffic profile, and which routes each generator exercised.
The Web server lesson: a single worker was something an unexpected setting
The platform checkout’s production Compose file sets WORKERS=1. Its entrypoint defaults the Gunicorn worker class to gevent. That detail matters. A single worker process does not mean every request is necessarily handled in a strictly one-at-a-time synchronous way: gevent can interleave I/O-bound work cooperatively. But a single process still has limits. Blocking calls can stall cooperative progress, CPU-bound Python work does not become parallel just because requests are concurrent, and a single worker configuration may leave available CPU capacity unused.
We saw latency grow as pressure rose and identified Gunicorn worker/concurrency configuration as an area to inspect. We should not present it as the proven root cause without profiling data. Database query time, connection pools, CPU saturation, WAF challenges, network distance, and application work can all contribute. Gunicorn’s own documentation distinguishes synchronous workers, which handle one request at a time, from asynchronous worker classes such as gevent. The right fix is measured: profile the request path, tune worker count and class, test with the real workload, and watch memory and database connections while doing so.
The maintenance incident we needed to include
An operating-system update stopped running processes, and we recovered them without rebooting the host. We do not yet have an event log that names the affected processes or explains exactly why they stopped, so we should not assign a root cause. The codebase gives us two useful clues, not a diagnosis: Terraform cloud-init upgrades packages during first-boot provisioning, while the Docker Compose services are configured with restart: always. The first setting is about initial provisioning, not proof that an unattended update caused this incident. The restart policy is a recovery aid, not a substitute for checking whether every dependency and application endpoint is healthy.
For the next maintenance run, record the update window, affected services, recovery commands, health checks, and player impact. Check Docker and host service logs around the event, confirm the containers actually restarted, and document dependency ordering, backups, and rollback steps. We should publish a duration or a more specific root cause only after checking logs or the team timeline. This is why an accurate reliability claim needs a time window: “no observed application errors during the load run” is more precise than “nothing ever went down.”
What held, and what the graphs asked us to improve
For the load exercise, the report recorded no observed 502, 503, or 504 responses. The team reports no request failures caused by competition traffic, while latency increased with load. Separately, an operating-system update stopped processes without rebooting the host, and the team recovered them. We still need the incident timeline and player-impact details before making an unqualified zero-downtime claim for the entire event. A challenge or HTTP 429 is a response, but it may not be the response a legitimate player expected.
Our dashboards helped us watch host and service behavior rather than rely on how calm the operations room felt. Calm is good. A p95 panel and a restart checklist are better. We used Prometheus for metrics collection and Grafana for the dashboard view. For the next event, we want service objectives that name a latency target, an acceptable error rate, and an alert threshold. That makes “reliable” something we can test and discuss rather than a feeling we had after the scoreboard stayed online.
Attack our own decisions before someone else does
The most useful part of SRE is learning to question decisions that worked once. These are the questions we are taking into the next planning cycle:
- Could one host disappear? We used one main server. We should decide what recovery time we can accept and whether an automated rebuild, spare host, or replicated application tier is worth the cost and operational complexity.
- Did one Gunicorn worker cap concurrency? The repository says one worker. We should profile first, tune worker count and class, and test against authenticated journeys rather than guessing at a universal worker formula.
- Are rate limits fair to teams behind shared networks? Calibrate rules against legitimate burst behavior, identify whether counters are keyed by IP or account/session, and check 429s alongside reports from players.
- Did our load test prove the player experience? Combine client machines carefully, record actual requests and percentiles, include real workflows, and separate a WAF block from an application failure.
- Was the challenge cluster part of this event? The repository contains Kubernetes and DOKS paths. Before claiming live dynamic challenge scale, map exactly which services ran on the cluster and which ran on the 16 GB host.
- Can another student operate the system? Keep infrastructure plans, access boundaries, dashboards, incident steps, backup checks, and rollback instructions understandable to the next person on call.
We also need to review the management plane with the same care as player traffic. The Terraform examples available to us allow SSH from any source address. Before reusing those examples, we would restrict administrative access to trusted addresses or a VPN, then confirm that the live cloud firewall matches the intended policy. That is the kind of uncomfortable code review that makes the next event safer.
Those are not reasons to dismiss what worked. They are the work left by a successful event. The site stayed useful under the competition traffic we observed, and our testing surfaced latency, concurrency, maintenance, and evidence gaps we can address. The aim is not to make the architecture sound invincible. It is to make the next version easier to operate and harder to surprise.
We started as students who cared about the event and the names attached to it. We came away with a more practical definition of reliability: provision it repeatably, protect it thoughtfully, push it before the deadline, watch it while real users depend on it, and be honest about what the graphs can prove.
In a CTF, finding the flag is the challenge. Keeping the service available while everyone searches for it is ours.
References
CTFd load and site reliability test repository
CTFd installation and deployment documentation
Terraform provider configurations and aliases
Grafana Cloud k6 limits and quotas
Creating Scalable CTF Infrastructure on Google Cloud Platform with Kubernetes and App Engine, Medium
How to run a CTF that survives the first 5 minutes, Medium