03 Jan 2020

Securing your Kubernetes WebApp on a Shoestring.

Here we’re going to cover:

Installing cert-manager on your Raspberry Pi k3s cluster.
Installing a LetsEncrypt Cluster Issuer
Getting a cert issued.
Using the nginx inbuilt OWASP Modsecurity Ruleset to protect your site from attackers.

It’s a pre-requisite that the ingress-nginx controller is being used.

Installing Cert-Manager

Cert-Manager makes managing SSL certs on your Kubernetes cluster child’s play. It watches your Kubernetes ingress resources for changes and if you add the required annotations, it will try to issue an SSL cert from the provider you configure. In this case we can use LetsEncrypt.

Thanks to the Cert-Manager team, from v0.12.0 it now has multi-arch builds, which means it has now an arm build and just works.

To install it on your cluster simply issue:

kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v0.12.0/cert-manager.yaml

You will see the Cert-Manager pods come up:

➜  ~ kubectl get pods -n cert-manager
NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-webhook-547567b88f-wvx5k      1/1     Running   0          16m
cert-manager-5c47f46f57-kdsz2              1/1     Running   0          16m
cert-manager-cainjector-6659d6844d-xlsqp   1/1     Running   0          16m

Lets Encrypt

Now you must set up LetsEncrypt as a cluster issuer. This is quite simple also.

Create a manifest like this one:

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
 name: letsencrypt-prod
spec:
 acme:
   # The ACME server URL
   server: https://acme-v02.api.letsencrypt.org/directory
   # Email address used for ACME registration
   email: [email protected]
   # Name of a secret used to store the ACME account private key
   privateKeySecretRef:
     name: letsencrypt-prod
   # Enable the HTTP-01 challenge provider
   solvers:
   - http01:
       ingress:
         class: nginx

Make sure to replace [email protected] with your actual email address

After you’ve loaded the manifest it should be visible using this command:

➜  ~ kubectl get clusterissuer
NAME               READY   AGE
letsencrypt-prod   True    5d20h

At this point we should be able to deploy an ingress definition that requests a cert.

It’s important the the DNS entry for the domain you want the cert for points to the load balancer.

➜  ~ kubectl get svc -n ingress-nginx
NAME            TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)                      AGE
ingress-nginx   LoadBalancer   10.43.81.154   188.141.110.72   80:30712/TCP,443:31239/TCP   5d21h

Cert issuance.

This is done by putting the correct annotations on an ingress definition:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: basic-ingress
  namespace: blogdemo
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    kubernetes.io/ingress.class: "nginx"

spec:
  tls:
  - hosts:
    - blogdemo.home.dmarkey.com
    secretName: blogdemo-tls
  rules:
  - host: blogdemo.home.dmarkey.com
    http:
      paths:
      - backend:
          serviceName: my-service
          servicePort: 80

After you create this ingress resource, the cert-manager pod will create a corresponding cert resource.

See the cert resources in your namespace by using this command.

➜  ~ kubectl get cert -n blogdemo
NAME           READY   SECRET         AGE
blogdemo-tls   False   blogdemo-tls   9s

Notice the Ready status is False. This means the cert has not been completely initialised yet.

If everything is configured correctly, this should switch to True, and then the cert should be ready(takes upto 30s).

➜  ~ kubectl get cert -n blogdemo
NAME           READY   SECRET         AGE
blogdemo-tls   True    blogdemo-tls   14m

You can see the new cert in action using curl.

  ~ curl -vvv https://blogdemo.home.dmarkey.com
*   Trying 188.141.110.72:443...
* TCP_NODELAY set
* Connected to blogdemo.home.dmarkey.com (188.141.110.72) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=blogdemo.home.dmarkey.com
*  start date: Jan  3 16:37:57 2020 GMT
*  expire date: Apr  2 16:37:57 2020 GMT
*  subjectAltName: host "blogdemo.home.dmarkey.com" matched cert's "blogdemo.home.dmarkey.com"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0xf36148)
> GET / HTTP/2
> Host: blogdemo.home.dmarkey.com
> user-agent: curl/7.67.0
> accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 503
< server: openresty/1.15.8.2
< date: Fri, 03 Jan 2020 17:52:59 GMT
< content-type: text/html
< content-length: 203
< strict-transport-security: max-age=15724800; includeSubDomains

Modsecurity Firewall

The final piece is to protect your site with a WAF(Web Application Firewall).

Ingress-nginx comes with the excellent Modsecurity by default, and it also comes with the OWASP ModSecurity Core Rule Set (CRS) which protects against various different attacks.

The documentation for the modsecurity 3rd party add on is here: here but it does detail that the firewall is by default in Audit mode only and will not block any traffic.

Given our ingress example from earlier, here is a further example with the ModSecurity WAF in blocking mode.

piVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: basic-ingress
  namespace: blogdemo
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/enable-modsecurity: "true"
    nginx.ingress.kubernetes.io/modsecurity-snippet: |
        SecRuleEngine On
        SecRequestBodyAccess On
        SecAuditEngine RelevantOnly
        SecAuditLogParts ABIJDEFHZ
        SecAuditLogFormat JSON
        SecAuditLogType Serial
        SecAuditLog /dev/stdout
        Include /etc/nginx/owasp-modsecurity-crs/crs-setup.conf
        SecAction \"id:900200,phase:1,nolog,pass,t:none,setvar:\'tx.allowed_methods=GET\'\"
        Include /etc/nginx/owasp-modsecurity-crs/rules/REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf
        Include /etc/nginx/owasp-modsecurity-crs/rules/REQUEST-901-INITIALIZATION.conf
        Include /etc/nginx/owasp-modsecurity-crs/rules/REQUEST-903.9001-DRUPAL-EXCLUSION-RULES.conf
        Include /etc/nginx/owasp-modsecurity-crs/rules/REQUEST-903.9002-WORDPRESS-EXCLUSION-RULES.conf
        Include /etc/nginx/owasp-modsecurity-crs/rules/REQUEST-905-COMMON-EXCEPTIONS.conf
        Include /etc/nginx/owasp-modsecurity-crs/rules/REQUEST-910-IP-REPUTATION.conf
        Include /etc/nginx/owasp-modsecurity-crs/rules/REQUEST-911-METHOD-ENFORCEMENT.conf
        Include /etc/nginx/owasp-modsecurity-crs/rules/REQUEST-912-DOS-PROTECTION.conf
        Include /etc/nginx/owasp-modsecurity-crs/rules/REQUEST-913-SCANNER-DETECTION.conf
        Include /etc/nginx/owasp-modsecurity-crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf
        Include /etc/nginx/owasp-modsecurity-crs/rules/REQUEST-921-PROTOCOL-ATTACK.conf
        Include /etc/nginx/owasp-modsecurity-crs/rules/REQUEST-930-APPLICATION-ATTACK-LFI.conf
        Include /etc/nginx/owasp-modsecurity-crs/rules/REQUEST-931-APPLICATION-ATTACK-RFI.conf
        Include /etc/nginx/owasp-modsecurity-crs/rules/REQUEST-932-APPLICATION-ATTACK-RCE.conf
        Include /etc/nginx/owasp-modsecurity-crs/rules/REQUEST-933-APPLICATION-ATTACK-PHP.conf
        Include /etc/nginx/owasp-modsecurity-crs/rules/REQUEST-941-APPLICATION-ATTACK-XSS.conf
        Include /etc/nginx/owasp-modsecurity-crs/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf
        Include /etc/nginx/owasp-modsecurity-crs/rules/REQUEST-943-APPLICATION-ATTACK-SESSION-FIXATION.conf
        Include /etc/nginx/owasp-modsecurity-crs/rules/REQUEST-949-BLOCKING-EVALUATION.conf
        Include /etc/nginx/owasp-modsecurity-crs/rules/RESPONSE-950-DATA-LEAKAGES.conf
        Include /etc/nginx/owasp-modsecurity-crs/rules/RESPONSE-951-DATA-LEAKAGES-SQL.conf
        Include /etc/nginx/owasp-modsecurity-crs/rules/RESPONSE-952-DATA-LEAKAGES-JAVA.conf
        Include /etc/nginx/owasp-modsecurity-crs/rules/RESPONSE-953-DATA-LEAKAGES-PHP.conf
        Include /etc/nginx/owasp-modsecurity-crs/rules/RESPONSE-954-DATA-LEAKAGES-IIS.conf
        Include /etc/nginx/owasp-modsecurity-crs/rules/RESPONSE-959-BLOCKING-EVALUATION.conf
        Include /etc/nginx/owasp-modsecurity-crs/rules/RESPONSE-980-CORRELATION.conf
        Include /etc/nginx/owasp-modsecurity-crs/rules/RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf

spec:
  tls:
  - hosts:
    - blogdemo.home.dmarkey.com
    secretName: blogdemo-tls
  rules:
  - host: blogdemo.home.dmarkey.com
    http:
      paths:
      - backend:
          serviceName: my-service
          servicePort: 80

Lines of note:

        SecRuleEngine On

This sets the rule engine to block traffic that ModSecurity deems to have broken the ruleset. Users will get a 403

        SecAction \"id:900200,phase:1,nolog,pass,t:none,setvar:\'tx.allowed_methods=GET\'\"`

Be default, the ruleset allows GET,POST and PATCH. In this case I only want GET requests to be allowed - be careful of escaping.

        SecAuditLogFormat JSON
        SecAuditLogType Serial
        SecAuditLog /dev/stdout

These lines instructs ModSecurity to log to stdout(this will be of stdout of the nginx ingress controller)

Assuming you’re using some log shipper to Elasticsearch etc, you will get nice JSON output when a request is blocked.

David's Tech Blog

Securing your Kubernetes WebApp on a Shoestring.

Installing Cert-Manager

Lets Encrypt

Cert issuance.

Modsecurity Firewall