Security, Agile and DevOps

June 13th, 2020 by Stephen Jones Leave a reply »

As we move to an era of no code citizen developers there is increasing risk that security remains an afterthought when organizations are building software. The latest Verizon threat report identified that web application attacks have doubled, and that cloud-based data is under attack. The surge in web app security breaches in 2019 further solidifies that ‘crowd funded’ testing is no substitute for proper QA. The whole agile /DevSecOps approach has done much to improve user feedback to developers to improve the functionality and speed to market of business solutions, but informal end user tests alone are not sufficient where security is concerned,

With the rush to embrace digital services, organizations are too often focused on the speed of release rather than on the quality of services. To accelerate the pace of digital transformation, security must be a fundamental part of software development. To develop code faster, you should also identify vulnerabilities sooner. Otherwise, you run the risk of DevOps, simply creating software with vulnerabilities, more quickly.Embed security within all aspects of your software deign and development process rather than expect it to be bolted on as an afterthought. The threat is real sophisticated and growing. Criminals also use automation and Machine intelligence to identify and to attack vulnerabilities faster.

Attackers recently hijacked powerful machine-learning clusters inside Microsoft’s Azure cloud-computing service so that they could mine cryptocurrency at the expense of the customers who rented services. The nodes, which were misconfigured by customers, made the perfect target for so-called cryptojacking schemes. Machine-learning tasks typically require vast amounts of computing resources. By redirecting thsoe to perform the compute-intensive workloads required to mine digital coins, the attackers found a means to generate large amounts of currency at little, or no cost.

The infected clusters were running Kubeflow, an open source framework for machine-learning applications in Kubernetes, which is itself an open source platform for deploying scalable applications across large numbers of computers. Microsoft said compromised clusters it discovered numbered in the “tens.” Many of those ran an image available from a public repository, apparently to save users the hassle of creating one themselves. Upon further inspection, Microsoft investigators discovered it contained code that surreptitiously mined the Monero cryptocurrency.

After finding the infected clusters, investigators turned their attention to how the machines were compromised. For security, the dashboard that allows administrators to control Kubeflow is, by default, accessible only through istio ingress, a gateway that’s typically located at the edge of the cluster network. The default setting prevents people across the Internet from accessing the dashboard and making unauthorized changes to the cluster.

This week Yossi Weizman, a security-research software engineer in the Azure Security Center, said : “We believe that some users chose to do it for convenience. Without this action, accessing the dashboard requires tunneling through the Kubernetes API server and isn’t direct. By exposing the Service to the Internet, users can access the dashboard directly. However, this operation enables insecure access to the Kubeflow dashboard, which allows anyone to perform operations in Kubeflow, including deploying new containers in the cluster.”

Once attackers have access to the dashboard, they have multiple options for deploying backdoored containers in the cluster. For instance, attackers can create what’s known as a Jupyter Notebook server that runs on the cluster. They can then place a malicious image inside of the Jupyter Notebook. If a Jupyter Notebook is already installed, it can be maliciously modified.
Weizman wrote.:” Azure Security Center has detected multiple campaigns against Kubernetes clusters in the past that have a similar access vector: an exposed service to the Internet. However, this is the first time that we have identified an attack that targets Kubeflow environments specifically.”


Comments are closed.