May 29, 2020
DevOps Security and Privacy—FP Complete’s
comprehensive, easy to understand guide designed
to help you understand why they’re so critical to
the safety of your DevOps strategy.
The following is a transcription of a live
webinar given by FP Complete
Founder and Chairman Aaron Contorer, on
FP Complete's YouTube Channel.
Introducing Aaron
I’m the Founder and Chairman of FP Complete,
where we help companies use state-of-the-art tools and
techniques to produce secure, lightning-fast,
feature-rich software, faster and more often.
Before founding FP Complete, I was an
executive at Microsoft, where I served as program
manager for distributed systems, and general
manager of Visual C++, the leading software
development tool at that time. Also, I
architected MSN’s move to Internet-based server
software, served as the full-time technology
adviser to Bill Gates, and I founded and ran the
company’s Productivity Tools Team for complex
software engineering projects.
Okay, so enough about me. Let’s begin this
discussion by recognizing our industry’s
unfortunate—but preventable—reality:
Breaches are happening far too often
We all know how bad the state of the
world is within security and privacy
right now. Projects are getting very
complicated. And I—just as a sample—want
to point out that this is a very typical
breach. Monzo said that for six months
unauthorized people had access to
people’s secret code numbers, their pin
numbers. I’m not singling them out at
all, but rather saying… “This is very
typical.” They’re a bank, and they
compromised this type of data for months
and months.
How does it happen? It’s not only
because of logging and monitoring not
being in place, although that can be a
big factor. It’s because of complexity.
Honestly, we’re all trying very hard to
do our jobs, but users keep asking and
executives keep asking for new features.
And that integration just creates point
after point where problems can happen,
and things get overlooked.
Opportunities for penetration are everywhere
I would argue that today’s
applications are more about assembling
building blocks than they are about just
writing new code. But every time you
increase that complexity by adding more
building blocks, you increase the number
of interface points between
components—the number of places where
somebody might have done something wrong.
And so we’re really creating a system of
entry points between component A and
component B. But entry points—that sounds
like something I would compromise if I
were a security violator, right?
Furthermore, we’re manually configuring
our systems. People aren’t using
continuous deployment. And so there is
some wizard who’s supposed to go set up
the latest server or integrate it with
the database or integrate it with the web
with a firewall or whatever they’re
supposed to do. Every manual step creates
further opportunities for penetration,
for defects, because people are
imperfect. Even the best person in your
team doing a process a hundred times
might do it wrong, one or two times. An
automated scanner is going to find that
time, and it’s going to break into your
system before you know it.
Let’s talk DevSecOps
DevSecOps—DevOps with security stuck
right in the middle. And I think that’s a
good way of looking at this problem. We
want to integrate all the different parts
of our engineering into one pool of
automation, and include security and
quality assurance as part of that
automated process. We talked earlier
about automated testing being part of our
builds. But we want to go much farther
than that, as technical teams. We want to
start from the beginning of our projects,
talking about how secure they need to be.
What are the risks that they’re supposed
to defend against or or not create? We
want every member of the team to
understand that system downtime—because
somebody broke in and trashed it, or even
worse privacy violations which you can
never undo, because when people’s
personal information has been published,
you can’t unpublish it—we need to let our
team members know that these are
priorities and put them on the to-do list
for the project. And we can’t call it
something done if the security part isn’t
done. It’s not something we tack on at
the end. We don’t build in unsecured,
crazy, poorly architected apps, and then
at the end, ask someone to build a brick
wall around them. Because as soon as one
little person gets through the brick
wall, it’s open season. So, we want the
engineers to know everything they do
should be checked for security. That’s a
culture change to say that it’s
everyone’s job.
We need to integrate quality assurance
with security, which means somebody is
checking the software we wrote for
weaknesses; somebody is trying to break
in or, at least, trying to run tools that
will show us common ways to break in and
weather their presence.
And we need to inspect our cloud
systems that are running to make sure
that our deployment, and our system
operations and administration, is as
secure as we meant it to be. Did somebody
omit a step? We want to discover that
right away and fix it. Or, ideally,
automate the way we set up all of our
systems using, for example, an
orchestration software package to
automatically configure our servers, so
it isn’t the case that late in the day,
people are more likely to make a mistake.
Because, well written scripts do just as
good a job even when they’re tired.
And we want to make sure that all of
our systems are updated and patched and
not tell people that security is a waste
of time and they should get back to work
on features.
Process tips
To do all this, we need to have a
simple design. And I would encourage
people to focus on the idea that
simplicity and modular design are great
ways to make a system easier to check for
security holes.
We want to make sure that credentials
that are used in our modular
systems—where one piece of software is
logging into another service or another
piece of software database—are kept in
properly secured credential storage. A
common form of security violations is you
look at somebody’s source code and… Oh
look! There’s the password for the
database server right there …because the
app had to connect to the server. That’s
inappropriate design. There are special
credential storage services—your team
should use them.
And we want to make sure that quality
control remains central to our culture,
as developers of software, and that
includes DevOps, that includes system
administration. Too often, we have a good
piece of software, and then it’s deployed
incorrectly. And that’s where the problem
occurs. So if you’re going to test
whether your code is written properly,
maybe also test whether the servers
configured properly, from time to time.
It’s time well spent.
How to strengthen your security
So how can you move forward on
security? The good news is, while it may
sound like a scary and intimidating area,
there are lots of practical steps you can
take right now, and you don’t even have
to take them all at the same time, you
can take them incrementally. Here are
some great steps though that I highly
recommend.
One is that—in your engineering team,
and if you have multiple teams—in each
engineering team somebody is explicitly
the security person. Somebody knows that
it’s their job to keep an eye out for
security issues and prevention and that
if there’s a problem they’re the person
who’s going to hear about it. They should
have the power to look into anything they
need to make sure there isn’t a security
hole in the system.
Use best practices from other
companies. This is a great idea
throughout all of DevOps, including
DevSecOps. You don’t have to reinvent
anything. You can learn best practices
and get a checklist together of what
other companies have found helpful to
look for to find opportunities to secure
your system incrementally. We just piece
by piece chip away at the risks that are
present in our systems. We don’t have to
wait until some magic day when all of
security happens at once.
Teach your people about security. A
lot of security problems happen because
one person didn’t realize… Who didn’t
know that you’re supposed to not put
passwords in the source code where
everyone can see them? Well, one person
typed a password into the source code,
but now it’s there for everyone. So be
sure that training and security, and how
important it is, and how to do it is
available to everyone in your team. And
make sure that there’s a checklist. Who
took the security training? Who’s not
been to security training yet?
Scary but true fact: You should,
according to Price Waterhouse Coopers, if
you want to be a normal IT operation, be
spending 11 to 15% of your IT budget on
security overall. That’s a significant
number. And I think we can all agree that
with more internet work and more
importing of modules and stuff, we, if
anything, could be worried that that
number is going to go up. So automation
through DevOps is really a way to keep a
lid on that number. But I wouldn’t think
of it as a way to make that number drive
down towards zero. Security is everyone’s
job, and it’s going to remain that
way.
Beyond that, I’d say use it use the
other techniques we talked about earlier
in this presentation. You don’t have to
be the next Equifax, of having no
monitoring. You don’t have to allow silly
mistakes by having no automation. And you
don’t have to create more security holes
by reinventing your own tools and
processes using components. Reuse is your
friend.
7 tech ideas you can start now
I won’t spend too long on this, but I
wanted this for people who are more
hands-on or the people who are
supervising hands-on engineers. These are
some practical steps that you can take to
start turning on pieces of security,
right now. Every one of these—except
perhaps service-oriented architecture—is
something that literally you could task
somebody to do this week or next
week.
These are straightforward tasks.
Ensure all databases have firewalls on them. They’re a common data breach source!
Use a password manager to generate secure passwords; enable two-factor authentication.
Use roles and policies to assign specific permissions to users and services instead of running everything from root credentials or privileged users.
Use bastion hosts or VPNs to limit access to internal machines.
Use service-oriented architecture (SOA) to break off components that need high privilege.
Include code analysis tools in the dev process and enforce fixes prior to deployment.
Test your servers with automated scanners for break-in vulnerabilities.
Fast to market, reliable, and secure
It’s a winning formula!
So, in short, you have a choice to
turn on DevOps to use a lot of technology
that’s been solved, a lot of best
practices and engineering techniques that
have already been solved and tested at
numerous other companies—clients of ours,
famous internet companies, everyone. When
I say “everyone”, the truth is the
minority of companies are already using
proper DevOps. But enough companies that
you don’t have to be the first, you don’t
have to be the Pioneer. DevOps is a
winning formula that will get you to
market faster, and more reliable, and
with better security. Or you could be the
next Equifax and the next Capital One,
which is the default situation.
Need help with DevOps Security and Privacy?
FP Complete offers corporations its
DevOps Success Program which offers
advanced Privacy and Security software
engineering mentoring among many other
moving parts in the DevOps world.
For more information, please contact us or see our DevOps homepage.