00:00:00.900
foreign
00:00:18.859
Portland
00:00:20.480
right uh so not that this joke requires
00:00:24.180
any further elaboration I hear jokes are
00:00:27.420
best uh when you have to explain them so
00:00:31.199
in a way I guess we don't because
00:00:32.460
Aaron's already done the whole Portland
00:00:33.899
Seattle thing but
00:00:36.239
apparently there's a rivalry between
00:00:38.340
these two cities
00:00:39.840
um did y'all know that I didn't uh but
00:00:42.059
clearly I thoroughly researched this
00:00:44.879
um I'm definitely a team Portland though
00:00:46.559
and that's not because we're physically
00:00:48.239
here and I'm scared any portlanders out
00:00:50.160
there will egg me
00:00:51.840
um anyway welcome back from lunch
00:00:54.539
everyone we're really excited to have
00:00:56.879
you all join us for a fun-filled half
00:00:58.620
hour of learning laughing and maybe a
00:01:01.559
little bit of groaning if none of our
00:01:02.940
jokes land
00:01:04.680
um but if you do make it through this
00:01:07.020
whole talk we have a fun little surprise
00:01:08.520
for you at the end so if you're ready
00:01:10.619
give us a verbal bat signal of sorts
00:01:13.619
we're looking for whoop whoops or uh
00:01:16.500
oh
00:01:19.640
nice we're getting lots of whoop whoops
00:01:22.380
and caca so uh let's get started well
00:01:25.320
hello everyone my name is Betty and I'm
00:01:27.659
feeling super lucky to Grace the stage
00:01:29.640
with the effortlessly smart and
00:01:31.500
wonderfully kind human Ashley Ellis
00:01:34.020
Pierce
00:01:41.700
now gassing ourselves up I honestly
00:01:44.040
think the world of Ashley by the way uh
00:01:46.680
I think Ashley is fire or as some kids
00:01:49.619
say Ashley is lit
00:01:52.399
quite literally too though like she just
00:01:56.100
lit herself on fire the other day I'm
00:01:59.040
not even joking
00:02:00.960
uh that's her favorite shirt
00:02:04.140
um I know yeah very embarrassing anyway
00:02:08.340
I'm so excited to do this talk with her
00:02:10.739
because I feel like we channeled the
00:02:12.300
energy of a badass dynamic duo I'd
00:02:15.660
compare us to the likes of Batman and
00:02:17.879
Robin or like Anna and Elsa but then I
00:02:21.720
started digging a bit deeper and with
00:02:23.879
some reflection I actually realized we
00:02:25.860
might be more like Pinky and the Brain
00:02:28.560
because like Ashley brings the smartest
00:02:31.500
obviously and I bring a lot of other
00:02:34.200
things to this uh relationship too okay
00:02:36.599
so it's not a knock on myself by any
00:02:38.940
means I also quickly want to share that
00:02:41.459
I'm a first time conference speaker and
00:02:46.160
thank you
00:02:47.879
and uh I'm speaking at railsconf is
00:02:50.940
definitely a career bucket list item for
00:02:52.620
me so it's an honor to be here but I'm
00:02:55.319
not gonna lie my Palms are a bit sweaty
00:02:57.599
knees weak arms are heavy the vomit on
00:02:59.459
my sweater already mom's spaghetti she's
00:03:00.959
nervous on the surface she looks calm
00:03:02.400
and ready to drop whoa I think Eminem
00:03:05.280
just took over my body just now I think
00:03:07.860
what he's trying to help me articulate
00:03:09.420
is that I may or may not be a little bit
00:03:11.400
nervous
00:03:14.040
um but I'm working on channeling all
00:03:15.420
that energy into some uh pure
00:03:17.900
unadulterated excitement
00:03:20.580
so bear with me if I'm a bit of a loose
00:03:22.800
cannon today like uh right now hi Mom
00:03:25.680
and also hello Sandy Metz the YouTube
00:03:28.800
algorithm is probably gonna get her to
00:03:30.599
watch our talk at some point right maybe
00:03:32.640
you say her name enough yeah if I say
00:03:34.319
her name okay uh I'm really taking us
00:03:36.840
off base right now and Ashley's gonna
00:03:38.700
Reign me in soon but before we do that
00:03:40.739
we want to build some positive energy in
00:03:42.659
this room so can everyone just face
00:03:44.940
their neighbors and give them a socially
00:03:46.620
distance but enthusiastic high five
00:03:49.760
nice nice okay and can you send some
00:03:52.980
high fives up here to Ashley and I as
00:03:54.659
well
00:03:55.560
thank you
00:03:57.140
I like to call these y5s by the way
00:04:00.000
because it's like a high five over the
00:04:02.220
Wi-Fi huh
00:04:04.159
okay okay now this is all out of my
00:04:06.659
system let's get to the good stuff
00:04:07.980
Ashley what are people here for great
00:04:10.860
question Betty so as you all know the
00:04:13.260
title of our talk is Gem install what
00:04:15.180
could go wrong and spoiler the answer is
00:04:18.120
a lot we put a lot of faith in gems and
00:04:20.880
we want to make sure that they're safe
00:04:22.860
Betty and I work together at Shopify on
00:04:24.960
the Ruby conventions team our team Works
00:04:27.840
to make sure that shopify's many
00:04:29.340
hundreds of Ruby apps are all in Tip-Top
00:04:31.500
shape so if for example there's a ruby
00:04:34.380
cve we would make sure none of the over
00:04:36.900
600 Ruby apps are running the vulnerable
00:04:38.940
version
00:04:39.960
and so in that work we started to
00:04:42.000
consider how do we make sure that none
00:04:45.000
of these apps download an unsafe gem
00:04:47.639
and that led us into the
00:04:49.080
all-encompassing multi-faceted world of
00:04:51.600
supply chain security so that's what
00:04:53.580
we're here to talk about today
00:04:55.560
when you install a gem what could go
00:04:57.660
wrong I mean that gem looks so cute and
00:05:00.479
friendly right totally like how could
00:05:02.280
that little gem be harmful
00:05:04.560
but turns out a lot could go wrong the
00:05:07.080
security of the gems that we use is
00:05:08.820
incredibly important
00:05:10.860
because when we run gem install we're
00:05:13.380
just putting unquestioned faith in
00:05:15.240
whatever we're installing and the
00:05:17.100
security of our gems is getting even
00:05:18.840
more important every year
00:05:21.660
according to sonotype's 2021 state of
00:05:24.120
the software supply chain report in 2021
00:05:27.000
there was a 650 percent increase in
00:05:30.060
software supply chain attacks aimed at
00:05:32.220
exploiting weaknesses and Upstream open
00:05:33.900
source ecosystems
00:05:35.639
in 2020 there was a 430 increase over
00:05:39.120
2019. so every year this is accelerating
00:05:42.539
and there are more and more people
00:05:43.800
trying to make sure that when you run
00:05:45.419
gem install something indeed goes wrong
00:05:48.780
there are a couple of notable supply
00:05:50.160
chain attacks from last year that you
00:05:51.840
might have heard of
00:05:53.220
one was the code COV attack codecov runs
00:05:56.039
a code coverage report on your CI
00:05:57.600
environment and then uploads that report
00:05:59.340
to codecut servers but between January
00:06:02.340
and April 2021 its stock range was
00:06:05.039
modified so for that for codecuts 23 000
00:06:07.680
customers that uploader also sent their
00:06:10.199
environment variables to the hackers
00:06:12.000
remote server
00:06:13.979
another supply chain attack from 2021
00:06:16.199
was the dependency confusion attack
00:06:17.900
unlike the code Cub attack this wasn't
00:06:20.460
taking advantage of one company in the
00:06:22.139
supply chain and going after all of
00:06:23.639
their customers this was taking
00:06:25.319
advantage of a more General
00:06:26.520
vulnerability and how companies
00:06:28.319
structure the way they get packages
00:06:30.360
basically taking advantage of when you
00:06:32.280
say gem install XYZ how do we know where
00:06:35.160
to go to get that gem from
00:06:37.259
is it from rubygems or from some other
00:06:39.360
gem hosting server
00:06:41.160
we're going to talk about this in more
00:06:42.840
detail later but suffice it to say
00:06:44.940
dependency confusion is bad and we want
00:06:47.280
to avoid this
00:06:49.259
these are just two small examples of
00:06:51.539
attacks in 2021
00:06:53.280
so we know for sure this is happening
00:06:55.199
why aren't all developers up in arms
00:06:57.660
doing something about this well
00:07:00.120
one reason may be that we want to leave
00:07:02.759
it to the experts right
00:07:04.620
in the past security is not one of the
00:07:06.720
areas that I have personally felt most
00:07:09.000
empowered to just go out and fix
00:07:11.340
it's really hard to know where to even
00:07:13.199
start looking web developers often rely
00:07:16.080
on security focused teams to be the
00:07:17.880
keepers of this knowledge
00:07:19.860
but the thing is we too can be Keepers
00:07:22.740
for some of this knowledge because the
00:07:25.139
most common attacks are actually quite
00:07:26.880
straightforward to execute and that's
00:07:29.460
why they're common because they're
00:07:30.840
relatively easy to pull off so yes that
00:07:33.479
means that most often there aren't any
00:07:35.580
super Haxorus tactics required and as
00:07:38.639
much as NCIS would love us to believe
00:07:40.500
that the way to to defend ourselves
00:07:42.360
against attacks is to have two people
00:07:44.639
fiercely keyboarding away on the same
00:07:46.680
keyboard unless
00:07:48.419
um that is sadly not the case otherwise
00:07:51.060
Ashley and I would single-handedly dare
00:07:53.699
say double-handedly uh a defend against
00:07:56.940
all attacks at Shopify attackers are
00:07:59.759
simply preying on the fact that
00:08:01.139
developers are imperfect human beings
00:08:03.599
and so they've set up booby trap to
00:08:05.639
catch us when we least expect it
00:08:07.680
for instance a common exploit is where
00:08:09.960
attackers take advantage of common
00:08:11.520
spelling mistakes or they take advantage
00:08:13.979
of a gem being hosted on a different
00:08:15.539
Source repository in the most classic
00:08:17.819
tactic in the hacker handbook
00:08:19.979
they take advantage of the fact that
00:08:21.360
some of us are lazy humans and as lazy
00:08:24.240
humans many of us are guilty of reusing
00:08:26.879
our passwords right
00:08:28.740
I'm sure people can relate here but the
00:08:31.379
good news is over the next 20 odd
00:08:34.140
minutes we'll unpack some types of
00:08:35.940
attacks how they happen and most
00:08:38.339
importantly you'll walk away with
00:08:39.659
strategies to defend against them
00:08:41.760
our goal is that you no longer need to
00:08:44.099
solely rely on security focused teams
00:08:46.320
and that you can be the first line of
00:08:48.420
defense
00:08:50.220
so as Betty said before we get into how
00:08:52.920
to mitigate attacks it's probably good
00:08:54.899
to understand what kinds of things can
00:08:56.700
happen when malicious code gets into our
00:08:58.500
gems I mean really what's the worst that
00:09:01.620
can happen what can attackers actually
00:09:04.140
do a fan of compromise or gems
00:09:06.720
well developers are secretive people
00:09:10.680
we live a double life part of it working
00:09:13.200
the open sharing a code with others and
00:09:15.420
the rest of the time we deal in secrets
00:09:18.360
that's why our servers are so big
00:09:19.980
they're full of secrets
00:09:22.040
I'll leave the jokes to Betty
00:09:25.320
you may have things like your AWS deploy
00:09:27.600
tokens rubygems access tokens and all
00:09:30.360
other kinds of Secrets stored somewhere
00:09:32.100
on your local machine or in your servers
00:09:34.620
the same machine and servers that you're
00:09:36.540
installing these random gems on
00:09:38.700
that's a lot of trust to put into gems
00:09:40.740
it's like inviting random people into
00:09:43.080
your home and just hoping that none of
00:09:44.580
them are thieves
00:09:46.200
a bad actor could put code into the gem
00:09:48.420
to steal these secrets and then send
00:09:50.459
them back to the attacker and this isn't
00:09:52.860
just hypothetical when the rest client
00:09:55.019
gym was compromised in 2019 this is one
00:09:57.660
of the things it did
00:09:59.399
this is actual code from the hack
00:10:02.040
it checked to see if we were on any
00:10:03.660
environment but localhost
00:10:05.640
if so we take all the environment
00:10:07.620
variables add them all to a hash
00:10:10.560
and then it does a post with that info
00:10:12.480
back to the Hacker's own website that
00:10:14.700
mirinaru endpoint there
00:10:18.120
and as you can see this is pretty simple
00:10:20.279
code the hacker is not some Genius Like
00:10:23.100
Betty said furiously keyboarding away
00:10:25.080
seeing who can type faster the good guy
00:10:27.120
or the bad guy
00:10:29.760
so what else can an attacker do
00:10:32.580
in addition to stealing secrets from you
00:10:34.560
another type of attack is ransomware
00:10:36.600
where instead of taking something from
00:10:38.640
you like your secret credentials
00:10:40.680
they're actually hiding your own data
00:10:42.660
from you
00:10:43.680
they will encrypt your data so that now
00:10:45.600
you can't access it or maybe they'll
00:10:47.760
lock you out of your system entirely
00:10:49.740
you can only gain access back to your
00:10:51.660
files or system once a ransom amount is
00:10:53.640
paid
00:10:55.079
there's an attack called reuke a few
00:10:56.940
years ago which was a widely publicized
00:10:58.800
ransomware attack between 2018 and 2020.
00:11:02.040
the FBI has estimated that it's caused
00:11:04.620
60 million in Damages worldwide
00:11:07.079
if an attack like that happens just like
00:11:09.240
700 times suddenly that's enough money
00:11:12.060
to acquire Twitter too soon so
00:11:16.459
ransomware is pretty detrimental
00:11:20.519
um similar to ransomware where the goal
00:11:22.500
is to get money out of you lots of
00:11:24.360
crypto based hacks exist as well
00:11:26.700
maybe they want to install a Bitcoin
00:11:28.620
miner on your machine and use your
00:11:30.540
server resources instead of paying for
00:11:32.220
their own or maybe they want to straight
00:11:34.140
up steal any crypto that you might have
00:11:37.140
one such example was in 2020 to ruby
00:11:40.200
gems called pretty color and Ruby
00:11:41.820
Bitcoin Ruby Bitcoin was pretty obvious
00:11:44.339
the name kind of gives it away
00:11:46.620
but pretty color was an exact replica of
00:11:48.839
the color as gem which lets you add some
00:11:51.060
color to your console so it actually did
00:11:53.940
something and if you were looking for a
00:11:55.740
gem to do this you could have very
00:11:57.300
easily downloaded the pretty color one
00:11:59.820
without realizing that it was malicious
00:12:02.579
with these attacks they took over your
00:12:04.320
clipboard so that if you ever copied a
00:12:06.480
cryptocurrency wallet address to your
00:12:08.100
clipboard it would replace it with the
00:12:09.959
Hacker's wallet address
00:12:11.279
so you ask someone to send you money and
00:12:13.260
you try to send them your your wallet
00:12:14.940
address and instead you're sending the
00:12:16.320
hackers wallet address so this is pretty
00:12:18.360
Niche and a very small potatoes attack
00:12:20.160
but it happens
00:12:23.160
what else can happen if you download a
00:12:24.839
bad gym
00:12:26.700
another type of attack is when an
00:12:28.380
attacker adds a back door this is kind
00:12:30.899
of a broad arbitrary term but it
00:12:32.579
generally means when a malicious script
00:12:34.079
is installed to give the hacker a
00:12:35.579
persistent Gateway into your system
00:12:38.579
this is from a hack of the strong
00:12:40.079
password gem in 2019 and it was also in
00:12:42.540
the rest client gem hack that we talked
00:12:44.220
about it's an attack method that's
00:12:46.200
pretty common
00:12:47.339
they've added a bunch of craft to try to
00:12:49.380
make it look confusing here and like
00:12:50.820
throw anyone off who might be looking at
00:12:52.380
the code but really they've just added
00:12:54.720
coach check to see if the rails
00:12:56.220
environment is production
00:12:57.959
and if so they eval open whatever is on
00:13:00.300
a pace bin
00:13:02.279
and so this is the back door because
00:13:03.779
they could use it to steal Secrets like
00:13:05.339
they did with rest quiet but they can
00:13:06.959
also use it to do whatever they want
00:13:08.399
they can keep changing the code on the
00:13:10.740
paste bin and whatever is there will run
00:13:13.380
so this is an Open Door to do whatever
00:13:15.120
they want to do
00:13:17.040
this is another example from an attack
00:13:18.720
on the bootstrap sash gem in 2019
00:13:21.600
a specific cookie sent by the client is
00:13:24.000
base64 decoded and then evaluated in
00:13:26.940
runtime to effectively allow remote code
00:13:29.040
execution
00:13:30.300
very similar to the page bin they can
00:13:32.279
put anything they want into the cookie
00:13:33.899
and it will run on your servers so this
00:13:36.000
is another example of a back door
00:13:38.160
as I mentioned back doors are a pretty
00:13:40.019
broad category because they can steal
00:13:41.700
credentials or install ransomware but
00:13:43.920
they can also do pretty much anything
00:13:45.120
else they want like install a crypto
00:13:46.680
Miner
00:13:47.820
the sky's the limit
00:13:50.279
so now that we've explored what's the
00:13:52.500
worst that can happen let's unpack how
00:13:54.360
malicious code actually gets into our
00:13:56.459
systems one way is through installing
00:13:59.100
the wrong gen now before I want to get
00:14:01.200
do that I'll get into that I want to do
00:14:03.360
a quick survey how many of you have ever
00:14:06.000
made a typo
00:14:08.820
I see some hands still aren't up yet and
00:14:11.399
so I'm assuming you've just abstained
00:14:13.440
from typing to keep your like typing
00:14:15.300
record pristine
00:14:16.760
I'm kidding
00:14:18.540
um so for our friends watching this at
00:14:20.459
home at railsconf home Edition and to
00:14:23.279
Sandy Metz uh basically the whole whole
00:14:26.339
room more or less had their hands up
00:14:28.139
which I guess isn't shocking but this
00:14:30.540
does bring us to one of the weaknesses
00:14:32.459
that attackers try to exploit they're
00:14:34.440
simply counting on our propensity to
00:14:36.060
mistype something
00:14:37.380
so instead of typing gem install RoboCop
00:14:40.260
you type gem install RoboCop or gem
00:14:43.680
install robocoop I'm hoping you get the
00:14:46.560
point here sneaky attackers squat on gem
00:14:48.899
names that are just a typo or two away
00:14:50.579
from legit gem names their goal is for
00:14:53.160
you to erroneously install their
00:14:54.720
malicious code in your applications and
00:14:56.639
have it run on your local machine or
00:14:58.079
production systems and at that point
00:15:00.180
they could be crypto mining like Ashley
00:15:02.459
had mentioned or it could be even worse
00:15:04.500
it's just Bad News Bears all around
00:15:08.160
now this type of attack is called typo
00:15:11.760
squatting and there's also a variation
00:15:13.680
or what I like to call a cousin of typo
00:15:15.899
squatting called combo squatting and an
00:15:18.899
example of that would be if you want to
00:15:20.820
include a gem called cool beans but you
00:15:24.120
didn't remember it properly and instead
00:15:26.100
install the fake gem that a bad actor
00:15:27.959
had been squatting on called beans cool
00:15:30.380
and at that point it'll definitely be
00:15:32.880
not so cool beans am I right
00:15:36.480
what's interesting about this class of
00:15:38.459
attacks though is that it's the most
00:15:40.260
common attack on software Supply chains
00:15:42.360
I'm going to let that sink in
00:15:44.279
the most common attack on software
00:15:46.380
Supply chains is basically an exploit on
00:15:48.660
our inability to type properly or
00:15:50.940
remember a sequence of words properly
00:15:53.880
pretty well right
00:15:55.620
another way a wrong gem gets installed
00:15:57.779
is when the source repository is not
00:15:59.760
explicitly stated so it's not clear
00:16:02.760
where the dependency should be
00:16:04.019
downloaded from
00:16:05.220
for instance let's say we have an
00:16:06.720
application with a bunch of gem
00:16:08.519
dependencies some are hosted privately
00:16:10.500
on our private server some are hosted
00:16:12.360
publicly on rubygems now honing in on
00:16:15.300
one of our private gems let's say we
00:16:17.100
have one called Soho house because y'all
00:16:19.380
know how private and exclusive so houses
00:16:21.360
right so an attacker learns of this
00:16:24.360
private dependency of ours and with a
00:16:27.060
quick search they learn that there's no
00:16:29.160
public version of this gem so they
00:16:31.380
proceed to create a public version of
00:16:32.880
Soho house and of course their version
00:16:35.459
is going to have all the naughty code in
00:16:37.139
it but the crucial part is what happens
00:16:39.540
when we run bundle install well blender
00:16:43.139
needs to decide where to download Soho
00:16:45.180
House from will it be from our private
00:16:47.820
server or public server which is
00:16:50.339
rubygems and the thing is it depends
00:16:53.579
bundling will decide which gen to
00:16:55.740
download based off of the gem version
00:16:58.259
whichever gem is the latest version it's
00:17:00.480
going to download from that Source
00:17:01.680
repository and this is the attack that
00:17:03.779
Ashley was alluding to earlier called
00:17:05.579
the dependency confusion attack very
00:17:08.220
aptly name if you ask me which gem
00:17:10.199
should be installed I don't know many
00:17:12.540
choices much confusion and on the note
00:17:15.720
of confusion
00:17:17.160
sometimes account ownership can be
00:17:19.140
confusing confusing too specifically the
00:17:21.780
confusion is who actually owns this
00:17:23.819
account I highlight this as a problem
00:17:25.799
because the second most common attack on
00:17:28.319
software Supply chains is a account
00:17:30.299
takeovers
00:17:31.679
this affects us because it's possible
00:17:33.480
for people who are the maintainers of
00:17:35.220
our dependencies to have their account
00:17:37.080
stolen the attacks that Ashley mentioned
00:17:39.720
earlier about the gems rest client
00:17:41.700
strong password and bootstrap SAS those
00:17:44.160
attacks all happen through account
00:17:45.840
takeovers
00:17:47.160
and once an account has been compromised
00:17:49.200
an attacker can perform all sorts of
00:17:51.539
privileged actions like yanking a gem or
00:17:54.179
pushing a new release of a gem with
00:17:55.980
malicious code injected it's like you
00:17:59.400
thought Aaron Patterson cut a new
00:18:01.620
version of rails but then it turned out
00:18:03.539
to be done
00:18:06.059
on Aaron Batterson
00:18:09.679
ah did you like that one I was really
00:18:11.640
proud when I came up with that you're
00:18:12.900
not gonna lie uh anyway another
00:18:15.120
Batterson way that uh malicious code can
00:18:18.120
get installed is if rubygems itself gets
00:18:20.700
pwned now this is way less common than
00:18:23.160
the previous attacks I mentioned but
00:18:25.320
it's the most dangerous for all our
00:18:27.120
rails applications
00:18:28.620
if ruby gems is compromised attackers
00:18:31.260
could just change the files on package
00:18:33.299
repository servers without maintainers
00:18:35.400
even knowing and this type of attack is
00:18:37.620
called repository compromise the
00:18:39.960
attackers basically have the ultimate
00:18:41.640
keys to the kingdom in fact within the
00:18:44.280
last few weeks rubygems just addressed a
00:18:46.679
critical vulnerability where any
00:18:48.660
rubygems user could actually yank and
00:18:50.940
then replace certain gems that fit a
00:18:52.799
particular criteria
00:18:54.360
fortunately white Source completed an
00:18:57.120
analysis last week and concluded that no
00:18:59.340
gems were compromised and so while this
00:19:02.100
attack is uncommon it is still a risk
00:19:04.980
that exists when we install and use gems
00:19:07.980
and so now that we're all sufficiently
00:19:10.620
paranoid I want to be clear here this is
00:19:13.620
not a PSA for y'all to stop using gems
00:19:16.860
right Ashley of course not
00:19:19.380
sure this all seems pretty scary but
00:19:22.080
gems are incredibly useful they're
00:19:24.480
convenient they help us iterate faster
00:19:26.340
we don't need to constantly reinvent the
00:19:28.320
wheel
00:19:29.160
besides rails is a gem and we're not
00:19:31.860
here at railsconf advocating for you to
00:19:33.720
stop using rails
00:19:35.100
so by all means stand on the shoulders
00:19:37.559
of giants it accelerates your
00:19:39.120
development and it accelerates
00:19:40.320
innovation
00:19:41.700
but as we've discussed here today there
00:19:43.980
are some very scary things that can
00:19:45.960
happen if we don't secure our gems
00:19:48.179
there are some inherent risks to using
00:19:50.220
gems
00:19:51.419
the goal is for us to understand the
00:19:53.400
risks and be diligent about mitigating
00:19:55.620
them
00:19:56.520
I mean driving can be risky right we
00:19:59.520
could all be approaching it like a game
00:20:01.080
of Grand Theft Auto
00:20:03.000
but we're not we wear our seat belt we
00:20:05.700
follow traffic laws we take safety
00:20:07.440
precautions to make our journey as safe
00:20:09.419
as possible
00:20:11.520
we need to do the same when taking
00:20:13.140
approach to supply chain security
00:20:16.020
to ensure our applications aren't
00:20:17.580
compromised we need to secure our gems
00:20:20.880
that we do that we do Ashley and so how
00:20:24.600
do we actually go about doing that
00:20:26.039
though now let's talk about
00:20:27.720
countermeasures for starters when I say
00:20:30.120
counter use the measure counter measure
00:20:32.940
counter measure oh my God that worked I
00:20:35.400
didn't think it would all are great
00:20:37.940
so the first countermeasure we get into
00:20:40.380
is a type of squatting and combo
00:20:42.780
squatting you're probably all thinking
00:20:44.460
what can save me from my erroneous typos
00:20:46.559
like a grammarly for gems
00:20:49.039
and my answer is yes kind of in the
00:20:52.860
sense that grammarly consists of a
00:20:54.299
defined list of words and grammar rules
00:20:56.460
you and your team can define a list of
00:20:58.740
vetted gems and that list can
00:21:00.780
essentially become your allow list okay
00:21:03.059
great so you've got an allow list but
00:21:04.919
how do you actually enforce it one
00:21:07.200
option is to create a GitHub action that
00:21:09.360
verifies whether the gems added in your
00:21:11.160
gem file are allowed or not
00:21:13.260
if you're curious about how to get
00:21:14.700
started with this we built a scrappy
00:21:17.160
prototype to help you get the ball
00:21:18.419
rolling but to be clear it is far from
00:21:21.120
feature complete but uh we hope it can
00:21:23.160
serve as inspiration
00:21:24.780
in addition to the allowless we'd also
00:21:26.760
recommend practicing the campsite rule
00:21:28.440
you know the one that says leave things
00:21:30.840
better than how you found it
00:21:32.940
so if you've come across a suspicious
00:21:35.460
package just report it
00:21:39.960
how about dependency confusion how do we
00:21:42.419
counteract that
00:21:44.100
last year under the bundler released a
00:21:46.860
fix for how it prioritizes sources this
00:21:50.220
was originally fixed in version 2210 but
00:21:53.280
we recommend being on at least two to
00:21:55.200
twenty two for all the best goodness
00:21:56.940
related to this issue
00:21:58.799
if you look at the bottom of your
00:22:00.360
gemfile.lock you'll see a bundled with
00:22:02.880
version if it's not at least two to
00:22:05.580
twenty two update your bundler in gem
00:22:08.159
install again
00:22:09.780
with this fix you can now specify a
00:22:12.419
block Source in your gem file
00:22:14.520
that'll enable bundler to prioritize
00:22:16.740
resolving the gems and its transitive
00:22:18.419
dependencies to the specific Source
00:22:20.520
listed within that block
00:22:22.559
and even if you don't explicitly list
00:22:24.600
the sources once you upgrade and then
00:22:26.880
run bundle install
00:22:28.380
your gemfile.lock will change from this
00:22:30.419
old format where the possible remotes
00:22:33.000
are just listed at the top and each time
00:22:34.799
you run bundle install any gem can
00:22:36.960
resolve to any remote depending on who
00:22:38.700
has the highest version
00:22:40.260
to this new format
00:22:42.059
where the remotes are listed one by one
00:22:44.580
so that for each gem it knows exactly
00:22:46.980
where to pull that Gem and its
00:22:48.419
dependencies from and it's consistent on
00:22:50.880
every bundle install so you never have
00:22:53.280
to worry about bundler being dependency
00:22:55.140
confused and just changing its mind
00:22:57.059
about where to pull the gem from because
00:22:58.679
someone out there in the world created a
00:23:00.360
higher version
00:23:01.740
as long as it was pulled from the right
00:23:03.419
place initially it'll continue pulling
00:23:05.760
from that place on every bundle install
00:23:08.640
so again check the bundler version that
00:23:10.620
was used to generate your gemfile.lock
00:23:12.419
if it's lower than 222 you should update
00:23:15.600
to get this protection
00:23:18.900
this was a problem we had to solve at
00:23:20.940
scale at Shopify since obviously we had
00:23:23.100
a lot of apps to update
00:23:24.780
one of our teammates actually wrote a
00:23:26.640
blog post for how this problem was
00:23:27.960
tackled if you're interested you can
00:23:29.700
check it out on the Shopify engineering
00:23:31.260
blog
00:23:34.140
now let's talk about how to counter raft
00:23:36.539
repository compromise as Betty mentioned
00:23:39.179
before someone hacking into rubygems
00:23:41.220
itself and changing the gems directly
00:23:43.020
there is not a super common scenario but
00:23:45.900
it is very destructive so luckily there
00:23:48.299
are some things that can be done to
00:23:49.559
mitigate that risk
00:23:51.480
gem signing is one mitigation technique
00:23:54.080
maintainers cryptographically sign their
00:23:56.340
gems and then when you install you can
00:23:58.440
use the high security or medium security
00:24:00.179
Flags to verify the signature to ensure
00:24:02.580
the gem has not been changed
00:24:04.740
this has been built into rubygems for
00:24:06.659
quite some time now but unfortunately
00:24:09.179
it's pretty cumbersome to use and so
00:24:11.400
effectively no one uses it
00:24:13.559
there's an RC open on rubygems right now
00:24:15.900
written by some other members of our
00:24:17.400
team to make signing easy and seamless
00:24:20.159
so if you're interested please check
00:24:22.260
that out and feel free to share your
00:24:24.240
opinion
00:24:25.980
and last but not least let's talk about
00:24:28.320
account takeovers the way I see it
00:24:30.480
there's two parts to this problem
00:24:32.460
firstly how can you assess your risk
00:24:34.620
exposure to account takeovers related to
00:24:36.539
your application's dependencies and
00:24:38.580
secondly how can you contribute to
00:24:40.919
account Integrity on improving intent
00:24:43.200
account Integrity on rubygems and the
00:24:45.240
software supply chain in general
00:24:46.980
so first off when talking about account
00:24:49.140
takeovers it behooves me to begin with
00:24:51.960
these four words enable
00:24:54.740
multi-factor Authentication
00:24:59.940
it's such an easy thing to do and it's
00:25:02.580
incredibly effective how effective well
00:25:05.520
Microsoft had released a report a few
00:25:07.740
years ago noting that MFA blocked 99.9
00:25:10.799
of automated account takeover attacks
00:25:13.320
like that level of protection is more
00:25:15.600
effective than the awesome n95 masks
00:25:18.299
we're all wearing right now
00:25:20.220
we might not have n99 but we've got MFA
00:25:23.220
99 right and who wouldn't want a piece
00:25:25.620
of that uh so let's get back to our two
00:25:28.919
key questions we'll start with gem
00:25:30.539
maintainers um is there a way for us to
00:25:32.640
determine whether their accounts or any
00:25:35.100
accounts for that matter have MS MFA
00:25:37.860
enabled if we head over to rubyjams.org
00:25:41.159
and check out our dear friend Aaron's
00:25:42.720
profile can any of you spot his MFA
00:25:44.820
status and where it's noted
00:25:48.960
got some shaking heads yeah me neither
00:25:51.960
um and as you can see or not see there's
00:25:54.600
no indication on his profile about his
00:25:56.400
MFA status and that's by Design while
00:25:59.760
you and I might have the best of
00:26:01.080
intentions uh publishing an account MFA
00:26:03.900
publicly is like letting a thief know
00:26:06.419
which home is easier to break into
00:26:09.120
I'm just gonna pause here for a moment
00:26:10.500
because it took me a long time to draw
00:26:12.059
these Doodles for like a two seconds
00:26:14.520
worth of speaking so there's little
00:26:17.100
piggies in a wall
00:26:20.400
anyway
00:26:22.100
to that point it was we don't want to
00:26:24.419
advertise which home or account is
00:26:26.220
easier to break into and that's why MFA
00:26:28.440
isn't on a profile page that said gems
00:26:31.260
can be configured to require all of its
00:26:33.299
maintainers to enable MFA before
00:26:35.100
performing privileged actions so if
00:26:37.260
you're a gym owner and you want to know
00:26:38.640
how to set that up you can check out the
00:26:40.380
docs here and for everyone else if you
00:26:43.020
want to verify whether this
00:26:44.220
configuration is actually set up so that
00:26:45.840
MFA is being enforced at least on a gem
00:26:47.940
level you can find that out by
00:26:49.620
navigating to a gems show page for
00:26:51.900
example here you'll notice that rails
00:26:53.880
requires its maintainers to have MFA
00:26:55.919
enabled
00:26:57.179
in addition to this the team Ashley and
00:26:59.400
I work on are very much invested in the
00:27:01.500
health of the ruby gems ecosystem
00:27:03.200
recently we've submitted an RFC and had
00:27:05.760
it accepted by the rubygems team to roll
00:27:08.039
out an MFA enforcement policy for the
00:27:10.260
top 100 most downloaded gems and this
00:27:12.900
policy is in line with what other
00:27:14.340
package managers are doing npm for
00:27:16.679
instance has already rolled out a
00:27:18.539
similar policy and in fact
00:27:20.940
GitHub has just recently announced that
00:27:23.580
their own policy to require users to
00:27:25.919
enable MFA by the end of 2023.
00:27:29.100
needless to say we're excited that in
00:27:31.919
the near future maintainers of some of
00:27:33.419
the most popular gems will be protected
00:27:35.520
from the Aaron battersons of the world
00:27:38.279
right don't want any more of those uh
00:27:41.400
and next how can you personally help
00:27:43.320
improve account Integrity on ruby gems
00:27:45.659
well if you have a ruby gems account
00:27:47.820
enable MFA because in the famous words
00:27:50.880
of my bestie Beyonce if you like it then
00:27:53.940
you better put a ring on it if you like
00:27:56.100
your account and you want to keep it you
00:27:58.020
better put a figurative ring on it right
00:28:01.020
if it's not obvious already we are big
00:28:03.600
on MFA like we are riding the MFA train
00:28:06.840
all the way home
00:28:11.880
so we'd extend the ask of enabling MFA
00:28:15.240
to be on just your ruby gems account to
00:28:17.640
put it simply just MFA all the things
00:28:21.020
npm GitHub Shopify if you're a merchant
00:28:24.299
uh if the service offers MFA enable it
00:28:28.140
and for those of you who end up enabling
00:28:30.000
MFA on your rubygems and or GitHub
00:28:32.340
account Ashley and I want to give you
00:28:34.320
this a dark ridiculously adorable
00:28:36.419
limited edition sticker to celebrate our
00:28:39.059
Collective love of MFA
00:28:41.400
and so yeah stickers
00:28:47.220
and so in closing if you loved our talk
00:28:50.940
or amateur Doodles tweet at us and I
00:28:53.340
don't know CC Sandy Mets if you want to
00:28:55.080
maybe uh I'm kidding I'm kidding we
00:28:57.720
honestly would love to hear from you
00:28:59.100
though like tell us if you've learned
00:29:00.360
something new or how much impact we've
00:29:02.580
had because you know we can also
00:29:04.020
shamelessly collect receipts for our
00:29:05.820
next impact review
00:29:07.640
okay took jokes over uh seriously thank
00:29:10.980
you so much uh MFA all the things come
00:29:13.620
find us for stickers and a final Wi-Fi
00:29:15.960
to you all
00:29:17.520
have a good afternoon everyone