List

Continuous Deployments and Data Sovereignty: A Case Study

Continuous Deployments and Data Sovereignty: A Case Study

by Mike Calhoun

In this presentation titled "Continuous Deployments and Data Sovereignty: A Case Study," Mike Calhoun addresses the challenges of deploying a Rails application within the constraints of data sovereignty laws across different countries. The discussion begins with the landscape of data regulations, particularly focusing on how these laws affect the deployment of applications dealing with sensitive data, like health information. Calhoun identifies two critical frameworks: the Health Insurance Portability and Accountability Act (HIPAA) and data sovereignty laws, emphasizing the necessity of maintaining compliance while scaling globally.

Key points discussed in the video include:

  • Introduction to Data Sovereignty: Calhoun explains data sovereignty, which mandates that data collected is subject to the laws of the country where it is stored. This requirement complicates cloud computing solutions, where data often crosses international boundaries.

  • Continuous Deployment vs. Continuous Delivery: The speaker clarifies these concepts, noting that continuous deployment wants to automate the entire deployment process across various geographic regions, while continuous delivery focuses on keeping code ready for deployment.

  • Case Study of a Healthcare Startup: Calhoun shares the journey of a healthcare startup that initially designed its backend to comply with HIPAA, only to later discover that it needed to accommodate various international data regulations. This highlighted the importance of preparing for global clients early in application development.

  • Deployment Challenges: The speaker discusses different deployment strategies, including the problematic approach of creating separate production branches for different regions and presents a more manageable solution of regional deployments. The latter allows for a single code base with separate translation files, reducing logistical complications.

  • Utilization of AWS for Regional Deployments: The case study demonstrates how using AWS allowed the startup to deploy their application effectively across different regions, ensuring compliance with local data laws while maintaining a streamlined code management process.

  • Lessons Learned: Calhoun emphasizes the steep learning curve associated with deploying across multiple regions. He stresses the need for robust legal guidance and the importance of being aware of international data laws, the potential costs associated with global data hosting, and the necessity of ensuring user data security.

Ultimately, the talk serves as a comprehensive guide for developers and technology leaders on how to navigate the complexities of continuous deployments while adhering to stringent data laws in a global landscape. The core conclusion revolves around the fact that understanding and planning for these regulations from the outset can significantly ease the deployment process and avoid costly missteps.

RailsConf 2018: Continuous Deployments and Data Sovereignty: A Case Study by Mike Calhoun

In any production rails application’s simplest form, there is one version of the app deployed to a single host or cloud provider of your choice, but what if there were laws and regulations in place that required your application to be replicated and maintained within the geographical boundaries of other countries. This is a requirement of countries that have data sovereignty laws and a regular hurdle to overcome when dealing with sensitive data such as protected health information. This talk provides a case study of how we devised an automatic deployment strategy to deploy to multiple countries.

RailsConf 2018

00:00:10.490 all right hi this is more of a turnout than I was expecting for such a dry
00:00:15.809 topic so thank you guys for coming I also want to say thank you very much to
00:00:20.910 the conference organizers thank you for the amazing keynote that was really great I walked out of that feeling
00:00:26.640 really inspired so hopefully I'm not a letdown right after that but maybe you
00:00:32.340 guys will find this interesting we'll see so firstly hello my name is Mike
00:00:37.399 Calhoun I'm super happy to be here in Pittsburgh I learned this is the birthplace of the
00:00:42.629 Big Mac which I didn't know about also the Klondike bar from Pittsburgh that's cool I spend a lot of time in
00:00:50.160 Philadelphia so I finally got a chance to decide between Wawa and sheets for myself and I will not disclose my pick
00:00:57.120 I'm not picking any fights and tribalism here my wife's actually she has some family family from Johnstown so we often
00:01:03.870 get asked if unions have talked to Bob lately yeah okay that aside I'm from
00:01:09.869 Vancouver Washington which is probably known more for its more popular suburb
00:01:15.090 Portland Oregon I lived there with my wife we have two cats we have a new baby also the only thing that I called out
00:01:22.290 specifically and my speaker bio is our Corgi Ruby aptly named paws Faraz okay
00:01:29.220 you know I may know me from I am the current chief technology officer at life i/o though I am sad to be stepping down
00:01:36.180 from that role and starting at stitch fix next month I'm super excited for that though so I talked really fast and
00:01:42.479 if you can hear the tremble my voice right now I'm trying really hard to concentrate on that last night I did a
00:01:48.119 test run I was at like 35 minutes exactly I previously talked about failure and I went through those slides
00:01:54.329 so quickly I think it was maybe 10 minutes long everybody extra time to grab coffee I'll try to do something
00:02:00.899 similar and I'm excited to talk about something other than failure let's not talk about how bad I am at my job and is
00:02:08.819 a little bit of preamble to that I'm gonna reference a lot of companies and a lot of products I'm not specifically endorsing any of
00:02:16.290 them over their competitors I think the products we use we're great I think the products we didn't use are
00:02:22.650 also great a lot of them have people here sponsors whatnot I love everybody
00:02:27.990 everybody is super awesome this is a great industry we work in so so please don't take that as an indictment or a
00:02:33.959 lack of indictment or indictment or endorsement name and then I have a
00:02:39.569 podium so I'm going to say a few words on data this is not exactly connected to
00:02:47.130 the topic of my talk but I think it's really important that we keep talking about it and we generally work in
00:02:55.350 information in data and we're have a certain amount of trust that our users expect to do the right thing with that
00:03:02.310 data and these are becoming like huge topics and they are getting thrust into
00:03:08.670 a national conversation especially in the light of things that are happening with Facebook and with Cambridge
00:03:14.040 analytic and when other industries made rapid advances in their arenas
00:03:21.290 regulatory control and oversight emerged I look at the Industrial Revolution we
00:03:26.489 were forced to establish fair labor practices and those were overseen by the government nuclear science develops with
00:03:33.299 the use of atomic weapons and the use of nuclear energy and we established the Nuclear Regulatory Commission the EPA
00:03:41.430 emerges in response to abuses from industry and so maybe something on a kin
00:03:47.370 to a consumer data protection agency is what we need I'm not the person to litigate that I am NOT in politics I
00:03:54.060 just again someone gave me a microphone but that said we do have to consider
00:04:00.060 that not all societal and political problems have technical solutions but
00:04:07.049 until then it is up to us to be aware of the laws that attempt to govern our industry and broker trust with our users
00:04:13.019 that unseemliness aside I want to outline a few terms for this talk
00:04:20.100 specifically these were things that we came into contact with and I just think it would behoove us to establish a
00:04:26.190 shared vocabulary Larry so first is the Health Insurance Portability and Accountability Act and
00:04:32.730 this is the main culprit for why we initially kind of took some initial
00:04:38.580 steps we did that turned out to get us into trouble or at least force our hand a lot of what we wound up building so
00:04:46.710 this is enacted in 1996 and it's for the United States this a little
00:04:52.440 foreshadowing I guess and a HIPAA had two main purposes it was to provide continuous health insurance coverage for
00:04:58.770 workers so there was a logistical coverage component to it and then more specifically to us it was to reduce the
00:05:05.730 administrative burdens and cost of health care by standardizing the electronic transmission of
00:05:10.950 administrative and financial transactions I wish I could read that without my notes but it can't so it's
00:05:18.000 the first time really the government is taking steps to protect your electronic health data and this is really important
00:05:23.490 because before there hadn't been much in this arena we didn't have much in terms of rules about disclosures of breaches
00:05:29.220 what the practice say and there's still a little ambiguous there's parts at HIPAA that really they literally say a
00:05:35.280 consumer of this data will make a best effort well how do you define a best effort I don't know I didn't write it
00:05:42.360 down in a piece of paper and leave it at the coffee shop in 2010 they add breach
00:05:48.750 notification rules to extend to covered non HIPPA entities so now it's not just doctors offices and hospitals it's
00:05:54.510 anybody that's capturing this data if if we encounter a breach we're required to notify the Health and Human Services
00:06:01.590 office and in 2013 they add what's
00:06:06.960 called hi-tech or the health information and technology for economic and clinical Health expansion so they continue to
00:06:14.970 expand the rules require regulation to accommodate new and developing technologies and then in 2016 we see
00:06:22.440 additions and provisions for cloud cloud services as that is direction the
00:06:27.780 industry is gradually starting to take a little late to the game but required nonetheless I guess we can't expect
00:06:34.080 rules and regulations to keep pace with technology that's that's a dream next up
00:06:39.960 is data sovereignty and this is sometimes used interchangeably with data residency and
00:06:46.470 that's not it's similar but not exact sovereignty data sovereignty is the idea
00:06:53.250 that data is subject or data are data's plural our subject to the laws and governance
00:06:58.470 structures of the nation it's collected so you know in a world I could be a
00:07:03.780 German citizen but I live and see a doctor in the United States if my data is stored in the United States it's
00:07:10.080 subject to United States law it is not German law so in this case it would be
00:07:15.930 in that case I aligned it would be subject to HIPAA and the common criticism here is a data sovereignty
00:07:21.720 measures do tend to impede or have the potential to destroy processes in cloud
00:07:27.210 computing this was a big reason why they started to make those cloud computing provisions to loosen those restrictions
00:07:34.790 data residency is law as a law there basically requires that say that's right
00:07:41.370 it if you're live in a country that has a data residency law your data must be stored and processed within the
00:07:47.880 geographic of that country oh okay oh
00:07:58.610 hey cool all right let's try this you
00:08:04.680 guys hear me okay still cool this is a good time killer okay so yeah so process
00:08:11.550 and are stored inside the country Australia is a great example of this Alistair is a great example of this they
00:08:19.040 if you're capturing any kind of health data there AWS Sydney foreshadowing may be a little
00:08:25.290 bit spoiler alerts is a great solution to those equations that problem so let's
00:08:31.140 talk about continuous deployment and I may have cheated fully admit to that continuous deployment versus continuous
00:08:37.110 delivery continuous delivery means your code is in a constant state of being ready to be deployed sometimes you can't
00:08:44.099 just automatically trigger that we had client related concerns they wanted to verify some things are staging servers
00:08:50.610 Oh oh go through a double speed now okay
00:09:02.540 for production this was continuous delivery I use an example here that I'm going to give that is more more akin to
00:09:10.830 continuous deployment but I mean there's it's it's like a half a step short of
00:09:15.930 continuous delivery I like that quote that I kind of dug up there from Twitter okay
00:09:21.870 so let's actually look at the case study aspect of this so the problem is I don't
00:09:27.150 know if it's the problem but we're gonna be a health care startup alright this is exciting everybody's fists are in the air we are I use a lot of sir just look
00:09:33.810 for images on Google and put them in my presentation and so great we're gonna be
00:09:39.210 a healthcare startup we're gonna capture sensitive user information we're gonna expect that our users trust us with this
00:09:45.170 and let's see how it goes but more specifically we're going to be a SAS
00:09:50.580 startup so we're gonna put this application and out there in the world we're good probably gonna use a cloud
00:09:55.680 provider and it's going to it's going to
00:10:00.900 have a multi-tenancy single platform everybody will will log into it which
00:10:06.330 brings up this occasional myth of convenience is that we have great tools we just saw an amazing keynote about
00:10:12.600 this like we have some great tools about reducing barriers to entry I don't need to know Devon I can deploy this to
00:10:18.030 Heroku I don't really necessarily know much sequel I have active record for that I don't have to build my own version
00:10:25.170 control of github back in the day we all had those cartoon turtles with SVN and
00:10:30.720 there's a whole world of CI apps out there to do this and this encompasses a majority of what we can reasonably
00:10:37.260 expect to need but then sometimes you wind up in these situations we're back
00:10:42.900 to our startup you've decided to be a SAS company you're going to collect sensitive user information we're going
00:10:49.890 to assume all of our clients are in the United States whoops and then let's have our first
00:10:55.140 client not be in the United States and let's look at their laws and let's evaluate our infrastructure and you find
00:11:02.220 one major conclusion is you've made a huge mistake all along you made these assumptions that are just
00:11:08.190 completely thrown out the window so you have to take a look at your
00:11:13.560 international logistics and this is the first time we'd considered requirements beyond HIPAA and this is kind of weird
00:11:20.100 because I said Australia Canada has their own set of rules the United States has less restrictive rules in some South
00:11:27.149 American countries you see these rules written into their constitutions the UK
00:11:32.600 had a set of rules and then gave them up to join the EU and then something else
00:11:38.850 happened where they're developing their own set of rules again so we took a look
00:11:45.240 at these potential global entities because we knew this was going to be a problem we work with a group like think of I'm from the organ areas of Roland
00:11:52.649 area so think of Nike they have a headquarters and Beaverton Oregon so you're gonna have a fair amount of users there but they also are global you're
00:11:59.550 gonna have a headquarters in Africa and Australia and Asia and South America
00:12:04.560 and these are all gonna have their own set of rules so you have to take stock of what works and that's when we came
00:12:10.529 across AWS and you can see they have the
00:12:16.050 United States more than covered there's that one up there in Canada all over the UK for us the big mover was the data
00:12:22.439 center down there in Sydney and we realized that we weren't replacing our
00:12:27.630 Heroku set up we just wanted to augment it we needed to accommodate these rules and so we knew we had a place where this
00:12:36.930 is possible now we had AWS we knew we had our American server the question was how were we going to integrate this with
00:12:42.930 our kind of tool chain for deployment so we came to option one and maybe the
00:12:48.930 images on these will give away how we went with this but so initially we had this discussion like what if we created
00:12:54.959 a new branch for every region so you have production USA
00:13:02.570 and this seemed really this is like the most obvious like let's just come to
00:13:08.000 this quick conclusion here's what we do we offer some as a company we offer some basic white labeling aspects for our
00:13:14.300 clients so that seemed like it would be a lot easier to accommodate those you're
00:13:19.310 gonna handle region specific requests easier if you have translations for example I can just swap out the English and put in whatever else I want and
00:13:27.980 there's a low initial time cost we're just kind of creating branches like we've all created a new git branch it's
00:13:34.730 pretty easy but the disadvantage is that this becomes a complete logistical nightmare this is kind of what that
00:13:41.060 image I have there was getting at is imagine your code gets approved on
00:13:46.310 staging everything's looking good and you're not just merging into production now you're merging it into five
00:13:51.650 different production branches and keeping all those squared away and then God forbid you wind up in a scenario
00:13:57.080 maybe one of those production branches doesn't get the same code as another one
00:14:02.960 maybe that's a translation case and like that it's just not it's not sustainable at least in a way that's timely and
00:14:08.720 efficient and then we looked at option two which was what we called regional deployments and this got is to the point
00:14:17.390 where we maintained this one code base it meant that all of the translation files would have to sit in the same repo
00:14:23.740 it continues the notion of the single platform multi-tenancy so let's do an
00:16:09.230 example I can't show the app we used our we we build excuse me but I made this little demo app and I
00:16:15.740 hope this comes through can everybody see that okay kind of walk through it
00:16:21.770 it's really easy it's really light on the top you have a test suite there's one spec one feature spec it
00:16:28.760 just says it expects the goggles to equal to nothing this is the stroke will come together and it's just gonna render
00:16:35.000 what's on the bottom it's just a hello world page that shows an image and then
00:16:40.190 I have a small test suite I did I had this once in a video I had tried this once in a life coach session both of
00:16:45.980 those didn't go well so we're going with screenshots so you see the test Suites passing and you can see it's on the
00:16:51.800 local computer localhost is kind of at the top and that's all it does very
00:16:56.839 small so this is all with an intention to get this to move a little quickly
00:17:02.350 so here's the summer for dashboard and there's a few things to call out on this
00:17:07.549 is at the top I have my master branch and that's passed and for all intents
00:17:13.280 and purposes I'm using this as my production branch below that there's this little section called servers and
00:17:20.689 we have our United States Roku server where we're deploying this to this kind of mirrors what we had for our
00:17:27.319 infrastructure at the start of this whole scenario so then we add our new
00:17:34.040 application in AWS this is your dashboard for elastic Beanstalk and you
00:17:40.910 can see in the top corner there we know we're in Oregon that's the region we're going to deploy this one for some reason
00:17:46.160 in this scenario Oregon in the United States are two different groups that have their own laws which sounds crazy
00:17:52.280 but actually Canada passes rules governing health data by province so it's not that crazy I have a container a
00:18:01.299 little demo environment rails Conference 2018 app that seemed is much funnier
00:18:08.780 when I wrote it and apparently this thing's my mom it's my full name
00:18:16.909 you can't see the first arrow the right
00:18:34.059 points to a toolbar and you can see the region where you're in so I would say Oregon in this case and this will change
00:18:40.070 to say Sydney to change where it's a WS is a way of letting you know you're into
00:18:45.409 the correct server let's see I'm gonna look over here more often now so I know
00:18:50.690 you guys okay so back to servers we're gonna add this new one that we just made
00:18:56.019 and now nope so on this next one I put
00:19:02.509 three screens here originally these were separate and now it's a little slap - but they're all kind of ideologically
00:19:08.539 linked the first one on the far left you have set up deployment for rails 2008 so
00:19:14.090 this is the app that we've made and they offer some out-of-the-box solutions that list Scrolls down for a while those are
00:19:20.389 the first four we needed elastic Beanstalk so we click that and it takes us to the one right shoes automatic if
00:19:30.289 you're gonna do continuous delivery choose manual and you retain some control over that for this purpose we'll
00:19:36.559 go with automatic and then the bottom right oh yeah it just asks what branch
00:19:43.639 you want to deploy we pick master you can use whatever branch you know your mileage may vary once you go through
00:19:52.399 that I won't give you my AWS credentials but I'm gonna call it tension to the
00:19:57.979 region so you get the list of regions that is offered associated with this account I would select Oregon in this
00:20:04.849 case for that little piece that you couldn't see but it was at the top of the screen I'm promising and it
00:20:10.849 automatically pulls in the name all the known application names and all the known environment names so I choose my
00:20:16.909 demo app I choose my demo railsconf 2018 application that's three buckets kind of
00:20:24.979 move it gives you an option to pick a new one or create one if you want to it's just kind of where it's going to dump all of your code to before it
00:20:30.470 deploy to its server oh I highlighted all of these and forgot to fast-forward alright
00:20:36.520 so that's it you give your server a name to make it meaningful for easier navigation and because you're a good a
00:20:43.340 good citizen developer and a fledgling a good fledgling DevOps person and it
00:20:49.280 takes you right to this this is awaiting your first deploy you click deploy my commit message was because you click
00:20:55.910 deploy because and you see that your application is now deploying going back
00:21:02.270 to your dashboard you have production Oregon in a state of being deployed your
00:21:08.120 tests are all still passing so this should be fine eventually your code shows up and you can navigate it to it
00:21:13.190 through whatever link you have we expanded this a little bit for this demo so now I have four regions we've added I
00:21:19.610 have Canada and that's I think you know if there's a Canadian national here I
00:21:25.790 pretty sure it's in Toronto or if somebody knows but I'm not positive or outside of Toronto still got Oregon
00:21:30.920 we've got Sydney in here now and I've still got me and it's Heroku app now
00:21:35.930 this is going to be uncomfortable and I don't know if we'll see y'all but let's see what happens so I put together a
00:21:41.420 video to kind of show all of this in action so is it playing ok cool it doesn't play
00:21:47.750 on my screen so I'm gonna try to navigate off of this thing this'll be great so I changed I made a change I'm going to commit this now the dumbest
00:21:55.670 commit yep there we go I forgot what the change was I just changed the title of
00:22:01.010 the page so this pushes up to github my master branch picks it up I playing this
00:22:06.920 at double speed so suddenly it's going to jump on me and I'll get really nervous not knowing how to navigate it
00:22:12.250 so the master branch is building it only has to pass that one test which
00:22:19.010 shouldn't take too long this is a free account I didn't pay extra money for the purpose of the demo not thinking I would have to
00:22:25.670 navigate it like or narrate it like this
00:22:33.240 give it one per second there goes all right so that passes and that kicks off all of these builds at once the first to
00:22:40.230 come up it's automatically now deploying to Canada and Sydney and those take
00:22:47.640 their own respective minutes or two so this is right I mean this has run the the test suite for me in the case of
00:22:54.720 these AWS builds it's taking the github repository zipping that up sending it off to that s3 bucket and then unpacking
00:23:02.669 that onto the server and in test runs I
00:23:09.240 would finish that sentence and this would have been done but I'm speaking a little fast there we go all right Sydney deployed first there's a winner Sydney
00:23:15.659 deploys Oregon starts building Canada finishes Heroku starts building
00:23:22.980 I have tabs open that you can't see but I'll click into them so there that one I don't know which one I clicked into I
00:23:28.289 can't see it at this stage I think this
00:23:35.070 is Sydney so we see it's deployed to Sydney I'm going way too fast I'm sorry I can come back on this or pause it
00:23:41.730 there's Heroku briefly I click into the Roku app so you know the Heroku United States one deployed there it is and so
00:23:50.370 now we're just waiting on Oregon so we saw we saw I don't know what Nora what we saw Canada we saw Sydney we saw
00:23:55.500 United States default Heroku is in Northern Virginia I think and Oregon's gonna be the last one to cross the
00:24:01.200 finish line and it's done I click over to it yep there it is I think that's the
00:24:06.990 end of that video yeah okay so that was kind of so that at that point that was
00:24:12.330 basically the exact same infrastructure we built out for ourselves every time we pushed a master it would automatically trigger these deploys
00:24:17.580 would go throughout the globe a really streamline a process that we were having
00:24:22.740 panic attacks about and so we had some findings from this because this is a
00:24:28.049 case study our pros were this was very effective and very scalable you saw a
00:24:33.330 lightweight demo it's even more effective absent the nervous narration if you all just like while we push this
00:24:39.870 up and it's done and we all got to sleep at night easily but there was a steep learning curve and getting there
00:24:47.100 everybody is super awesome I love all of these products AWS elastic Beanstalk it's it's setup was a bit it was a bit
00:24:54.809 more complex than Heroku and then getting all of this to work in harmony was even a little bit trickier but once
00:25:01.260 you've got that learning curve it's pretty it's pretty easy to manage managing all these server configurations
00:25:06.900 themselves could be tricky you have your environment Muriel's you need a kind of a more scalable solution for replicating
00:25:12.929 your application harness and that initial loss of functionality going back
00:25:17.940 to the social features that we had lost so we were thinking about next steps and
00:25:24.500 this feels a little bit more weird to talk about after the keynote but like it
00:25:30.360 seems like there could be a case here to be made for decomposition of the application this is a monolith we were
00:25:35.610 deploying and kind of the vector we were narrowing in on is what if we took our
00:25:41.179 identifying information or PII protected identifying information and pH I protected health information and what if
00:25:47.789 we built like a data service to put those in those regions and then sent off to a social server wherever only user ID
00:25:55.380 so as users requested friendships you're just capturing those IDs you can encrypt those a yes to 56 and in theory again
00:26:04.679 I'm also not a lawyer but in theory this would accommodate those rules because you're not actually sending this identifying information out to have any
00:26:12.000 kind of backtracked attack on that you'd have to breach the server with the
00:26:18.150 social data with user ID five six seven eight is friends with eight five three
00:26:23.970 or nine and then know which regions those users were in and then breached
00:26:29.580 those databases as well ideally you'd be able to detect when someone's orchestrating that sophisticated it I
00:26:35.549 mean ideally attacks happen all the time that you're kind of dumbfounded by then
00:26:41.940 you have to consider beyond that the operational cost like this is not cheap yeah you went from supporting one server
00:26:49.440 one for one lowly Heroku server in Northern Virginia to all the servers across the globe so it's it's like in
00:26:56.130 those regions their prices expand depending on the remoteness of the region cost of electricity there
00:27:01.889 so you need to build that cost in if you suddenly find yourself dealing with and I mean let's not kid ourselves anybody
00:27:08.759 like a nun part with a Nike who has this many global offices probably has deeper pockets so you can build price that into
00:27:15.389 your contracts but if you're doing if you're operating it just out of the gate
00:27:20.850 I would not advise doing this as like step one with your startup investment
00:27:25.919 capital but that said there are some recommendations I can make on this which
00:27:31.409 is very hard about your audience before
00:27:37.559 building something I granted we would never have expected that r4 we expected
00:27:44.940 as like our first clients were going to be in the United States next thing I know I found myself flying to Australia
00:27:51.330 and flying to the United Arab Emirates and learning their laws and it was a bit
00:27:56.909 jarring to think that that you know if I had even just considered a global infrastructure out of the gate not to
00:28:02.309 say I would have built it but we could have made provisions to accommodate that early on or at least had a more robust plan of attack perhaps it was like
00:28:09.539 just-in-time research that we did for it are you storing sensitive data know that that
00:28:16.919 data is subject to laws and those laws probably not going to change at the same
00:28:23.549 pace as your application but are going to change and need to be aware of those
00:28:29.899 need to be aware of how they may affect your compliance or if compliance is even
00:28:35.759 a requirement and I mean at the end of the day is just because it's there doesn't mean you need it this is kind of
00:28:43.440 like going back to the beginning of this like we could have had those considerations we could have we talked initially about building this
00:28:48.509 application like oh yeah let's do micro services out of the gate we didn't because we wanted to move quicker and
00:28:56.750 building a model that was more native to all of us now we know that maybe down the road we probably would have changed
00:29:02.610 that but you know that's all I have for everybody um thank you again my name is
00:29:08.340 Mike Calhoun you find me on Twitter or github or anything that's social media that I might signed up for it usually cool
00:29:14.580 Michael one that seems to always be available
00:29:34.100 at some point time we made an accommodation to say that what was
00:29:39.980 working in our default heroku production would probably be working across the globe and this is more or less true the
00:29:48.659 biggest cue a burden in that case is translations Australian English is a different translation from Spanish
00:29:54.360 different translation from American English so but yes it is a tough process test very robust test suite and when we
00:30:04.080 have to deploy to five servers we kind of give her the heads up that's about it yeah I'm gonna try to restate that the
00:30:11.340 question is do we ever have a feature that we want to deploy to the United States or just any region but didn't
00:30:17.549 want available in other regions and yes that has happened a couple of times some
00:30:24.419 cases have required just creative database tricks like kind of having feature flags there's a gem that we've
00:30:30.419 had a lot of success with called flipper it was really useful for that and that kind of allowed allowed us to enable and
00:30:35.820 disable our database models based this predicated on this notion of you have a parent organization the parent
00:30:41.340 organization has many companies and companies have many branches so I don't know what a paranoid like II could have
00:30:46.350 multiple branches throughout the world and maybe they're an apparent organization shoe companies and so we
00:30:52.350 have features that we only want shoe companies to see we could enable that to just everybody through flipper yeah
00:31:01.080 yeah I can speak a little bit to that the question is could we speak to
00:31:06.710 basically how we had team coordination to determine what the requirements were and how we would be in accordance with
00:31:12.900 up with them Widom so the way you phrased that question was great because
00:31:19.140 it implied that our team is a lot bigger than he is in most cases our side of it
00:31:25.169 was myself and one or two engineers usually discussing we retain a legal
00:31:31.429 counsel and usually if we're going to a new region we'll try to find some legal counsel there to make sure we're
00:31:38.030 accommodating them but then on the other side of the table enterprise-level clients that are operating at this scale
00:31:43.179 they have their own counsel and like security checks they want to verify so you work very closely with them and you
00:31:51.950 push back where things are unreasonable and you identify kind of what their requirements are Australia was great our
00:31:58.190 contact there was a I'll protect his identity buddies the CIO named Tim and
00:32:04.059 he just worked with us very carefully about what the Australian law was and didn't kind of have expectations that we
00:32:09.710 went into it knowing what it was but it's asking questions when you have them and making sure every step of the way
00:32:16.360 here's what we see is appropriate from our side here's what we know we can implement here's where we have to reduce
00:32:21.650 scope because it's not gonna be in accordance and then verifying with them like please this is what we're seeing have your team check it as well because
00:32:28.990 most of these laws most of these laws it's like whether or not the the breach
00:32:35.210 or anything like that is one person's fault everybody will take blame if we had a breach for a major client bill I
00:32:42.620 mean sure we'll get scapegoated be read probably rightfully so in that story because for the fault of it but they
00:32:48.500 will take that heat as well so there is this notion that no matter what you're in this together and they want to make sure you're not a vulnerability and
00:32:55.330 working to make sure they're satisfied is the best you can do if if they
00:33:01.940 couldn't be exposed to the internet if it was old because they dated was solely on an intranet is that yeah that's we
00:33:08.450 would regrettably turn down some money and wish them well oh right thank you