Helvetic Ruby 2024
Lessons learned from running Rails apps on-premise

Summarized using AI

Lessons learned from running Rails apps on-premise

Andy Pfister • May 17, 2024 • Zürich, Switzerland

In this talk, Andy Pfister presents insights on managing Rails applications in on-premise environments, shaped by his company's collaboration with Doco Team, experts in digital archiving. The session emphasizes the complexities and lessons learned when running Rails apps on customer infrastructure, particularly for systems that are offline or require specific database and operating system configurations.

Key Points Discussed:
- Understanding the Client’s Needs: Doco Team faced challenges in running their custom software on-premise, prompting simplif to explore deeper solutions beyond simple deployment scripts.
- Database Support: The application needed compatibility with various database systems including MySQL, Microsoft SQL Server, and PostgreSQL. Pfister discusses using Rails' Active Record for database communication and highlights specific challenges, such as charset information in MySQL and column size discrepancies across databases.
- Operating System Compatibility: The presentation covers how Ruby code can largely be kept the same across Windows and Linux, with essential adaptations for Windows environments. Pfister explains the significance of Ruby Version Manager (RVM) for version control on Windows and the necessity of including build tools for gem installations.
- Continuous Integration (CI): CI was paramount in ensuring that multiple database systems and environments worked seamlessly. Pfister shares their experience using Azure Pipelines for testing and maintaining code integrity across platforms.
- Offline Installation: Addressing the requirement for offline software installations, the team opted for a comprehensive zip package that contains the Ruby interpreter, application code, dependencies, and a PowerShell installation script to automate setup.
- Update Management: Pfister emphasizes the complexities in managing updates for on-premise installations, often dictated by customer policies. The company settled on a yearly update schedule to meet client needs and maintain development efficiency.

Conclusions and Takeaways:
- The experience underlines the importance of adaptability in deployment strategies, especially when dealing with on-premise infrastructures.
- Establishing a regular update cycle is crucial as continuous deployment can be impractical in this context, requiring careful planning and customer communication.
- Continuous integration practices play a vital role in supporting diverse environments and ensuring application stability throughout the development and deployment processes.

In summary, Pfister's insights provide valuable knowledge for developers looking to manage Rails applications in complex on-premise settings.

Lessons learned from running Rails apps on-premise
Andy Pfister • May 17, 2024 • Zürich, Switzerland

37signals recently published a new software called Campfire. The main catch of it is "you only pay once and can host it yourself." For our customer, we have been shipping three Rails apps for years to their customer's on-premise environments. Since these environments tend to be very different from each other, the deployment process is optimized for Linux and Windows systems, PostgreSQL, and Microsoft SQL servers, as well as installation without any internet access. In this talk, I'd like to share how we approach this, lessons learned over the years, and how you might apply this to your apps.

Helvetic Ruby 2024

00:00:04.359 My name is Andy, and I want to talk about the lessons learned from running Rails applications on-premise.
00:00:06.359 Before we dive into the technical details, I want to establish some context for what we're discussing.
00:00:10.519 As I mentioned, I work for a company called simplif, and we build custom software for various businesses. In this case, our customer is Doco Team, who are experts in digital archiving.
00:00:20.640 They help clients sort out digital information and bring it into a digital format, safely storing it in a long-term archive. They have an in-house department that assists with this, and they also develop custom software to facilitate these processes. Our mission at simplif is to support them with that custom software.
00:00:37.399 You can also run that software on your own infrastructure if you choose, which I will refer to as on-premise throughout this talk.
00:00:45.239 In late 2020, they approached us and expressed that it was quite difficult to get the software running on their own-premise infrastructure. They asked if we could simplify that process.
00:00:58.800 Initially, I thought it might be as simple as writing a couple of deployment scripts, and then we would be done. However, after our discussions, I realized there were numerous optimizations we could apply in our development process.
00:01:10.560 Today, I will focus particularly on Rails applications that help their on-premise team streamline deployment and operation of these applications on their infrastructure. So, where do we want to run our Rails applications? First and foremost, it should work on both Windows and Linux servers. Additionally, we aimed to support three different database systems: MySQL, the infamous Microsoft SQL Server, and PostgreSQL.
00:01:49.920 As an added challenge, there are servers where the Doco Team software runs that are completely offline, with no internet access. We needed to figure out how to support that as well.
00:02:31.599 Let's start by discussing how we supported different database systems in our Rails applications. Looking back, it wasn't too much trouble to implement this. By using Rails, we benefited from Active Record, which takes care of most of the communication with the database through various adapters.
00:03:03.919 As you can see on screen, this part of our Gemfile illustrates how we structured support for each database system. We formed different groups corresponding to each adapter gem. When we set up the Rails application, for instance, for PostgreSQL, we specify that we only need the PostgreSQL gem and can clarify which other database gems we do not need.
00:03:37.440 Of course, there are differences between these database systems. One issue we ran into was that when using Rails with MySQL, it always tries to add charset information into your schema.rb file. This forced us to primarily develop against MySQL since PostgreSQL and Microsoft SQL Server do not require this and will ignore it if present.
00:04:02.200 At one point, we had an interesting incident involving a long text column in PostgreSQL being significantly smaller than that in MySQL, prompting us to chase down some bugs.
00:04:08.279 Moreover, we eventually introduced JSON columns, and if you work with JSON in PostgreSQL, you'll find it works really well.
00:04:12.640 Interestingly, you can also configure Microsoft SQL Server for type conversion between the text columns and the JSON data your application utilizes.
00:04:27.600 A common theme in this discussion is the importance of continuous integration for projects like this. We utilized Azure Pipelines because, spoiler alert, we also had to run Rails pipelines on Windows. In this regard, Microsoft is arguably the go-to company for Windows runners.
00:04:52.440 On the right side of the screen, you can see parts of our YAML definition. Here, we define separate jobs for each database system and version we want to support. If you noticed, MySQL is no longer displayed on the left side because we dropped support for it earlier this year to streamline things.
00:05:36.679 To recap the lessons learned thus far: organizing the different database adapters in distinct groups within the Gemfile worked really well, and continuous integration has helped us maintain confidence that all changes we make will indeed work.
00:05:58.600 Now that we've addressed different database systems, let's move on to supporting different operating systems. Regardless of whether you run Windows or Linux, Ruby code largely remains the same, with only minor adjustments necessary based on the operating system.
00:06:16.440 For instance, we needed to optimize our Ruby code for Windows specifically, but we found that 99% of the Ruby code remains unchanged.
00:06:39.600 We started looking into how to install Ruby on Windows since simplif mainly used Linux, making Windows a bit of a new territory for us. We discovered that we only really need one Ruby version running on a server, but we also learned about a tool called Ruby Version Manager (RVM) for Windows that allows for easy version switching.
00:07:00.520 You can obtain Ruby for Windows from rubyinstaller.org, which provides a minimal system that includes the necessary tools and libraries to build and run Windows applications.
00:07:13.480 It's worth mentioning that Ruby is written in C, and you can also write gems in C code. For example, while installing Puma, if it contains C code, Ruby will attempt to build native extensions, which is Ruby's way of compiling the C code.
00:07:34.360 On Linux servers, you typically have GCC installed. With the Ruby installer, you get the necessary build tools, which allow you to compile this C code.
00:08:04.680 One exception was when we installed the gem 'noiri,' which has a lot of C code, but they chose to pre-compile it for us. Thus, we didn't need to compile C code ourselves.
00:08:41.120 However, for gem maintainers, this means they have to release a different version for each platform (Windows, Linux, Mac) and CPU architecture they support. Additionally, if they want to support multiple Ruby versions, they need to release unique files for each version.
00:08:54.680 This burden of maintaining different pre-compiled versions can lead to delays in us being able to upgrade Ruby versions on Windows. Currently, we are still on Ruby 3.2, waiting for one gem to issue a new release before we can upgrade to Ruby 3.3.
00:09:33.879 Once again, continuous integration has proven invaluable in testing changes. In our CI setup, we now run tests against both PostgreSQL and Microsoft SQL Server, with adjustments to save time on Windows.
00:10:19.920 Due to limited pre-installed Ruby versions and databases, we often have to install Ruby ourselves during CI, adding about four to five minutes to our testing time. The current setup causes longer feedback loops on Windows, which usually ranges from 11 to 15 minutes, depending on the specific job configuration.
00:11:00.720 We learned that as we optimized our Ruby code for Windows, the process largely remained similar, save for some conditional statements. However, the existence of pre-compiled gems can indeed delay Ruby upgrades.
00:11:28.239 More importantly, continuous integration allows us to spot differences between operating systems, and we realized the Windows build could be slow.
00:12:11.219 We currently use Azure DevOps, but it's worth noting that performance can vary depending on the tools used. To mitigate slow builds, we keep a dedicated Windows machine handy so that we can address issues more quickly as they arise.
00:12:50.920 Shifting focus now, let's discuss offline installation. This means being able to install software without an internet connection, requiring us to provide a package with everything necessary to run the Rails application.
00:13:59.040 Unlike other languages where you compile code into a single executable file, Ruby requires you to ship the Ruby interpreter itself, along with all gems, application code, and Rails assets.
00:14:26.720 Docker was considered for building our packages with Rails applications, but we faced hurdles. In Windows, running Linux containers relies on WSL (Windows Subsystem for Linux), which, while useful for development, adds complexity in production.
00:14:52.240 On the other hand, Windows Docker containers can lead to large image sizes, and compatibility issues arise between different versions of Windows, making a straightforward build process complicated.
00:15:47.720 Instead of using Docker for Windows, we chose to ship a zip file containing the Ruby version, application code, and assets. This zip file must also remove any JavaScript runtime requirements—like needing to install Node.js.
00:16:34.600 Using a special bundle command, we can download all the required gems and store them in the vendor cache folder. Upon unzipping the package, the application will be ready to run without any additional dependencies.
00:17:09.840 To ensure smooth installations, we created a PowerShell script. This script extracts the zip contents, installs Ruby along with the required gems, sets up configurations, runs Rails migrations, and starts essential services without the need for an internet connection.
00:17:29.760 As this evolved, the PowerShell script initially meant solely for Rails now automates installations for various third-party dependencies too, becoming quite comprehensive. We have CI jobs that test the PowerShell scripts, including a smoke test to confirm that the application runs correctly post-installation.
00:18:25.239 One tool we explore is Ansible. However, Ansible does not run natively on Windows unless you leverage WSL, creating challenges during installations. We wanted to avoid introducing additional tooling that could complicate deployments.
00:19:34.920 Instead, we've opted for PowerShell since our team is proficient in it, allowing them to read and contribute to the scripts easily while keeping infrastructure simple.
00:20:10.719 Reflecting further on processes, we are aware of a project called Okan, which promises to turn Ruby projects into single executable files for Windows. Although I haven't tested it yet, it seems like a promising solution.
00:20:43.440 In summary, we have our zip file containing everything required to run the Rails app. The PowerShell script facilitates turning the zip into a runnable application. Continuous integration plays a vital role in ensuring our deployment scripts function correctly.
00:21:07.560 With that, we have addressed all the challenges posed by our initial discussions with the customer. As we move further into 2023, adaptability remains crucial.
00:21:47.679 In managing on-premise installations, releases become more complex. For feature requests, we often implement and test features before pushing to production.
00:22:27.919 With on-premise installations, we don't have access to customer systems, and often, policies exist that impact release schedules. One notable instance involved a requirement to announce downtime a week in advance due to policy limitations.
00:22:49.160 We therefore settled on a compromise for updates, releasing once a year to on-premise customers. Regular cloud updates are quicker, but on-premise releases can take time.
00:23:11.599 Regarding versioning, we emphasize semantic versioning where our major releases are intentional, signaling significant changes in customer-facing functionality.
00:23:46.840 From planning to execution, we announce changes ahead of time when we plan to deprecate features, allowing customers to adapt over time to the new updates. The following annual release is when we can effectively implement these changes.
00:24:35.200 In conclusion, continuous deployment for on-premise infrastructure may be impractical. Establishing a regular update cycle is essential to balance customer needs and development resources.
00:25:32.840 Thank you all very much for listening. I am grateful to Helvetic Ruby for inviting me. Here are my contact details if you are interested in reaching out. Enjoy the rest of the event.
Explore all talks recorded at Helvetic Ruby 2024