Security Services Delivery Kit

GitLab Professional Services offers two engagement types related to application security.

From the GitLab website:

The GitLab platform provides an opportunity to transform the way your business automates security processes. However, transitioning and transforming from your current security process to an automated, manageable, and consistent workflow in your GitLab deployment requires customized configuration from security professsionals. Engage with GitLab experts to solve specific security challenges and unlock all of the benefits of the GitLab Platform.

DevSecOps Workshop

The DevSecOps Workshop is a fixed-price advisory engagement for customers new to GitLab Ultimate that demonstrates core security features through a pre-built demo project and provides roadmapping sessions to help organizations adopt and integrate GitLab's security capabilities into their development workflows. It is a purely advisory engagement where delivery team members do not have access to the client's environments and code. For hands-on engineering, customers should purchase the DevSecOps Transformation service (below).

From the GitLab website:

The DevSecOps workshop offering allows for GitLab engineers to show off GitLab Security features and demonstrate core security policy automation benefits as well as provide strategic guidance to create a roadmap to kick start your security journey with GitLab.

Fact sheet (Google Drive)

Security Consulting / DevSecOps Transformation

The DevSecOps Transformation directly supports customers along their adoption journey with Application Security Testing, Security Risk Management, and Software Supply Chain Security, to reduce security and compliance risk across the organization. It is a hands-on engineering engagement where delivery team members will have access to the client's environments and code.

From the GitLab website:

For customers who want to transform the way they secure software using GitLab CI, we provide a skilled GitLab engineer to help define and implement security policies, triage vulnerabilities, and audit your GitLab deployment in an automated, scalable way.

Fact sheet (Google Drive)

DevSecOps Workshop

This fixed price advisory service is recommended for customers new to GitLab Ultimate, including upgrades from Premium, who are looking to understand core DevSecOps principles and how GitLab can instill them into their workflows to get the most out of GitLab Ultimate features and benefits. The workshop is particularly valuable for organizations that are new to application security practices or are looking to enhance their existing security workflows.

It includes a brief introduction to GitLab Security features, including the shift-left mindset concept (MR security approvals on dev branch), security dashboard, vulnerability management, in-house GitLab security scan capabilities, and scan execution and merge request approval policies.

The roadmapping sessions advise customers on how to plan for and adopt GitLab's application security features, helping clients to:

  • Define security policies including:
    • Types of security scans required for different types of projects.
    • When to run each scan.
  • Determine which projects should be scanned, and how to automate and enforce this.
  • Understand what the GitLab Security Dashboard is and how to interact with its data.
  • Define a vulnerability remediation strategy.
  • Educate both developers and security team members on the new GitLab compliance process.

Sales and pre-launch considerations

  • This service offering does not involve touching customer code at any point. We use a pre-built project and walk through security features in that project, dive deep into each of the security topics mentioned above, and then deploy the project to the customer environment as an example.
  • This is a fixed price SKU service (not customized or variable), the scope and price estimate of this service is small.
  • For any customer looking for a more advanced security focused engagement, they should look at our DevSecOps App Transformation service, whose scope is larger because it is a custom engagement and involves working on customer projects.

For fixed-price engagements, we use a service description document instead of the traditional SOW. This still has a list of all of the activities that will be covered in the engagement, but doesn't tie specific hours to specific tasks. You can find the DevSecOps Workshop Service Description here..

Delivery requires access to the GitLab demo environment and maintainer access to the DevSecOps Workshop Compliance Group. To receive access, generate credentials by following the demo system documentation. Join the #demo-systems slack channel with questions. If you are a partner please work with your Partner Delivery Manager or Services Practice team to import the project from the Demo Environment into LearnLabs adding appropriate permissions.

Professional Services team, please use the demo architect portal to request a workshop LearnLabs group to be created. First, use GitLab Admin login to sign in and follow the instructions to do a content/lab request and complete the form with the necessary information. Afterwards, you will be notified when the issue to create your group is created and then the group link when it is actually created. Please export/import the DevSecOps workshop project demos you'd like to use into the LearnLabs group and provide the partner with access permissions.

Target Audience

The target customer for this service offering is for either customers moving from Premium to Ultimate or a new Ultimate customer or an Ultimate seat expansion. The main service offering is meant for a PSE to show GitLab's security product capabilities via a quick demo, and then facilitate deep-dive sessions, Q&A, roadmapping, and/or strategic discussions about GitLab Security and how it can support and transform the customer's current security process and vision.

Unlike GitLab's education services, there is no limit to how many people from the customer organization can participate in this workshop, making it an excellent opportunity for cross-team collaboration.

Workshop Structure

The DevSecOps Workshop engagement typically breaks down into three main components:

  1. Discovery/Inquiry (approximately 2 hours of meetings) - An open discussion to understand the customer's current security practices, tools, and challenges. This helps tailor the subsequent parts of the engagement to the customer's specific needs.

  2. Demonstration (approximately 2 hours of meetings) - A hands-on showcase of GitLab's security capabilities and vulnerability management features using a pre-built project. This gives customers practical experience with the tools.

  3. Roadmap Discussion - A strategic session to map the customer's specific challenges to GitLab Ultimate features, with a focus on shifting security left in the development process.

Demo Workshop Overview

This project lives on our customer success cloud environment.

This project is a pre-built GitLab project used to deliver the DevSecOps Workshop service. This particular example was built using the Node Express GitLab project template. There is no "application" to showcase - it's not needed to show off security features. Note that the sample project currently focuses on SAST and Secret Detection, with limited DAST capabilities, and does not include dependency scanning or Infrastructure as Code (IaC) scanning.

This project will be showcased to customers to teach them the following core GitLab Ultimate concepts and benefits:

  1. Shift-left mindset in MR workflow as well as DevSecOps best practices.
  2. Inclusion of GitLab Secret Detection and SAST templates as well as policies around using them.
  3. Creation of a scan execution policy and merge request approval policy.
  4. Built-in GitLab security dashboard and vulnerability remediation.

Delivery Steps

In case you prefer folloiwng along to a video, here's a recording of how to run through the project.

1. Access the DevSecOps Workshop Compliance Group in our sandbox environment and request maintainer access, and that should have everything you need for delivery. See prerequisites above for more details details.

2. Create a new project named "DevSecOps Workshop - JohnDoe", replacing "JohnDoe" with your name, in the DevSecOps Workshop Compliance Group based on the DevSecOps Workshop - Original project by either:

  • Creating an export (Settings-->General-->Advanced-->Generate & Download Export), hitting the "New Project" button, and attaching the GitLab export file (*tar.gz)
  • Cloning the DevSecOps Workshop - Original project down to your local machine, create a new blank project in the GitLab UI within the DevSecOps Workshop Compliance Group, and pushing it to "DevSecOps Workshop - JohnDoe", replacing "JohnDoe" with your name.

3. Showcase the .gitlab-ci.yml file on the default branch and briefly walk through the stages that exist for this NodeJs Express template project.

4. Create an MR approval rule by navigating to Settings-->Merge Requests-->Merge Request Approvals. Under the Approval Rules section, not Security Approvals, simply update the existing approval rule for "all eligible users" with a target of all branches so that the approvals required is 1, instead of 0. Additionally, in "Approval Settings" below, uncheck the box labeled "Prevent approval by author." While this is not considered best practice -- typically, it's recommended to require an approval from a user other than the author to ensure an independent review -- for the purpose fo this demo, it will simplify the process by allowing you to approve your own MR without needing to rely on someone else. Be sure to explain to the customer that, although this setup deviates from best practice, it is intended to streamline the demonstration process. Also, Explain how this MR workflow setup enables the shift-left mindset and its benefits (you will show this later in a merge request).

5. Create a security policy by navigating to Secure-->Policies. Select "New Policy" and then "Merge Request Approval Policy". Name the policy "SAST Policy Approval Rule" and optionally provide a description. In the Rules section, select the dropdown options set up the policy to follow: "When Security Scan SAST runs against the default branch with No exceptions and find(s) Any vulnerabilities that match all of the following criteria: severity is high or critical and status is new. Under the Actions tab, "Require 1 approval from: Individual users" and add yourself as the approver. Configure this policy with a merge request and merge it in.

  • You'll notice a separate security policy project has been created with a similar name to your project that has the policy yaml specified. Verify in your non security policy project Settings-->Merge Requests-->Security Approvals section, the SAST Policy security approval rule you created exists. This requires an extra approval to happen if a high or critical vulnerability is discovered in the SAST scan. Explain how this MR workflow set up enables the shift-left mindset and its benefits (you will show this later in a merge request).

  • Note that only the latest templates allow scanners to run in Merge Requests, so you have to explicitly set this as the default is to use the default template. Your resulting policy YAML in the policy project should then contain a block like this:

scan_execution_policy:
- name: Secret Detection and SAST Enforcement
  description: ''
  enabled: true
  rules:
  - type: pipeline
    branch_type: all
  actions:
  - scan: secret_detection
    template: latest
  - scan: sast
    template: latest

6. Create a second security policy by navigating to Secure-->Policies. Select "New Policy" and then "Scan Execution Policy". Name the policy "Secret Detection and SAST Enforcement". In the Actions tab, "Run a Secret Detection scan with the following options:" and leave the default settings. Add a second Action "Run a SAST scan with the following options" and leave the default settings. In the Conditions, "Triggers every time a pipeline runs for all branches with No exceptions.". Configure this policy with a merge request and merge it in. This will force those security scans to run on every commit, so that developers don't need to worry about writing the scans in themselves or not adhering to security rules.

7. Create a new branch called "demo" from the master or main branch. You can do this via the UI by navigating to Code-->Branches-->New branch. You can also do this locally by cloning the repository into your terminal (git clone), checking out a new branch named demo (git checkout -b demo), and setting the upstream branch (use the --set-upstream-to flag on git branch or git push after making the following changes). Add in these 2 changes:

  • First, edit the secrets.txt file by either adding a space at the end or changing the last letter of each password to ensure the secrets detection scanner looks at this file (by default it only runs on files with a diff). This will purposefully break the secrets detection scan.
  • In routes/index.js, add or uncomment the below line of code within the "router get" block right after the "res.render" line. This will purposefully break the SAST scan.
res.send('id: ' + req.params);

8. Commit those changes, and create a merge request. A new merge request pipeline should start that runs the dummy "build" job, and the jobs injected by the Scan Execution Policy. Note that the containerize and test jobs do not get run as they lack a rules: block that specify running in a Merge Request pipeline. This has been done to reduce the time taken for each pipeline in a demo environment.

  • Use this time to explain why the security scans are running (sast, secrets detection) even though they aren't specified in the .gitlab-ci.yml file. Talk about the scan execution policy enforcing that pipeline to be run in addition to the application pipeline.

9. Now back to the MR UI, after the pipeline has completed, you should see that the 2 scanners detected several vulnerabilities, as shown in the image below. Note: With the updates to scan execution policies going in, this may not automatically populate on the MR widget just yet, but you can dig through the artifact report to see the vulnerabilities. Additionally, the merge request will be blocked because it requires 2 approvals: 1 from the general MR approval rule you created in step 4. The other approval is the SAST Policy Approval rule that was created in step 5 which is blocking the merge because a critical or high vulnerability was detected. Use this to explain how different user groups can be added as approvers depending on which security scan is run. Explain the value of seeing all of this information in the MR widget to ensure all checks and balances happen before code is merged.

MR view with SAST & Secret Detection Vulnerability

10. For demo purposes, go ahead and approve the MR yourself (if configured correctly, it should cover both approval rules) and merge the MR into the default branch with the vulnerabilities and you should see a pipeline kicking off that runs build, containerize, test, and deploy jobs in a single pipeline.

11. Once the pipeline completes, navigate to Secure-->Security Dashboard and show the latest vulnerabilities that were introduced into the default branch with the last merge. Walk through the vulnerability report and it's capabilities, including the remediation process.

12. So far we have focused on setting up policies on a given project, many organisations need to enforce standards across many projects, this can be achieved with Compliance Frameworks. In the next steps we will demonstrate configuring Compliance Frameworks, Security Policies and Pipeline Execution Policies.

12.1 At group level, go to to Secure -> Compliance center. From the Frameworks tab, click the New Framework button, on the New compliance framework, enter a Name for the new framework, a Description, a Background color, ignore the compliance pipeline configuration and leave the 'Set as default' checkbox unchecked. Click the 'Add framework' button to complete the process.

12.2 When the framework is initially created it is empty, the next step is to create our policies and attach them to the framework.

12.2.1 In this step we are going to create a pipeline execution policy which enforces container scanning, this could be done with a Scan execution policy, however we want to demonstrate defining pipeline changes as part of a Compliane framework and how they can be enforced.

12.2.2 From the top level group, create a new project Pipeline Execution Policies. With the project created, create a new file container-scan.yaml and add the content below -

container_scanning:
  variables:
    CS_IMAGE: $CI_REGISTRY_IMAGE

include:
  - project: $CI_PROJECT_PATH
    ref: $CI_COMMIT_SHA
    file: $CI_CONFIG_PATH
  - template: Jobs/Container-Scanning.gitlab-ci.yml

Commit the changes with a suitable commit message.

12.2.3 From the top level group goto Secure -> Policies click the New policy button, scroll to Pipeline execution policy and click select policy. On the subsequent page enter a name and description for your policy, ensure the status is enabled and in the Policy Scope section select projects with compliance frameworks and then select the compliance framework created above. In the actions section select inject, select the PipeLine Execution Policies project, enter container-scan.yaml in the Link existing CI file field. Choose the appropriate branch for the File reference and leave the Add job name suffix set to On conflict. With all the information added, you can click the Update via merge request button and merge the changes.

12.2.4 Next, go back to the top level group Secure -> Compliance center. Click the projects tab and click the action button next to the demo project you have been using, This will open the select frameworks dialog, choose the compliance framework created above and click away from the Select frameworks dialog. Click on the frameworks tab and demonstrate to the participants that the Framework is now assiciated with your demo project.

12.2.5 The final step is to run a pipeline for your demo project, when the pipeline is triggered you will be able to demonstrate that the Container scanning job has been added to the test stage. Allow the pipeline to complete.

Pipeline execution with container scanning

13. This is the end of the workshop portion of the service. It is expected that the attendees will have more questions about each of the security topics you've showed, so schedule deeper dive sessions as needed. The topics are listed below. Please clean up the individual project and security projects you may have created if no longer needed. Do not touch the Original Project.

14. The last part of this engagement involves moving the test project into the customer environment. To do this, you will need access to the customer GitLab instance. Have the customer create and provide access to an empty top-level group on their instance. Import the project and configure the merge request approval and scan execution policies as you did in the workshop. Consider recording the demo session so that the customer can review it again later, especially if they want to share it with team members who couldn't attend. In cases where the services delivery team lacks direct access to the customer's GitLab instance, consider other ways to deliver the example if the customer requests it.

Additional steps for an airgapped or security locked down environment (aka gitlab instance cannot access internet)

From our GitLab docs,

At a high-level, the security analyzers are delivered as Docker images and may leverage various package repositories. When you run a job on an internet-connected GitLab installation, GitLab checks the GitLab.com-hosted container registry to check that you have the latest versions of these Docker images and possibly connect to package repositories to install necessary dependencies.

In an offline environment, these checks must be disabled so that GitLab.com isn’t queried. Because the GitLab.com registry and repositories are not available, you must update each of the scanners to either reference a different, internally-hosted registry or provide access to the individual scanner images.

  1. Add this configuration described in these SAST docs to the project in the customer environment to run SAST offline with locally imported docker images and not depend on GitLab's Container Registry.

  2. Add this configuration described in these Secret Detection docs to the project in the customer environment to run Secret Detection scanning online using a local copy of the Secret Detection Docker image instead of GitLab's Container Registry.

Guide to Facilitate Deeper Dive On Each Concept

It's important to note that customers often face process challenges more than technical challenges. Be prepared to discuss topics such as:

  • Organizational structure for security (many organizations don't have dedicated security personnel)
  • Branching strategies and workflows
  • GitLab group structure and organization
  • Integration with third-party scanners (e.g., SonarQube for code quality)
  • Secret push protection

Shift-Left Security Mindset + Merge Request Setup

Watch this video from the socumentation that summarizes the shift left mindset.

  • The stages and jobs defined in the .gitlab-ci.yml file will run and output meaningful results (test outputs, logs, etc.) on the GitLab UI.
  • Using Auto DevOps templates, we can include security scans with 1 line of code to run security checks and output logs to developers before code gets merged into the default branch. The templates run the scans in a test stage that is automatically created for you.
  • Merge request approvals can be set up so that certain users or groups of users need to review a merge request before that branch is merged into a specific branch. This encourages code reviews, ensures high code quality, and prevents unwanted changes from being released.
  • Customization allows for multiple approval rules to be created, different rules per branch, and rules based on security scan results.

Secret Detection

  • GitLab's Secret Detection solution scans through source code to look for over 100 common patterns that would signify a token or password (i.e. access tokens, popular cloud provider keys, URLs, etc.).
    • The GitLab solution spins up a container that runs the GitLeaks scan for you in the test stage.
    • You can override certain predefined rules or create custom rules if you need to.
  • By default, secret detection only scans commits, not the working tree, but the scan template can be configured to do a historical scan, scan all commits in a merge request (use the secret-detection.latest job for this), scan all branches, etc.
  • You need a docker or k8s executor on a Linux based GitLab runner to use the secret detection GitLab template.
  • You can configure the scan to omit certain paths, secrets, and use a specific version of the analyzer.
  • The job outputs an artifact in JSON format which by default is available for 7 days.

SAST

  • GitLab's SAST scanner checks out source code for vulnerabilities and outputs the report into a JSON artifact.
    • Examples of SAST vulnerabilities include a dangerous class attribute or unsafe code that can lead to unintended code execution or code that is available to cross-site scripting (XSS) attacks where unauthorized access to session data can be attained.
    • The SAST job runs in the test stage when including the template.
      • Because the job automatically runs in the test stage, and by default does not contain the needs keyword (aka it does not depend on any other job), it will run after the build and/or containerize job is run and scan the entire working tree. Therefore, if anything from those jobs have been saved as artifacts or within the cache (i.e. a poetry install in a build stage), secrets detection could potentially detect dozens of vulnerabilities based on the dependencies that were installed.
      • The solution here would to be to use the needs keyword to ensure secret detection runs first or encourage teams to run it in the .pre stage instead of the test stage.
  • You need a docker or k8s executor on a Linux based GitLab runner to use the SAST GitLab template.
  • GitLab's SAST scanner detects the language used in the project and automatically runs the corresponding analyzer. For example, GoSec runs when Go is detected, Bandit is run for Python projects, brakeman for Ruby, nodejs scan for Node.js projects, etc.
  • False positive detection, currently is only available for Ruby and Go projects for Ultimate customers.
  • Certain projects need to have code pre-compiled before running a SAST analyzer, and this can be achieved by outputting project dependencies into a working directory via the artifacts keyword, setting the COMPILE variable to false, and making sast depend on the build job with the needs keyword.
  • You can configure the scan to omit certain paths, use a specific version of the analyzer, increase the debug log level, modify the depth of scan, pass credentials to a private repository, etc.

Security Dashboard & Vulnerability Remediation

  • The GitLab Security Dashboard is the one stop view to track the results of the most recent GitLab security scans on the default branch.
  • The dashboard provides a view of vulnerabilties over time at both the project and group level, and for the latter, will assign a grade based on the highest severity open vulnerability (does not include dismissed or resolved vulnerabilities).

Vulnerability Management

  • The Vulnerability Report for a project shows details on each vulnerability including when it was detection, description, where in the code it was detected, linked issues, and available actions.
  • The report can be filtered by date, severity, scanner, activity, and project.
  • Here's a short video on vulnerability management.
  • The default vulnerability status is Detected, which shows up as "Needs Triage" in the UI. Dismissing a vulnerability ensures it won't pop up on subsequent security scans while resolving a vulnerability will change its status, but allow the vulnerability to pop up again if the problem still exists.

Scan Result & Scan Execution Policies

Scan result policies are used to enforce a custom approval rule when a security scan detects a vulnerability.

  • Scan result policies can be enforced at the group level with a corresponding security policy project and customizable policy.yml file that gets created within the group.
  • Customize the approval group to ensure certain people need to approve an MR with vulnerabilities for it to be merged.
  • This enables developers to collaborate with security to determine whether the vulnerability is a false positive, insignificant, or requires a code change.
  • You can customize your scan result policy to only go in effect for a certain severity of vulnerability (low, medium, high, critical).
  • The policies are only enforced when the target branch for the MR is a protected branch.

Scan execution policies can be used to schedule pipeline runs on the desired interval.

  • Enforce these policies on all projects, whether or not they have a .gitlab-ci.yml file.
  • These are the future direction for GitLab and will be customizable to allow for custom CI job inputs into the policies.

Discovery + Roadmapping

The engagement is 8 days, and yet the demo above and deep dive sessions should only take a few days to deliver. So what does the rest of the engagement entail?

  • Remember, the goal of this engagement is to ensure the customer leaves with a foundational understanding of GitLab Security features AND a path for how to transition to using it. While we don't touch customer code in this engagement, the remaining engagement time can and should be dedicated to the following activities. Time spent on each activity may vary as this is just a general outline of discussion points.

  • Setting expectations: It's crucial to clearly communicate what this service includes and what it doesn't. Some customers may have different expectations about what they've purchased, so clarify the scope early to avoid misunderstandings. For example, this engagement doesn't cover general CI/CD components beyond security-related aspects.

    1. Understanding the customer's current security setup and design.

    • What is your security team structure and group responsibility breakdown?
      • Do they have separate teams for QA, application security, compliance/governance, etc.?
      • Who is responsible for ensuring code in production is secure? How is this done today?
    • What security tools are you currently using and what are they used for?
      • Are the GitLab scanning solutions able to displace the customer's existing solutions?
      • What technology stacks does your business use?
      • Are there any applications that use languages or frameworks that aren't supported by GitLab?
      • What are your current security requirements, processes, and/or policies?
        • Do you have certain code scans that are required for certain applications or when code meets between environments?
        • When, if at all, do you require manual intervention and review of code?
        • How do you manage vulnerabilities today?
        • Do you have a standard way of enforcing all applications run the required security scans?
        • How do you manage security data across a large number of applications?

    2. Identify why the customer is interested (i.e. what is not working so well with their current setup).

    • Are there too many disparate tools to manage?
    • Is there a lack of consistent process or security standardization between applications?
    • Are there too many manual processes that drain the resources?
    • Is there a lack of collaboration or even tensions between the security and development teams?
    • Is it too difficult to manage security data and/or ensure all of the applications meet the necessary requirements?

    3. Using the above information, discuss topic by topic how GitLab can solve the customer's issues.

    • Examples:
      • Security policies can achieve enforcement of standardized security templates across groups of applications.
      • Shifting security to the left with merge request approvals combined with scan result policies can foster collaboration between security and developers to ensure all code is checked before moving to production.
      • The security dashboard allows you to manage vulnerabilities over time across all of your projects, gather data about vulnerability remediation trends, and export that data for your own reporting purposes.
      • Security can easily be automated within pipelines.
      • All of the above can be done within a single platform aka GitLab, which simplifies the user experience for both developers and security team members.

    4. Using the above discovery information, design the security vision of what the customer's end state GitLab setup could look like.

    • Example:
      • 10,000 foot view: GitLab SAST, DAST, Code Quality, Secrets Detection, and Container scanning are used to displace existing Snyk and Checkmarx implementation.
      • Organization: Top level group contains centralized repository of security templates with different scanning setups that are automatically enforced to different types of projects. Developers don't need to worry about including any security scans in their code as these security pipelines are automatically injected into the application pipelines. The security team has a single location for all of the templates, which makes it easy to manage and update as needed. Developers can also easily contribute back to those templates through issues and merge requests. The necessary security folks will be automatically asked to review any application code MRs that violate a security policy.
        • Workflow: Every code commit triggers a SAST and Secrets Detection scan. On merge request pipelines, additional Code Quality and Container Scans are run. There are different security policies assigned to differentiate between web applications, payment applications, business applications, etc., each with different templates to match the specific security requirements. Security professionals can use the Security Dashboard to pull reporting and trend data for vulnerabilities broken down by severity at both a group and project level.

DevSecOps Transformation

Our Security and Compliance transformation work supports the GitLab Value Driver of Reduce Security and Compliance Risk - Simplify processes to comply with internal processes, controls and industry regulations without compromising speed.

Product capabilities

Many of the capabilities require an Ultimate tier GitLab subscription, so Security and Compliance Risk Reduction falls under the broader category of "GitLab Ultimate Adoption".


Start with the Transformation Mindset

A Transformation Mindset

Typically, application security adoption follows broad adoption of GitLab for SCM and CICD. But it's possible that some team members are still onramping with some aspects of GitLab Flow such as feature branches and shifting left, and might even be new to Git itself (i.e. from Perforce or svn) - that's OK.

Often, senior leadership and team members have a beginning understanding of GitLab's application security offerings (i.e. from the sales process) and want to learn more. That's a hook for getting started.

It's best if the company has CI/CD in place for other operations (unit testing and cloud deployments, for example) - but not entirely necessary.

For success, it's critical that we provide leadership across both aspects of the application security transformation.

Technical Aspects of Success

The actual GitLab features that support business goals and the integration/configuration of those features in the groups, subgroups, and projects. Plus enablement of hands-on teams in efficient utilixation of the product's capabilities.

Organizational

Aligning the way people plan, communication, collaborate, work, and think with business goals, organizational values, regulatory concerns, and each other.

Organizational transformation has 5 parts:

  1. Strategy: Clear goals and objectives (OKRs, KPIs, MBOs, etc) combined with meaningful organizational values
  2. Skill: Knowledge and abilities of individuals, backed by training, enablement, and support
  3. Styles: A generative culture that fosters both autonomy and collaboration
  4. Structure: Appropriate roles, job titles, reporting relationships, dotted lines, and incentives
  5. Systems: Tools and technology that underlie successful processes (including but not limited to GitLab)

As a product/engineering-driven software company, we have a tendency to focus on the technical aspects of transformation and neglect the organizational ones - but those are the hard ones, and must be addressed to ensure success.

Technical Aspects of Transformation

GitLab features related to application security

  • Scanners
    • All the types
    • Results in artifacts
    • Results in MRs
  • Security policies
    • Policy projects
    • Scan execution policies
    • Scan results policies
  • Executive reporting
    • Security dashboard
    • Security center
    • Vulnerability report
  • Vulnerability management
  • Compliance frameworks
  • Integration of 3rd party scanners

Organizational Aspects of Transformation

For developers

  • Agile best practices
  • Git best practices (if coming from pre-Git SCM)
  • MR best practices (if coming from other Git SCM)
  • Branching/MR strategy (first agreed at org level)
  • Integrating scanning with automated build/test
  • Feature/MR approval/merge process/workflow
  • Runner resource optimization (maybe some scans still only run post-merge or pre-release)

For devops teams

  • Maintenance of GitLab (is this SM?) and Runners
  • Release verification and deployment/distribution process
  • GitLab and scanner administration
  • Process for future enablement of dev and infosec teams re Git, GitLab, scanners, and workflows

For infosec team

  • Decisionmaking processes (which scanners, what stages)
  • Responsibility (DRIs) for scanner implementation, vuln management, MR review, release approval, etc
  • Process and staff allocations for handling results from first-time scanning passes

For everyone

  • Regular process for handling results and vulnerabilities, including escalations
  • Collaboration and communication process with other teams
  • Integration of vulnerability management with agile planning (i.e. issues to resolve vulns)

Getting started

Project phases might include:

  • Advanced education - even after digesting courseware and documentation, teams might require informal enablement to address specific requirements and questions
  • Pilot project selection - start small with 3-5 projects - enable scanners first to develop remediation workflows
  • Prioritization / sequencing - GitLab has a lot of application security capabilities - choose an order that makes sense for the org
  • GitLab organization - Teams such as CS, SA, Field CTO, product marketing, product mangement, and engineering all have deep insight and love to help customers
  • Initial scan remediation - how to handle the onslaught of vulns - process and communication - pay particular attention
  • DevOps fundamentals - group hierarchy, branching strategy, agile, merge methods, approval policies, etc. - all prerequisite for effective app security

Shared Responsibility

The Security and Compliance capabilities of the GitLab product cover only one aspect of security -- application security. Other aspects, such as network, infrastructure, social, and endpoint security remain the responsibility of the customer organization. Some customers need help understanding the difference.

Furthermore, while GitLab provides the tooling (and sometimes infrastructure) for application security testing and vulnerability management, the ultimate responsibility for application security lies with the customer.

The language below might help to explain the nuance of shared responsibility, loosely based on the AWS Shared Responsibility Model for cloud services.

Customer infrastructure and applications

Customers bear responsibility for implementing and managing the security of their applications and applications, including how they utilize GitLab's security tools within their development pipelines. This encompasses reviewing and acting upon identified vulnerabilities, managing dependency updates, and implementing recommended security fixes

Customers must configure security scanning tools appropriately, interpret results, and make informed decisions about vulnerability remediation priorities.

For example, while GitLab provides dependency scanning capabilities to identify known vulnerabilities in external libraries, customers are responsible for actually updating these dependencies and ensuring their application remains secure. Similarly, while GitLab provides SAST scanning capabilities, customers must enable and configure the rules and execution policies appropriately for their specific use cases.

GitLab platform capabilities

GitLab develops, maintains, and delivers capabilities that support customers in ensuring security of their infrastructure and applications. The Application Security capabilities including several types of vulneravility scanning engines as well as various user interfaces and APIs for understanding and addressing vulnerabilities.

Our focus is on preventing software vulnerabilities from entering the supply chain - an approach known as "shift left".

We focus on improving scanning accuracy, reducing false positives, and enhancing detection capabilities through continuous research and development. In addition to the scanners themselves, GitLab maintains a database of known vulnerabilities and groups of analyzer rules. Most of the vulnerability information and rules come from public sources. Parts of the company focus on performing security research, improving GitLab's security capabilities, and developing proofs of concept while maintaining the databases and fulfilling CVE Numbering Authority duties.

GitLab platform security

With regard to the GitLab platform itself, responsibility for security falls mostly with GitLab.

GitLab maintains responsibility for providing and securing the fundamental security scanning and vulnerability management infrastructure. This includes maintaining and updating the security scanning engines, vulnerability databases, and the platform infrastructure that enables security testing.

GitLab also manages the infrastructure of GitLab.com, handling maintenance and addressing any incidents that arise.

In the case of self-managed GitLab, instance administrators take primary responsibility for infrastructure security. The GitLab support organization is available to help.

Security incidents

(Including breaches and "zero-day" vulerabilities)

When handling a security incident, follow guidance on the Responding to security incidents page in the documentation.

Customers must handle security incidents in their area of responsibility, including:

  • CI/CD security incidents, including exposed job tokens and pipeline secrets
  • Project and group setting misconfigurations
  • Credential exposure incidents
  • User account compromises within customer organizations

GitLab provides capabilities to support customers, including:

  • Audit events
  • Credentials inventory
  • Secrets detection

With regard to "zero-day" vulnerability discoveries, the best defence is to ensure that application SBOMs (Software Bill of Materials) are up-to-date so teams may quickly identify vulnerable applications.

Managing vulnerabilities

The goal is to prevent breaches and attacks. Active vulnerability management is key. At a high level:

  • The scanners scan and identify potential vulnerabilities
  • The Vulnerability Report provides a mechanism to understand, accept, dismiss, comment, and assign vulnerabilities
  • A developer or infosec professional has to make the decision as to which vulnerabilities to accept or dismiss
  • The “Explain This Vulnerability” and “Security Training” capabilities can help in some cases

One of our team members shared:

Identifying false positives requires investigation. There may be times when it’s “obvious” but those are likely rare. When you have identified a false positive, though, it might be best to tweak the rules or configuration to avoid detecting such instances in the future, where tweaking rules is available.

Resources

The Nested Nature of Security

Security responsibilities are like Russian (Matryoshka) dolls, fractals, or turtles all the way down - the same patterns repeat at different scales. For example:

GitLab <-> Customer

  • GitLab provides the software, security features, and patches
  • Customer decides how to use them and when to apply updates
  • For self-managed, customer owns infrastructure security
  • For GitLab.com, GitLab handles the infrastructure

Within the Customer org: Security Team <-> Platform Team

  • Security sets policies and requirements
  • Security maintains vulnerability management program
  • Platform implements and maintains controls
  • Platform handles infrastructure patching and hardening

Platform Team <-> Dev Teams

  • Platform provides secure infrastructure and tooling
  • Platform configures security scanners and guardrails (components, frameworks, etc.)
  • Dev teams configure security in their pipelines (sometimes)
  • Dev teams manage application dependencies

Dev Teams <-> Individual Devs

  • Teams set security standards and review processes
  • Teams handle vulnerability remediation priorities
  • Devs make day-to-day security decisions in code
  • Devs implement secure coding practices

The common pattern being one party provides capabilities, the other decides how to implement them. This creates a chain of shared responsibility that scales from the organizational level right down to individual code commits. Even within each level, you'll find the pattern repeating - like how dev teams handle both infrastructure-as-code security and application security, each with their own provider/implementer relationships and alignment to enterprise strategies.

Preparation

"Phase Zero" activities

Process

(Enablement as required throughout)

  1. Set expectations
    • Everything will be red at first
    • Continuous improvement
    • Requires research, decisionmaking, collaboration, and additional coding
  2. Establish a cross-functional team
    • Executive sponsor
    • InfoSec
    • DevOps
    • Product
    • Application team roles- Technical Architect and Dev Leads
  3. Identify and prioritize pilot projects/teams

Collaboration focus:

  • Select pilot team(s)
  • Determine integration requirements (Jira, 3rd party scanners, etc)
  • Determine current state (i.e., some scans already running)
  • Summarize languages and frameworks used (to determine scanner applicability)

Pilot Project Selection

Characteristics of a pilot team/project

  • Already using GitLab, Merge Requests, and CI/CD
  • Already partially familiar with concepts like SAST and Secret Management
  • Publishing a web application or API service, to provide effective confirmation of DAST capabilities
  • Having projects in standard conventions like containerization via Dockerfile or package management with package managers like poetry,maven,yarn etc. in order to use Container Scanning or Dependency Scanning capabilities
  • A standard model, for how most other teams work. Especially how branches and merge request approvals are managed should be matching overall setup

Custom role

The security team will need the ability to work with vulnerabilities and compliance frameworks in order to do their work. Fortunately, the Custom Roles capability supports such activity. Specifically, we need a Custom role at the top-level group with the following properties:

  • Name: Application Security (or similar)
  • Description: Manage application security policies, vulnerabilities, and workflow (or similar)
  • Base role: Reporter
  • Custom permissions:
    • Admin compliance framework
    • Admin merge request
    • Admin vulnerability
    • Manage security policy link
    • Read dependency
    • Read vulnerability

(Note there is an active issue to standardize such a role in GitLab.)


Application Security Testing

All activities related to enabling scanning

Capabilities

Capabilities under the heading "Application Security Testing" include:

  • Scanners
    1. Secret detection
    2. Dependency scanning
    3. Container scanning
    4. SAST + SAST-IaC
    5. API security testing
    6. DAST browser-based scanning
    7. Fuzz (both kinds)
  • Scan Execution Policies
  • Pipeline Execution Policies

Approaches

Note that there are three distinct ways to implement security scans

  1. Direct CI/CD - Individual scanners in the CICD configuration for individual projects - perfect for ensuring operational integrity and generating some initial data so that workflow, triage, and remediation process development can begin.
  2. CI/CD library - alongside other shared CICD configuration, wrappers for the scanner configurations - allows parameterized configuration to be shared among projects - best way to organize CICD configs in a large organization.
  3. Scan Execution Policies - enforce scanning across a portfolio of projects with minimal adaptation. More complicated for DAST and API scanning. Doesn't support Fuzz scanning (?)

Third-party scanners

...

Security Risk Management

Information security professionals face a daunting task when first adopting GitLab Application Security Testing, whether coming from other scanning software or entering the world of automated security testing for the first time. The workload can overwhelm them. We break it into two distinct phases:

Bulk Triage Phase

Handling vulnerabilities that appear in the Vulnerability Report when scanners are first run on a project

Continuous Security Phase

Handling vulnerabilities that appear after bulk triage has completed, whether because of code changes or newly understood vulnerabilities

Other resources

Issues within AppSec SME channel (GitLab team only)

Bulk Triage Phase

When first adopting GitLab Secure

  1. Prioritize projects according to exposure risk: High/Medium/Low based on business impact and network exposure. Start with the high-priority projects and move down. Approach each project one at a time for initial bulk triage
  2. For one project at a time, ensure that scan execution policies have been applied to the project and have run successfully on the default branch
  3. Filter the Vulnerability Report for
    • Project: The project under examination
    • Status: Needs triage
    • Severity: Critical
    • If necessary because of the size of the results list, a scanner in the following order:
      1. Secret detection
      2. Dependency scanning
      3. Container scanning
      4. SAST + SAST-IaC
      5. API security testing
      6. DAST browser-based scanning
      7. Fuzz (both kinds)
  4. Starting at the top of the search results, open each vulnerability
    • Read the vulnerability page thoroughly
    • Click through to the links as necessary to understand the vulnerability
    • If needed and available, click "Explain this vulnerability with AI" for more information
    • Decide how to handle this vulnerability
    • Execute the decision by choosing a Status in the top right corner
    • Add a comment to the vulnerability explaining the decision
    • If the status is Confirmed, create and assign a Jira issue
  5. Continue through the remaining vulnerabilities in the search results, performing the previous step for each
  6. If you filtered by scanner, update the filter to the next scanner and repeat
  7. Complete triage of all the Critical vulnerabilities in one project before moving to the next project
  8. Complete triage of all vulnerabilities in the following order shown in the table below
  9. While triaging vulnerabilities, when trends appear, consider adjusting the scan execution policies by referencing the GitLab documentation. If different projects and/or groups seem to require different policies, consider creating separate scan execution policies and linking them appropriately
    • All the scanners have variables to customize
    • Secret detection and SAST have RULESET_GIT_REFERENCE variables
    • DAST browser-based scanning and API scanning have site profiles

Proposed order for vulnerability bulk triage

High risk projectsMedium risk projectsLow risk projects
Critical vulnerabilities124
High severity vulns357
Medium severity vulns6810
Low severity vulns91112
  1. Perform initial application risk assessments
  2. Prioritize applications
  3. Set non-intrusive scanners
  4. Establish guardrails with exceptions
  5. Monitor and adjust

Continuous Security

  1. Triage vulnerabilities within 24 hours to determine:
    1. Does the vulnerability actually apply to your application. Is it reachable, does the impact apply to your application, etc.
    2. Who reported the vulnerability? If the issue is reported through a bug bounty, customer, or penetration test, it is often more critical to solve compared to a security scanner
    3. If the vulnerability is valid, how likely is it to be exploited? Is there a public exploit, is the attack complex, is it easy to reach the vulnerable component?
    4. What impact would the vulnerability have if it were exploited? This is commonly done using impact to CIA (Confidentiality, Integrity, Availability)
  2. If the vulnerability does apply, assign a priority based on:
    1. Severity, usually based on the CVSS score. The higher the CVSS score, the higher priority the fix is
    2. How likely an exploitation of the issue is. If there is active exploitation or the possibility for it, higher priority could be assigned
  3. Develop a fix to the vulnerability - Of note, a vulnerability fix will typically apply to the latest release, but may also apply to older releases depending on your support model
  4. Determine a release strategy - Larger companies often follow a specific cadence for security releases, like Patch Tuesday, with out of band releases for critical issues. I've also seen monthly release cycles like Android security bulletin. The release cadence will vary a lot by company and release cycle preferences
  5. For fixing common vulnerabilities, there are a few details that might be good to discuss:
    1. Default to known secure patterns for development. For example, use parameterized SQL vs concatenation. Our security outputs do a great job of showing these approaches and linking to CWE resources to help here
    2. In addition to this, using DRY to standardize is helpful. For example, having a single class of SQL functions helps ensure all SQL queries use the same logic, avoiding mistakes in implementation
    3. Resources like OWASP, CWE, and NVD are often helpful for understanding these patterns
  6. Looking for variants on a security vulnerability is often advisable. If for example you find a SQL injection vulnerability, you should look at your code to try to identify if other examples exist of the same weakness

Vulnerability Evaluation

During both the initial bulk triage phase and the later continuous security phase, an information security professional must evaluate individual vulnerabilities and make decisions.

How to evaluate

  • Criticality and impact - Vulnerabilities that are highly critical and could result in severe consequences should be prioritized
  • Attack vector - Vulnerabilities with a clear and practical attack vector should be prioritised
  • Industry best practices - Stay informed with your industry best practices and compliance standards that are explicitly highlighted by the regulatory bodies
  • Prioritization with other project work - Organizational leaders must agree that vulnerability remediation is as important to the business as other development activities

Vulnerability Assessment and Severity Ratings

Vendor-Specific Vulnerability Ratings

Different vendors may have varying criteria for assessing vulnerability impact. For instance, historically, a "low impact" rating from Red Hat typically indicated that a vulnerability was either unlikely to be exploited or that its exploitation would not likely lead to severe consequences. While specific vendor criteria may evolve over time, the fundamental principles often remain consistent.

National Vulnerability Database (NVD) and Vendor Assessments

The National Vulnerability Database (NVD) maintained by NIST has faced challenges in recent years, including a lack of contextual information and significant backlogs. As a result, many security professionals often prioritize vendor-specific vulnerability assessments. Vendors frequently employ more dedicated researchers than the NVD, potentially allowing for more nuanced and timely vulnerability evaluations.

Mitigation Strategies

For certain types of vulnerabilities, such as those potentially leading to Distributed Denial of Service (DDoS) attacks, implementing rate limiting can often serve as an effective mitigation strategy.

Contextual Nature of Severity Ratings

It's crucial to understand that severity ratings should not be interpreted in isolation. The impact of a vulnerability can vary significantly depending on the specific context of the affected application or system. This contextual nature underscores the importance of human review in the security assessment process.

Example

A potential DDoS vulnerability might have catastrophic implications for a mission-critical or safety-critical system, while its impact on an e-commerce website, though still significant, might be comparatively less severe.

Best Practices for Vulnerability Management

  1. Consider multiple sources when evaluating vulnerabilities, including vendor-specific assessments and broader databases like the NVD.
  2. Understand the specific criteria used by different sources to determine severity ratings.
  3. Always contextualize vulnerability severity within your specific application or system environment.
  4. Implement a robust vulnerability management process that includes both automated tools and human expert review.
  5. Prioritize vulnerabilities based on their potential impact on your specific systems and business operations.

Risk and Confidence

From issue under GitLab UX Research entitled Further understand the job of prioritizing security scan results and assessing their risk

Users want to estimate the potential impact of a vulnerability if it is exploited.

Users want to estimate the likelihood of a particular vulnerability being exploited.

Most organizations have a clear process for assessing the risk of security scan results and prioritizing them alongside other vulnerabilities, even if it's not always formally named.

Risk assessment is more of a top-down process compared to vulnerability management. Risk assessment tries to determine how the organization can be exploited, assess the likelihood of those events happening, estimate the event's impact, and take necessary actions to prevent that exploitation.

The data points that users currently use to estimate the likelihood of exploitation and possible impact are Severity rating, what branch/group/project the vulnerability was in, the title and description of the vulnerability, what type of scanner was used to find that vulnerability, and the CVE number if available.

From survey results:

  • Most (6/8) participants did not look at security scan results outside the default branch.
  • Most (6/8) participants only circumvented their typical workflow if an immediately exploitable vulnerability surfaced.
  • The majority (5/8) of participants did not expect a risk assessment tool to contextualize results to the organization's business needs or infrastructure. But, everyone expressed a desire if a tool could do that.

A business-critical threat is one that: 1. has a high chance of being exploited 1. disrupts the business model in a significant way 1. can extract customer or user data from the organization.

Also see the discussion of "confidence level" as a concept.

Capabilities that help with understanding and prioritizing vulnerabilities

Software Supply Chain Security

Adoption Roadmap

  • Integrate this content into the structure of the delivery kit

Phase 1: Perform pilot with 1-4 dev teams/projects

Key staff: Pilot project development team

Key GitLab capabilities: CI/CD, Scanner templates, Pipeline security tab

Collaboration focus:

  • Implement individual scanners in the CICD configuration for individual projects
  • Run on both default branch and MR scans
  • Include static and dynamic scans where appropriate
  • Ensure operational integrity (do the scans run and generate data?)
  • Identify performance optimization opportunities

Phase 2: Initiate vulnerability management

Key staff: Information Security team

Key GitLab capabilities: Vulnerability Report

Collaboration focus:

  • Immediately upon the arrival of scan data, begin reviewing vulnerability report

Phase 3: Activate scanners universally

Key staff: Information Security team

Key GitLab capabilities: Scan Execution Policies, Pipeline Execution Policies, CI/CD components

Collaboration focus:

  • Generate security policy project(s)
  • Replace one-off scanner implementation in pilot project(s) with links to security policy project(s)
  • Implement Pipeline Execution Policies (or CI/CD components) for scan types unsupported by Scan Execution Policies
  • Draft plan for universal policy rollout

Why Scan Execution Policies?

  • They can be rolled out in pilot group
  • Security scans by default are allowed to fail, therefore should not affect pipeline scalability.
  • That being said, if pipeline duration for any monorepos are affected, due to slow SAST execution, this may need to be adressed.
  • Using latest version is possible via template variable, in order to activate scans for both Merge Request Pipelines and Branch Pipelines
  • For air-gapped environments you can use SECURE_ANALYZERS_PREFIX with Scan Execution Policy variable to activate scans.

Phase 3: Develop vulnerability management workflow

Key staff: Information Security team

Key GitLab capabilities: Vulnerability Report, Merge Request Approval Policies

At this stage, we have running pipelines and scan results. Based on the results now we can finally start identifying how a Merge Request Approval Policy and Vulnerability Management process should actually work. At this stage we develop the concept together with the teams involved related to

  • who can change state of a vulnerability.
  • who can approve an MR, when there is a certain level of vulnerability.
  • when are you allowed to deploy a change, even when there is a vulnerability.

Bear in mind, that current GitLab Vulnerability Management is based on GitLab Flow branching model. Identifying, if there are any gaps related to trunk based branch models is crucial in this stage, to consider potential workarounds or customisations to the Vulnerability MAnagement flow

Phase 4: Continuously reduce security risk

Once the piloting stage is complete, the tooling and concepts in place should achieve following.

A self-service rollout VM framework to other teams should be implemented. Teams have the option based on their workflow to trigger rollout themselves with minimal support requirements from Security or pilot teams. Some additional features to cover

  • Type of different compliance frameworks should be identified. Certain organisations like Finance, can require different compliancies. Whilist organisations like HR may be subject to data regulations. Also the separation of duties, may play a role here, in regards to how vulnerabilities are differently classified.

  • We can now consider establishing Pipeline Execution Policies to integrate with other 3rd party scanners or enforce other custom CI yamls in the organisation.

Technical and organizational changes

  • Access control for infosec team
  • Integration with Jira if required
  • Developer teams learn DevOps best practices such as trunk-based development, MR approvals, review apps
  • Infosec and dev teams develop combined communications/workflows for vulerability triage and management

Meeting Teams Where They Are

Security is a journey, not a destination (as with most transformation topics). While we want every team to be secure, trying to implement everything at once is like trying to run before you can walk - you'll likely stumble and create resistance to security initiatives.

Instead, meet teams where they are and help them level up gradually

Crawl:

  • Start with easy wins like dependency scanning
  • Focus on fixing critical vulnerabilities first
  • Build security awareness through documentation and training
  • Establish basic security hygiene in pipelines

Walk:

  • Add SAST scanning with focused rule sets
  • Implement security review processes
  • Start tracking security metrics
  • Expand vulnerability management practices

Run:

  • Full suite of security scanners
  • Advanced configuration and custom rules
  • Automated security gates in pipelines
  • Proactive security practices

The key is to make each step manageable while maintaining momentum. This builds confidence, shows value, and creates security champions within teams. For different customers the definition of crawl, walk, and run may vary, but the concepts apply regardless.