Security Services Delivery Kit

GitLab Professional Services offers two engagement types related to application security.

From the GitLab website:

The GitLab platform provides an opportunity to transform the way your business automates security processes. However, transitioning and transforming from your current security process to an automated, manageable, and consistent workflow in your GitLab deployment requires customized configuration from security professsionals. Engage with GitLab experts to solve specific security challenges and unlock all of the benefits of the GitLab Platform.

Security and Compliance Advisory

The Security and Compliance Advisory is a fixed-price advisory engagement for customers new to GitLab Ultimate that demonstrates core security features through a pre-built demo project and provides roadmapping sessions to help organizations adopt and integrate GitLab's security capabilities into their development workflows. It is a purely advisory engagement where delivery team members do not have access to the client's environments and code. For hands-on engineering, customers should purchase the DevSecOps Transformation service (below).

From the GitLab website:

The Security and Compliance Advisory offering allows for GitLab engineers to show off GitLab Security features and demonstrate core security policy automation benefits as well as provide strategic guidance to create a roadmap to kick start your security journey with GitLab.

Fact sheet (Google Drive)

Security Consulting / DevSecOps Transformation

The DevSecOps Transformation directly supports customers along their adoption journey with Application Security Testing, Security Risk Management, and Software Supply Chain Security, to reduce security and compliance risk across the organization. It is a hands-on engineering engagement where delivery team members will have access to the client's environments and code.

From the GitLab website:

For customers who want to transform the way they secure software using GitLab CI, we provide a skilled GitLab engineer to help define and implement security policies, triage vulnerabilities, and audit your GitLab deployment in an automated, scalable way.

Fact sheet (Google Drive)

Security and Compliance Advisory

This fixed price advisory service is recommended for customers new to GitLab Ultimate, including upgrades from Premium, who are looking to understand core DevSecOps principles and how to get the most out of GitLab Ultimate features. The workshop is particularly valuable for organizations that are new to application security practices or are looking to enhance their existing security workflows.

It includes an introduction to GitLab Security features, including shift-left concepts, security dashboard, vulnerability management, security scan capabilities, policies, compliance frameworks, and other features.

The roadmapping sessions advise customers on how to plan for and adopt GitLab's application security features, helping clients to:

  • Define security policies including:
    • Types of security scans required.
    • When to run each scan.
  • Determine which projects should be scanned, automate scans and enforce scans.
  • Understand the GitLab Security Dashboard.
  • Define a vulnerability remediation strategy.
  • Educate both developers and security team members on the GitLab compliance process.

Sales and pre-launch considerations

  • This service offering does not involve touching customer code at any point. We use pre-built, open source projects to showcase GitLab Ultimate security features, and dive deep into each of the security topics mentioned above.
  • This is a fixed price SKU service (not customized or variable), the scope and price estimate of this service is small.
  • For any customer looking for a more advanced security focused engagement, they should look at our DevSecOps App Transformation service, whose scope is larger because it involves working on customer projects.

For fixed-price engagements, we use a service description document instead of the traditional SOW. This still has a list of all of the activities that will be covered in the engagement, but doesn't tie specific hours to specific tasks. You can find the Security and Compliance Advisory Service Description here..

Delivery requires access to the GitLab demo environment and maintainer access to the Security and Compliance Advisory Compliance Group. To receive access, generate credentials by following the demo system documentation.

Professional Services team, please use the demo architect portal to request a workshop LearnLabs group to be created. First, use GitLab Admin login to sign in and follow the instructions to do a content/lab request and complete the form with the necessary information. Afterwards, you will be notified when the issue to create your group is created and then the group link when it is actually created. Please export/import the Security and Compliance Advisory project demos you'd like to use into the LearnLabs group and provide the partner with access permissions.

Target Audience

The target customer for this service offering is a customer moving from Premium to Ultimate or a new Ultimate customer or an Ultimate seat expansion. The main service offering is meant for a PSE to showcase GitLab's security product capabilities, facilitate deep-dive sessions, Q&A, and/or strategic discussions about GitLab security features.

Unlike GitLab's education services, there is no limit to how many people from the customer organization can participate in this advisory, making it an excellent opportunity for cross-team collaboration. The customer should ensure that security SMEs and champions are in attendence.

Advisory Structure

The Security and Compliance Advisory engagement typically breaks down into three main components:

  1. Discovery/Inquiry (approximately 2 hours of meetings) - An open discussion to understand the customer's current security practices, tools, and challenges. This helps tailor the subsequent parts of the engagement to the customer's specific needs. Aim to establish 1 or 2 items from discovery that you can directly showcase in the demo. Leverage these discovery questions to help.

  2. Demonstration (approximately 2 hours of meetings) - A hands-on showcase of GitLab's security capabilities and vulnerability management features using pre-built projects. This gives customers practical experience with the tools.

  3. Strategic Discussion - A strategic session to map the customer's specific challenges to GitLab Ultimate features, with a focus on shifting security left in the development process. Review a roadmap that details what implementing information security into an organization could look like.

Demo Overview

The projects to use are widely known, open-source projects. They are:

Slides are available for use and are located in the deck Security and Compliance Advisory. The slides were designed to help pave the way for how the demo can flow. Please add comments and make changes as necessary so that everyone can benefit.

These projects were chosen because they provide:

  • Intentionally vulnerable applications - Designed with known security flaws to simulate real world applications
  • Multiple vulnerability types - Include SAST, DAST, secret detection, dependency scanning, and container scanning findings
  • Realistic codebases - Written in commonly-used languages and frameworks (PHP, Java, Node.js) that customers actually work with
  • Consistent demo experience - Externally well-maintained projects that produce predictable scan results without requiring customer code
  • Educational value - Widely recognized in the security community, making them excellent teaching tools for security concepts
  • Multi-language coverage - Allows demonstrating GitLab's language detection and appropriate analyzer selection across different technology stacks

These projects will be showcased to customers to teach them the following core GitLab Ultimate concepts and benefits:

Remember this delivery is designed to be flexible. Make changes you feel necessary, within reason, based on what would provide the most benefit to the customer.

  1. Shift-left mindset in MR workflow and DevSecOps best practices.
  2. Inclusion of GitLab scanners.
  3. Creation of a policies.
  4. Creation of compliance frameworks. You can use these prebuilt compliance frameworks templates.
  5. Built-in GitLab security dashboard and vulnerability remediation.

Delivery Steps

For delivery, the intent is to provide as much relevant information to the customer, based on discovery, as you can in the ~2 hours of demo time. This delivery is designed to be flexible, within reason. It is encouraged to set this up fully ahead of time so that time is not used during the demo setting up items like policies or frameworks from scratch.

It's important to faciliate conversation with the customer to allow them to start thinking about how they can utilize these features and practices in their organization. Stop frequently to ask if the customer has questions or if they would like to discuss a topic in greater depth.

1. Access the Security and Compliance Advisory Compliance Group in our sandbox environment and request maintainer access, and that should have everything you need for delivery. See prerequisites above for more details details.

2. Create a new subgroup named "Security and Compliance Advisory - JohnDoe", replacing "JohnDoe" with your name. In your subgroup add the three projects DVWA, juice-shop, and Webgoat. You can find the original projects in the Security and Compliance Advisory Group and add them to your new subgroup by:

  • Creating an export (Settings-->General-->Advanced-->Generate & Download Export), hitting the "New Project" button, and attaching the GitLab export file (*tar.gz)
  • Cloning the Security and Compliance Advisory - Original project down to your local machine, create a new blank project in the GitLab UI within the Security and Compliance Advisory Compliance Group, and pushing it to "Security and Compliance Advisory - JohnDoe", replacing "JohnDoe" with your name.

3. Showcase the .gitlab-ci.yml file on the default branch and briefly walk through the stages that exist for the three sample projects.

  • While going through the .gitlab-ci.yml file, take time to point out which type of scanner each template is since they were introduced in the slides.

4. Create various policies: Merge Request Approval Policy, Scan Execution Policy, and Pipeline Execution Policy. Discuss how these policies enable the shift-left mindset.

  • Showcase the separate policy project that has been created with a similar name to the project/group where the policy was created.

5. Create 2-3 compliance frameworks. Point out that you can use these compliance frameworks to enforce policies based on the compliance framework assigned to the project. Showcase the Compliance Dashboard and point out the various reports that can be exported.

6. Showcase other items under the Secure tab such as the Security Dashboard, Vulnerability Report, and the Dependency List. Demonstrate how the view will change based on where you are in the hierarchy when looking at the Security Dashboard, Vulnerabilty Report, and the Dependency List. Navigate around the various functionality within each tab, such as showing how to do mass triage on the vulnerability report and how you can export SBOM from the dependency list. Audit Events is available as well and Ultimate gets Audit Event Streaming. It is encouraged to setup Audit Event Streaming to showcase how it works and mention how actions can be performed automatically based off using event information.

7. It's important to remind the customer that GitLab is a tool. It may not contain all the functionality they would like out of the box. That is where functionality like CI/CD components can come in and help extend pipeline functionality. Remember, just because it's not native to GitLab does not mean it cannot be done through GitLab.

Guide to Facilitate Deeper Dive On Some Concepts

It's important to note that customers often face process challenges more than technical challenges. Be prepared to discuss topics such as:

  • Organizational structure for security (many organizations don't have dedicated security personnel)
  • Branching strategies and workflows
  • GitLab group structure and organization
  • Integration with third-party scanners (e.g., SonarQube)
  • Secret push protection

Shift-Left Security Mindset + Merge Request Setup

Watch this video from the documentation that summarizes the shift left mindset.

  • The stages and jobs defined in the .gitlab-ci.yml file will run and output meaningful results (test outputs, logs, etc.) on the GitLab UI.
  • Using Auto DevOps templates, we can include security scans with 1 line of code to run security checks and output logs to developers before code gets merged into the default branch. The templates run the scans in a test stage that is automatically created for you.
  • Merge request approvals can be set up so that certain users or groups of users need to review a merge request before that branch is merged into a specific branch. This encourages code reviews, increases quality, and prevents unwanted changes from being released.
  • Customization allows for multiple approval rules to be created, different rules per branch, and rules based on security scan results.

Secret Detection

  • GitLab's Secret Detection solution scans through source code to look for over 100 common patterns that would signify a token or password (i.e. access tokens, popular cloud provider keys, URLs, etc.).
    • The GitLab solution spins up a container that runs the GitLeaks scan for you in the test stage.
    • You can override certain predefined rules or create custom rules if you need to.
  • By default, secret detection only scans commits, not the working tree, but the scan template can be configured to do a historical scan, scan all commits in a merge request (use the secret-detection.latest job for this), scan all branches, etc.
  • You need a docker or k8s executor on a Linux based GitLab runner to use the secret detection GitLab template.
  • You can configure the scan to omit certain paths, secrets, and use a specific version of the analyzer.
  • The job outputs an artifact in JSON format which by default is available for 7 days.

SAST and GLAS (GitLab Advanced SAST)

  • GitLab's SAST scanner checks out source code for vulnerabilities and outputs the report into a JSON artifact.
    • Examples of SAST vulnerabilities include a dangerous class attribute or unsafe code that can lead to unintended code execution or code that is available to cross-site scripting (XSS) attacks where unauthorized access to session details can be attained.
    • The SAST job runs in the test stage when including the template.
      • Because the job automatically runs in the test stage, and by default does not contain the needs keyword (aka it does not depend on any other job), it will run after the build and/or containerize job is run and scan the entire working tree. Therefore, if anything from those jobs have been saved as artifacts or within the cache (i.e. a poetry install in a build stage), secrets detection could potentially detect dozens of vulnerabilities based on the dependencies that were installed.
      • The solution here would to be to use the needs keyword to ensure secret detection runs first or encourage teams to run it in the .pre stage instead of the test stage.
  • You need a docker or k8s executor on a Linux based GitLab runner to use the SAST GitLab template.
  • GitLab's SAST scanner detects the language used in the project and automatically runs the corresponding analyzer.
  • False positive detection, currently is only available for Ruby and Go projects for Ultimate customers.
  • Certain projects need to have code pre-compiled before running a SAST analyzer, and this can be achieved by outputting project dependencies into a working directory via the artifacts keyword, setting the COMPILE variable to false, and making sast depend on the build job with the needs keyword.
  • You can configure the scan to omit certain paths, use a specific version of the analyzer, increase the debug log level, modify the depth of scan, pass credentials to a private repository, etc.

Security Dashboard & Vulnerability Remediation

  • The GitLab Security Dashboard is the one stop view to track the results of the most recent GitLab security scans on the default branch.
  • The dashboard provides a view of vulnerabilties over time at both the project and group level, and for the latter, will assign a grade based on the highest severity open vulnerability (does not include dismissed or resolved vulnerabilities).

Vulnerability Management

  • The Vulnerability Report for a project shows details on each vulnerability including when it was detection, description, where in the code it was detected, linked issues, and available actions.
  • The report can be filtered by date, severity, scanner, activity, and project.
  • Here's a short video on vulnerability management.
  • The default vulnerability status is Detected, which shows up as "Needs Triage" in the UI. Dismissing a vulnerability ensures it won't pop up on subsequent security scans while resolving a vulnerability will change its status, but allow the vulnerability to pop up again if the problem still exists.

Scan Result & Scan Execution Policies

Scan result policies are used to enforce a custom approval rule when a security scan detects a vulnerability.

  • Scan result policies can be enforced at the group level with a corresponding security policy project and customizable policy.yml file that gets created within the group.
  • Customize the approval group to ensure certain people need to approve an MR with vulnerabilities for it to be merged.
  • This enables developers to collaborate with security to determine whether the vulnerability is a false positive, insignificant, or requires a code change.
  • You can customize your scan result policy to only go in effect for a certain severity of vulnerability (low, medium, high, critical).
  • The policies are only enforced when the target branch for the MR is a protected branch.

Scan execution policies can be used to schedule pipeline runs on the desired interval.

  • Enforce these policies on all projects, whether or not they have a .gitlab-ci.yml file.
  • These are the future direction for GitLab and will be customizable to allow for custom CI job inputs into the policies.

Discovery + Roadmapping

The engagement is 8 days, and yet the demo above and deep dive sessions should only take a few days to deliver. So what does the rest of the engagement entail?

  • Remember, the goal of this engagement is to ensure the customer leaves with a foundational understanding of GitLab Ultimate security features AND a path for how to transition to using it. While we don't touch customer code in this engagement, the remaining engagement time can and should be dedicated to the following activities. Time spent on each activity may vary as this is just a general outline of discussion points.

  • The roadmap outline is a template that should be modified based on the customer's current information security maturity. To simplify this process, there is a Claude Project created to help generate a roadmap based on the information received from the customer through this engagement.

  • Setting expectations: It's crucial to clearly communicate what this service includes and what it doesn't. Some customers may have different expectations about what they've purchased, so clarify the scope early to avoid misunderstandings.

    1. Understanding the customer's current security setup and design.

    • What is your security team structure and group responsibility breakdown?
      • Do they have separate teams for QA, application security, compliance/governance, etc.?
      • Who is responsible for ensuring code in production is secure? How is this done today?
    • What security tools are you currently using and what are they used for?
      • Are the GitLab scanning solutions able to displace the customer's existing solutions?
      • What technology stacks does your business use?
      • Are there any applications that use languages or frameworks that aren't supported by GitLab?
      • What are your current security requirements, processes, and/or policies?
        • Do you have certain code scans that are required for certain applications or when deploying to certain environments?
        • When, if at all, do you require manual intervention and review of code?
        • How do you manage vulnerabilities today?
        • Do you have a standard way of enforcing all applications run the required security scans?
        • How do you manage security across a large number of applications?

    2. Identify why the customer is interested (i.e. what is not working so well with their current setup).

    • Are there too many disparate tools to manage?
    • Is there a lack of consistent process or security standardization between applications?
    • Are there too many manual processes that drain the resources?
    • Is there a lack of collaboration or even tensions between the security and development teams?
    • Is it too difficult to manage security and/or ensure all of the applications meet the necessary requirements?

    3. Using the above information, discuss topic by topic how GitLab can solve the customer's issues.

    • Examples:
      • Security policies can enforce security controls across applications.
      • Shifting security to the left can foster collaboration between security and developers to ensure all code is checked before moving to production.
      • The security dashboard allows you to manage vulnerabilities across projects, gather data about vulnerability remediation trends, and export that data for your own reporting purposes.
      • Security can easily be automated within pipelines.
      • All of the above can be done within a single platform to simplify the user experience for both developers and security team members.

    4. Using the above discovery information, design the security vision of what the customer's end state GitLab setup could look like.

    • Example:
      • 10,000 foot view: GitLab SAST, DAST, Secrets Detection, and Container scanning.
      • Organization: Top level group contains centralized repository of security templates with different scanning setups that are automatically enforced. Developers don't need to worry about including any security scans in their code as these security scans are automatically injected into the application pipelines. The security team has a single location for all of the templates, which makes it easy to manage and maintain. Developers can easily contribute back to those templates through issues and merge requests. The necessary security folks will be automatically asked to review any application code MRs that violate a security policy.
        • Workflow: Every code commit triggers a SAST and Secrets Detection scan. On merge request pipelines, additional scans can be run. There are different security policies assigned to differentiate between web applications, payment applications, business applications, etc., each with different objectives to match specific security requirements. Security professionals can use the Security Dashboard to pull reporting and trend data for vulnerabilities broken down by severity at both a group and project level.

DevSecOps Transformation

Our Security and Compliance transformation work supports the GitLab Value Driver of Reduce Security and Compliance Risk - Simplify processes to comply with internal processes, controls and industry regulations without compromising speed.

Product capabilities

Many of the capabilities require an Ultimate tier GitLab subscription, so Security and Compliance Risk Reduction falls under the broader category of "GitLab Ultimate Adoption".


Start with the Transformation Mindset

A Transformation Mindset

Typically, application security adoption follows broad adoption of GitLab for SCM and CICD. But it's possible that some team members are still onramping with some aspects of GitLab Flow such as feature branches and shifting left, and might even be new to Git itself (i.e. from Perforce or svn) - that's OK.

Often, senior leadership and team members have a beginning understanding of GitLab's application security offerings (i.e. from the sales process) and want to learn more. That's a hook for getting started.

It's best if the company has CI/CD in place for other operations (unit testing and cloud deployments, for example) - but not entirely necessary.

For success, it's critical that we provide leadership across both aspects of the application security transformation.

Technical Aspects of Success

The actual GitLab features that support business goals and the integration/configuration of those features in the groups, subgroups, and projects. Plus enablement of hands-on teams in efficient utilixation of the product's capabilities.

Organizational

Aligning the way people plan, communication, collaborate, work, and think with business goals, organizational values, regulatory concerns, and each other.

Organizational transformation has 5 parts:

  1. Strategy: Clear goals and objectives (OKRs, KPIs, MBOs, etc) combined with meaningful organizational values
  2. Skill: Knowledge and abilities of individuals, backed by training, enablement, and support
  3. Styles: A generative culture that fosters both autonomy and collaboration
  4. Structure: Appropriate roles, job titles, reporting relationships, dotted lines, and incentives
  5. Systems: Tools and technology that underlie successful processes (including but not limited to GitLab)

As a product/engineering-driven software company, we have a tendency to focus on the technical aspects of transformation and neglect the organizational ones - but those are the hard ones, and must be addressed to ensure success.

Technical Aspects of Transformation

GitLab features related to application security

  • Scanners
    • All the types
    • Results in artifacts
    • Results in MRs
  • Security policies
    • Policy projects
    • Scan execution policies
    • Scan results policies
  • Executive reporting
    • Security dashboard
    • Security center
    • Vulnerability report
  • Vulnerability management
  • Compliance frameworks
  • Integration of 3rd party scanners

Organizational Aspects of Transformation

For developers

  • Agile best practices
  • Git best practices (if coming from pre-Git SCM)
  • MR best practices (if coming from other Git SCM)
  • Branching/MR strategy (first agreed at org level)
  • Integrating scanning with automated build/test
  • Feature/MR approval/merge process/workflow
  • Runner resource optimization (maybe some scans still only run post-merge or pre-release)

For devops teams

  • Maintenance of GitLab (is this SM?) and Runners
  • Release verification and deployment/distribution process
  • GitLab and scanner administration
  • Process for future enablement of dev and infosec teams re Git, GitLab, scanners, and workflows

For infosec team

  • Decisionmaking processes (which scanners, what stages)
  • Responsibility (DRIs) for scanner implementation, vuln management, MR review, release approval, etc
  • Process and staff allocations for handling results from first-time scanning passes

For everyone

  • Regular process for handling results and vulnerabilities, including escalations
  • Collaboration and communication process with other teams
  • Integration of vulnerability management with agile planning (i.e. issues to resolve vulns)

Getting started

Project phases might include:

  • Advanced education - even after digesting courseware and documentation, teams might require informal enablement to address specific requirements and questions
  • Pilot project selection - start small with 3-5 projects - enable scanners first to develop remediation workflows
  • Prioritization / sequencing - GitLab has a lot of application security capabilities - choose an order that makes sense for the org
  • GitLab organization - Teams such as CS, SA, Field CTO, product marketing, product mangement, and engineering all have deep insight and love to help customers
  • Initial scan remediation - how to handle the onslaught of vulns - process and communication - pay particular attention
  • DevOps fundamentals - group hierarchy, branching strategy, agile, merge methods, approval policies, etc. - all prerequisite for effective app security

Shared Responsibility

The Security and Compliance capabilities of the GitLab product cover only one aspect of security -- application security. Other aspects, such as network, infrastructure, social, and endpoint security remain the responsibility of the customer organization. Some customers need help understanding the difference.

Furthermore, while GitLab provides the tooling (and sometimes infrastructure) for application security testing and vulnerability management, the ultimate responsibility for application security lies with the customer.

The language below might help to explain the nuance of shared responsibility, loosely based on the AWS Shared Responsibility Model for cloud services.

Customer infrastructure and applications

Customers bear responsibility for implementing and managing the security of their applications and applications, including how they utilize GitLab's security tools within their development pipelines. This encompasses reviewing and acting upon identified vulnerabilities, managing dependency updates, and implementing recommended security fixes

Customers must configure security scanning tools appropriately, interpret results, and make informed decisions about vulnerability remediation priorities.

For example, while GitLab provides dependency scanning capabilities to identify known vulnerabilities in external libraries, customers are responsible for actually updating these dependencies and ensuring their application remains secure. Similarly, while GitLab provides SAST scanning capabilities, customers must enable and configure the rules and execution policies appropriately for their specific use cases.

GitLab platform capabilities

GitLab develops, maintains, and delivers capabilities that support customers in ensuring security of their infrastructure and applications. The Application Security capabilities including several types of vulneravility scanning engines as well as various user interfaces and APIs for understanding and addressing vulnerabilities.

Our focus is on preventing software vulnerabilities from entering the supply chain - an approach known as "shift left".

We focus on improving scanning accuracy, reducing false positives, and enhancing detection capabilities through continuous research and development. In addition to the scanners themselves, GitLab maintains a database of known vulnerabilities and groups of analyzer rules. Most of the vulnerability information and rules come from public sources. Parts of the company focus on performing security research, improving GitLab's security capabilities, and developing proofs of concept while maintaining the databases and fulfilling CVE Numbering Authority duties.

GitLab platform security

With regard to the GitLab platform itself, responsibility for security falls mostly with GitLab.

GitLab maintains responsibility for providing and securing the fundamental security scanning and vulnerability management infrastructure. This includes maintaining and updating the security scanning engines, vulnerability databases, and the platform infrastructure that enables security testing.

GitLab also manages the infrastructure of GitLab.com, handling maintenance and addressing any incidents that arise.

In the case of self-managed GitLab, instance administrators take primary responsibility for infrastructure security. The GitLab support organization is available to help.

Security incidents

(Including breaches and "zero-day" vulerabilities)

When handling a security incident, follow guidance on the Responding to security incidents page in the documentation.

Customers must handle security incidents in their area of responsibility, including:

  • CI/CD security incidents, including exposed job tokens and pipeline secrets
  • Project and group setting misconfigurations
  • Credential exposure incidents
  • User account compromises within customer organizations

GitLab provides capabilities to support customers, including:

  • Audit events
  • Credentials inventory
  • Secrets detection

With regard to "zero-day" vulnerability discoveries, the best defence is to ensure that application SBOMs (Software Bill of Materials) are up-to-date so teams may quickly identify vulnerable applications.

Managing vulnerabilities

The goal is to prevent breaches and attacks. Active vulnerability management is key. At a high level:

  • The scanners scan and identify potential vulnerabilities
  • The Vulnerability Report provides a mechanism to understand, accept, dismiss, comment, and assign vulnerabilities
  • A developer or infosec professional has to make the decision as to which vulnerabilities to accept or dismiss
  • The “Explain This Vulnerability” and “Security Training” capabilities can help in some cases

One of our team members shared:

Identifying false positives requires investigation. There may be times when it’s “obvious” but those are likely rare. When you have identified a false positive, though, it might be best to tweak the rules or configuration to avoid detecting such instances in the future, where tweaking rules is available.

Resources

The Nested Nature of Security

Security responsibilities are like Russian (Matryoshka) dolls, fractals, or turtles all the way down - the same patterns repeat at different scales. For example:

GitLab <-> Customer

  • GitLab provides the software, security features, and patches
  • Customer decides how to use them and when to apply updates
  • For self-managed, customer owns infrastructure security
  • For GitLab.com, GitLab handles the infrastructure

Within the Customer org: Security Team <-> Platform Team

  • Security sets policies and requirements
  • Security maintains vulnerability management program
  • Platform implements and maintains controls
  • Platform handles infrastructure patching and hardening

Platform Team <-> Dev Teams

  • Platform provides secure infrastructure and tooling
  • Platform configures security scanners and guardrails (components, frameworks, etc.)
  • Dev teams configure security in their pipelines (sometimes)
  • Dev teams manage application dependencies

Dev Teams <-> Individual Devs

  • Teams set security standards and review processes
  • Teams handle vulnerability remediation priorities
  • Devs make day-to-day security decisions in code
  • Devs implement secure coding practices

The common pattern being one party provides capabilities, the other decides how to implement them. This creates a chain of shared responsibility that scales from the organizational level right down to individual code commits. Even within each level, you'll find the pattern repeating - like how dev teams handle both infrastructure-as-code security and application security, each with their own provider/implementer relationships and alignment to enterprise strategies.

Preparation

"Phase Zero" activities

Process

(Enablement as required throughout)

  1. Set expectations
    • Everything will be red at first
    • Continuous improvement
    • Requires research, decisionmaking, collaboration, and additional coding
  2. Establish a cross-functional team
    • Executive sponsor
    • InfoSec
    • DevOps
    • Product
    • Application team roles- Technical Architect and Dev Leads
  3. Identify and prioritize pilot projects/teams

Collaboration focus:

  • Select pilot team(s)
  • Determine integration requirements (Jira, 3rd party scanners, etc)
  • Determine current state (i.e., some scans already running)
  • Summarize languages and frameworks used (to determine scanner applicability)

Pilot Project Selection

Characteristics of a pilot team/project

  • Already using GitLab, Merge Requests, and CI/CD
  • Already partially familiar with concepts like SAST and Secret Management
  • Publishing a web application or API service, to provide effective confirmation of DAST capabilities
  • Having projects in standard conventions like containerization via Dockerfile or package management with package managers like poetry,maven,yarn etc. in order to use Container Scanning or Dependency Scanning capabilities
  • A standard model, for how most other teams work. Especially how branches and merge request approvals are managed should be matching overall setup

Custom role

The security team will need the ability to work with vulnerabilities and compliance frameworks in order to do their work. Fortunately, the Custom Roles capability supports such activity. Specifically, we need a Custom role at the top-level group with the following properties:

  • Name: Application Security (or similar)
  • Description: Manage application security policies, vulnerabilities, and workflow (or similar)
  • Base role: Reporter
  • Custom permissions:
    • Admin compliance framework
    • Admin merge request
    • Admin vulnerability
    • Manage security policy link
    • Read dependency
    • Read vulnerability

(Note there is an active issue to standardize such a role in GitLab.)


Application Security Testing

All activities related to enabling scanning

Capabilities

Capabilities under the heading "Application Security Testing" include:

  • Scanners
    1. Secret detection
    2. Dependency scanning
    3. Container scanning
    4. SAST + SAST-IaC
    5. API security testing
    6. DAST browser-based scanning
    7. Fuzz (both kinds)
  • Scan Execution Policies
  • Pipeline Execution Policies

Approaches

Note that there are three distinct ways to implement security scans

  1. Direct CI/CD - Individual scanners in the CICD configuration for individual projects - perfect for ensuring operational integrity and generating some initial data so that workflow, triage, and remediation process development can begin.
  2. CI/CD library - alongside other shared CICD configuration, wrappers for the scanner configurations - allows parameterized configuration to be shared among projects - best way to organize CICD configs in a large organization.
  3. Scan Execution Policies - enforce scanning across a portfolio of projects with minimal adaptation. More complicated for DAST and API scanning. Doesn't support Fuzz scanning (?)

Third-party scanners

...

Security Risk Management

Information security professionals face a daunting task when first adopting GitLab Application Security Testing, whether coming from other scanning software or entering the world of automated security testing for the first time. The workload can overwhelm them. We break it into two distinct phases:

Bulk Triage Phase

Handling vulnerabilities that appear in the Vulnerability Report when scanners are first run on a project

Continuous Security Phase

Handling vulnerabilities that appear after bulk triage has completed, whether because of code changes or newly understood vulnerabilities

Other resources

Issues within AppSec SME channel (GitLab team only)

Bulk Triage Phase

When first adopting GitLab Secure

  1. Prioritize projects according to exposure risk: High/Medium/Low based on business impact and network exposure. Start with the high-priority projects and move down. Approach each project one at a time for initial bulk triage
  2. For one project at a time, ensure that scan execution policies have been applied to the project and have run successfully on the default branch
  3. Filter the Vulnerability Report for
    • Project: The project under examination
    • Status: Needs triage
    • Severity: Critical
    • If necessary because of the size of the results list, a scanner in the following order:
      1. Secret detection
      2. Dependency scanning
      3. Container scanning
      4. SAST + SAST-IaC
      5. API security testing
      6. DAST browser-based scanning
      7. Fuzz (both kinds)
  4. Starting at the top of the search results, open each vulnerability
    • Read the vulnerability page thoroughly
    • Click through to the links as necessary to understand the vulnerability
    • If needed and available, click "Explain this vulnerability with AI" for more information
    • Decide how to handle this vulnerability
    • Execute the decision by choosing a Status in the top right corner
    • Add a comment to the vulnerability explaining the decision
    • If the status is Confirmed, create and assign a Jira issue
  5. Continue through the remaining vulnerabilities in the search results, performing the previous step for each
  6. If you filtered by scanner, update the filter to the next scanner and repeat
  7. Complete triage of all the Critical vulnerabilities in one project before moving to the next project
  8. Complete triage of all vulnerabilities in the following order shown in the table below
  9. While triaging vulnerabilities, when trends appear, consider adjusting the scan execution policies by referencing the GitLab documentation. If different projects and/or groups seem to require different policies, consider creating separate scan execution policies and linking them appropriately
    • All the scanners have variables to customize
    • Secret detection and SAST have RULESET_GIT_REFERENCE variables
    • DAST browser-based scanning and API scanning have site profiles

Proposed order for vulnerability bulk triage

High risk projectsMedium risk projectsLow risk projects
Critical vulnerabilities124
High severity vulns357
Medium severity vulns6810
Low severity vulns91112
  1. Perform initial application risk assessments
  2. Prioritize applications
  3. Set non-intrusive scanners
  4. Establish guardrails with exceptions
  5. Monitor and adjust

Continuous Security

  1. Triage vulnerabilities within 24 hours to determine:
    1. Does the vulnerability actually apply to your application. Is it reachable, does the impact apply to your application, etc.
    2. Who reported the vulnerability? If the issue is reported through a bug bounty, customer, or penetration test, it is often more critical to solve compared to a security scanner
    3. If the vulnerability is valid, how likely is it to be exploited? Is there a public exploit, is the attack complex, is it easy to reach the vulnerable component?
    4. What impact would the vulnerability have if it were exploited? This is commonly done using impact to CIA (Confidentiality, Integrity, Availability)
  2. If the vulnerability does apply, assign a priority based on:
    1. Severity, usually based on the CVSS score. The higher the CVSS score, the higher priority the fix is
    2. How likely an exploitation of the issue is. If there is active exploitation or the possibility for it, higher priority could be assigned
  3. Develop a fix to the vulnerability - Of note, a vulnerability fix will typically apply to the latest release, but may also apply to older releases depending on your support model
  4. Determine a release strategy - Larger companies often follow a specific cadence for security releases, like Patch Tuesday, with out of band releases for critical issues. I've also seen monthly release cycles like Android security bulletin. The release cadence will vary a lot by company and release cycle preferences
  5. For fixing common vulnerabilities, there are a few details that might be good to discuss:
    1. Default to known secure patterns for development. For example, use parameterized SQL vs concatenation. Our security outputs do a great job of showing these approaches and linking to CWE resources to help here
    2. In addition to this, using DRY to standardize is helpful. For example, having a single class of SQL functions helps ensure all SQL queries use the same logic, avoiding mistakes in implementation
    3. Resources like OWASP, CWE, and NVD are often helpful for understanding these patterns
  6. Looking for variants on a security vulnerability is often advisable. If for example you find a SQL injection vulnerability, you should look at your code to try to identify if other examples exist of the same weakness

Vulnerability Evaluation

During both the initial bulk triage phase and the later continuous security phase, an information security professional must evaluate individual vulnerabilities and make decisions.

How to evaluate

  • Criticality and impact - Vulnerabilities that are highly critical and could result in severe consequences should be prioritized
  • Attack vector - Vulnerabilities with a clear and practical attack vector should be prioritised
  • Industry best practices - Stay informed with your industry best practices and compliance standards that are explicitly highlighted by the regulatory bodies
  • Prioritization with other project work - Organizational leaders must agree that vulnerability remediation is as important to the business as other development activities

Vulnerability Assessment and Severity Ratings

Vendor-Specific Vulnerability Ratings

Different vendors may have varying criteria for assessing vulnerability impact. For instance, historically, a "low impact" rating from Red Hat typically indicated that a vulnerability was either unlikely to be exploited or that its exploitation would not likely lead to severe consequences. While specific vendor criteria may evolve over time, the fundamental principles often remain consistent.

National Vulnerability Database (NVD) and Vendor Assessments

The National Vulnerability Database (NVD) maintained by NIST has faced challenges in recent years, including a lack of contextual information and significant backlogs. As a result, many security professionals often prioritize vendor-specific vulnerability assessments. Vendors frequently employ more dedicated researchers than the NVD, potentially allowing for more nuanced and timely vulnerability evaluations.

Mitigation Strategies

For certain types of vulnerabilities, such as those potentially leading to Distributed Denial of Service (DDoS) attacks, implementing rate limiting can often serve as an effective mitigation strategy.

Contextual Nature of Severity Ratings

It's crucial to understand that severity ratings should not be interpreted in isolation. The impact of a vulnerability can vary significantly depending on the specific context of the affected application or system. This contextual nature underscores the importance of human review in the security assessment process.

Example

A potential DDoS vulnerability might have catastrophic implications for a mission-critical or safety-critical system, while its impact on an e-commerce website, though still significant, might be comparatively less severe.

Best Practices for Vulnerability Management

  1. Consider multiple sources when evaluating vulnerabilities, including vendor-specific assessments and broader databases like the NVD.
  2. Understand the specific criteria used by different sources to determine severity ratings.
  3. Always contextualize vulnerability severity within your specific application or system environment.
  4. Implement a robust vulnerability management process that includes both automated tools and human expert review.
  5. Prioritize vulnerabilities based on their potential impact on your specific systems and business operations.

Risk and Confidence

From issue under GitLab UX Research entitled Further understand the job of prioritizing security scan results and assessing their risk

Users want to estimate the potential impact of a vulnerability if it is exploited.

Users want to estimate the likelihood of a particular vulnerability being exploited.

Most organizations have a clear process for assessing the risk of security scan results and prioritizing them alongside other vulnerabilities, even if it's not always formally named.

Risk assessment is more of a top-down process compared to vulnerability management. Risk assessment tries to determine how the organization can be exploited, assess the likelihood of those events happening, estimate the event's impact, and take necessary actions to prevent that exploitation.

The data points that users currently use to estimate the likelihood of exploitation and possible impact are Severity rating, what branch/group/project the vulnerability was in, the title and description of the vulnerability, what type of scanner was used to find that vulnerability, and the CVE number if available.

From survey results:

  • Most (6/8) participants did not look at security scan results outside the default branch.
  • Most (6/8) participants only circumvented their typical workflow if an immediately exploitable vulnerability surfaced.
  • The majority (5/8) of participants did not expect a risk assessment tool to contextualize results to the organization's business needs or infrastructure. But, everyone expressed a desire if a tool could do that.

A business-critical threat is one that: 1. has a high chance of being exploited 1. disrupts the business model in a significant way 1. can extract customer or user data from the organization.

Also see the discussion of "confidence level" as a concept.

Capabilities that help with understanding and prioritizing vulnerabilities

Software Supply Chain Security

Adoption Roadmap

  • Integrate this content into the structure of the delivery kit

Phase 1: Perform pilot with 1-4 dev teams/projects

Key staff: Pilot project development team

Key GitLab capabilities: CI/CD, Scanner templates, Pipeline security tab

Collaboration focus:

  • Implement individual scanners in the CICD configuration for individual projects
  • Run on both default branch and MR scans
  • Include static and dynamic scans where appropriate
  • Ensure operational integrity (do the scans run and generate data?)
  • Identify performance optimization opportunities

Phase 2: Initiate vulnerability management

Key staff: Information Security team

Key GitLab capabilities: Vulnerability Report

Collaboration focus:

  • Immediately upon the arrival of scan data, begin reviewing vulnerability report

Phase 3: Activate scanners universally

Key staff: Information Security team

Key GitLab capabilities: Scan Execution Policies, Pipeline Execution Policies, CI/CD components

Collaboration focus:

  • Generate security policy project(s)
  • Replace one-off scanner implementation in pilot project(s) with links to security policy project(s)
  • Implement Pipeline Execution Policies (or CI/CD components) for scan types unsupported by Scan Execution Policies
  • Draft plan for universal policy rollout

Why Scan Execution Policies?

  • They can be rolled out in pilot group
  • Security scans by default are allowed to fail, therefore should not affect pipeline scalability.
  • That being said, if pipeline duration for any monorepos are affected, due to slow SAST execution, this may need to be adressed.
  • Using latest version is possible via template variable, in order to activate scans for both Merge Request Pipelines and Branch Pipelines
  • For air-gapped environments you can use SECURE_ANALYZERS_PREFIX with Scan Execution Policy variable to activate scans.

Phase 3: Develop vulnerability management workflow

Key staff: Information Security team

Key GitLab capabilities: Vulnerability Report, Merge Request Approval Policies

At this stage, we have running pipelines and scan results. Based on the results now we can finally start identifying how a Merge Request Approval Policy and Vulnerability Management process should actually work. At this stage we develop the concept together with the teams involved related to

  • who can change state of a vulnerability.
  • who can approve an MR, when there is a certain level of vulnerability.
  • when are you allowed to deploy a change, even when there is a vulnerability.

Bear in mind, that current GitLab Vulnerability Management is based on GitLab Flow branching model. Identifying, if there are any gaps related to trunk based branch models is crucial in this stage, to consider potential workarounds or customisations to the Vulnerability MAnagement flow

Phase 4: Continuously reduce security risk

Once the piloting stage is complete, the tooling and concepts in place should achieve following.

A self-service rollout VM framework to other teams should be implemented. Teams have the option based on their workflow to trigger rollout themselves with minimal support requirements from Security or pilot teams. Some additional features to cover

  • Type of different compliance frameworks should be identified. Certain organisations like Finance, can require different compliancies. Whilist organisations like HR may be subject to data regulations. Also the separation of duties, may play a role here, in regards to how vulnerabilities are differently classified.

  • We can now consider establishing Pipeline Execution Policies to integrate with other 3rd party scanners or enforce other custom CI yamls in the organisation.

Technical and organizational changes

  • Access control for infosec team
  • Integration with Jira if required
  • Developer teams learn DevOps best practices such as trunk-based development, MR approvals, review apps
  • Infosec and dev teams develop combined communications/workflows for vulerability triage and management

Meeting Teams Where They Are

Security is a journey, not a destination (as with most transformation topics). While we want every team to be secure, trying to implement everything at once is like trying to run before you can walk - you'll likely stumble and create resistance to security initiatives.

Instead, meet teams where they are and help them level up gradually

Crawl:

  • Start with easy wins like dependency scanning
  • Focus on fixing critical vulnerabilities first
  • Build security awareness through documentation and training
  • Establish basic security hygiene in pipelines

Walk:

  • Add SAST scanning with focused rule sets
  • Implement security review processes
  • Start tracking security metrics
  • Expand vulnerability management practices

Run:

  • Full suite of security scanners
  • Advanced configuration and custom rules
  • Automated security gates in pipelines
  • Proactive security practices

The key is to make each step manageable while maintaining momentum. This builds confidence, shows value, and creates security champions within teams. For different customers the definition of crawl, walk, and run may vary, but the concepts apply regardless.