The rise in secret exposure incidents within source code management (SCM) systems has highlighted the need for a comprehensive strategy to detect and prevent such vulnerabilities. This article outlines our comprehensive journey from initial challenges to the implementation of a robust secret detection and prevention framework within our SCM system. Our goal? To create a multi-faceted strategy that not only addresses technological solutions but also includes process improvements and human factors to promote a security-conscious culture.

Past approaches to detecting secrets in source code

Acknowledging these flaws, we started a series of improvements:

At the start, our approach to managing secrets was limited to using an in-house secret scanner. This tool, which was set up to scan all repositories and pull requests (PRs) periodically for secrets using regular expressions (regex), had considerable limitations. These included maintenance issues, missed notifications, and the restriction of results to the scanner's user interface (UI). The lack of proactive prevention measures meant secrets could be committed, with developers remaining unaware of any accidental exposure, presenting a significant risk.

Improvements to secret detection

Realising the shortcomings of our initial system, we set out on a series of strategic improvements to strengthen our secret detection and prevention capabilities:

• Scanner detection improvement
• Continuous monitoring across organisations
• Pre-commit scans for prevention
• Scanning on PRs
• Secret exposure response process
• Awareness and training

Let’s study each in more detail.

Scanner detection improvement

Among the many paths that could lead to improving scanner detection, we decided to utilise a well-maintained open-source tool. Choosing this tool as our starting point was crucial. Its development and upkeep by a large community gave us a reliable base. However, we went further by developing careful plans for scanning member repositories and determining scan frequency.

Aware of the different risks associated with various types of secret disclosures, we decided to treat exposures of both public and private secrets with the same level of concern. This balanced method highlights our dedication to building a strong defence against potential security threats, ensuring our SCM is safeguarded against a broad range of vulnerabilities.

We focused on developing custom logic for periodic scans to thoroughly cover our SCM system. Additionally, we added a continuous scanning feature, detailed in the "Continuous monitoring across organisations" section, enabling swift alerts for potential secret exposures.

A challenge was the high number of false positives from the scans, adding unnecessary noise. The open-source tool was crucial again for its feature to validate secrets, significantly reducing false alerts and helping us concentrate on genuine threats, ensuring every alert was worth investigating.

GitHub SCM and advancing continuous monitoring in organisations

Scanning our organisation's repositories was not enough, as developers could fork repositories to their profiles, potentially committing secrets there. We had to develop a change detection system. Existing hooks worked well for organisation-wide repositories but struggled to monitor individual member repositories, leading to significant implementation challenges.

To address this, we developed a custom monitoring solution for scanning our entire GitHub SCM. This system meticulously scans for any alterations within the organisation's repositories and extends its surveillance to all member repositories, ensuring comprehensive coverage.

Once the scan is complete, the system starts a parallel secret scanning process on the identified repositories, alerting us to any findings. While this parallel scanning process is effective, it offers room for improvement, especially in terms of efficiency and accuracy.

Implementing pre-commit secret scans to secure GitHub repositories

Having detection in place wasn't enough; we needed to prevent secrets from being committed from the start. Therefore, we added secret scanning to our pre-commit process. This was part of our 'shift left' strategy, which included incorporating a secret scanner into the global pre-commit hooks to check repositories for secrets right on the developer's machine. We also aimed to integrate secret detection early in the development pipeline.

Getting the pre-commit hooks deployed and active on all developer machines was tricky. Relying on developers to manually add these hooks to their repositories was unreliable, as we couldn't guarantee their consistent use or effectiveness.

To solve this, we created a mobile device management (MDM) deployment script to install these hooks automatically on all managed devices, ensuring the pre-commit is configured and active. This solution took into account various factors like different device types, MDM platforms, and architectures, with a strong focus on security.

Despite a careful rollout, we received some feedback about the scans slowing down developer workflows, which we improved upon iteratively. This cautious approach helped us refine the system thoughtfully, avoiding rushed decisions and ensuring a smoother and more effective deployment.

Enhancing security with GitHub actions secret scanning in PRs

Despite all these efforts, we overlooked an important area: Pull Requests (PRs).

Our scanner was great at checking repository content but failed to monitor PRs and their comments. Recognising this gap, our Cybersecurity team earmarked it for future attention. Meanwhile, our DevOps team developed a review bot with a secret scanner for PRs to spot potential exposures. This caused a lot of noise due to the lack of a validation mechanism, but using multiple tools provides a safety net, catching vulnerabilities that one might miss.

Our journey to strengthen our defences against secret exposures is ongoing. While there's room for improvement, especially in reducing noise and improving validation, these challenges set the stage for future progress, including our efforts to adopt a shift-left approach from the start.

Secret exposure response process

After setting up the necessary tools, our next step was to develop a quick and effective playbook to handle alerts and reduce exposure time. We needed a detailed plan to decide which secrets to focus on first, considering various factors such as whether the exposure was public or private, the asset linked to the secret, permissions, steps before revoking a secret, the revocation process, investigation methods, and the roles of team members.

Once we had this prioritisation framework, it was crucial to make sure all security teams were fully informed on how to use it. This would ensure a unified and knowledgeable response to any incidents of secret exposure.

Fostering a culture of effective secret detection

Technological solutions alone can't ensure the security of sensitive information. We've realised the importance of human factors in our security framework and have implemented awareness and training programmes. These initiatives aim to educate team members on the importance of secret detection, using preventative tools such as pre-commit hooks and understanding the potential impacts of secret exposure. This comprehensive approach ensures our team members are not just aware of the tools available but also understand their vital role in maintaining our security posture.

Cybersecurity in software development strategy

Our journey towards strengthening secret detection and prevention within our SCM system has been both challenging and rewarding. By addressing technological, process-related, and human factors comprehensively, we have established a robust defence against the threat of secret exposure. This multifaceted strategy not only mitigates current risks but also sets a foundation for continuous improvement, reflecting our commitment to maintaining a high standard of security in an ever-evolving digital landscape.

About the author

Harish Poorna Chander is a product security engineer at Deriv with a shift-left mindset who loves anything related to security, specifically AppSec.